Routing

The routing engine provides an interface to describe how messages from all the topics should be received and consumed.

Due to the dynamic nature of Kafka, you can use multiple configuration options; however, only a few are required.

Routing DSL organization

Karafka uses consumer groups to subscribe to topics. Each consumer group needs to be subscribed to at least one topic (but you can subscribe with it too as many topics as you want). To replicate this concept in our routing DSL, Karafka allows you to configure settings on two levels:

• settings level - root settings that will be used everywhere
• topic level - options that need to be set on a per topic level or overrides to options set on a root level

Note: most of the settings (apart from the consumer) are optional and if not configured, will use defaults provided during the configuration of the app itself.

Karafka provides two ways of defining topics on which you want to listen:

Single consumer group with multiple topics mode

In this mode, Karafka will create a single consumer group to which all the topics will belong.

It is recommended for most of the use-cases and can be changed later.

class KarafkaApp < Karafka::App
  setup do |config|
    # ...
  end

  routes.draw do
    topic :example do
      consumer ExampleConsumer
    end

    topic :example2 do
      consumer Example2Consumer
    end
  end
end

Multiple consumer groups mode

In this mode, Karafka will use a single consumer group per each of the topics defined within a single #consumer_group block.

class KarafkaApp < Karafka::App
  setup do |config|
    # ...
  end

  routes.draw do
    consumer_group :group_name do
      topic :example do
        consumer ExampleConsumer
      end

      topic :example2 do
        consumer ExampleConsumer2
      end
    end

    consumer_group :group_name2 do
      topic :example3 do
        consumer Example2Consumer3
      end
    end
  end
end

Multiple subscription groups mode

Karafka uses a concept called subscription groups to organize topics into groups that can be subscribed to Kafka together. This aims to preserve resources to achieve as few connections to Kafka as possible.

Each subscription group connection operates independently in a separate background thread. They do, however, share the workers poll for processing.

All the subscription groups define within a single consumer group will operate within the same consumer group.

class KarafkaApp < Karafka::App
  setup do |config|
    # ...
  end

  routes.draw do
    subscription_group 'a' do
      topic :A do
        consumer ConsumerA
      end

      topic :B do
        consumer ConsumerB
      end

      topic :D do
        consumer ConsumerD
      end
    end

    subscription_group 'b' do
      topic :C do
        consumer ConsumerC
      end
    end
  end
end

You can read more about the concurrency implications of using subscription groups here.

Overriding Defaults

Almost all the default settings configured can be changed on either on the topic level. This means that you can provide each topic with some details in case you need a non-standard way of doing things (for example, you need batch consuming only for a single topic).

Topic level options

There are several options you can set inside of the topic block. All of them except consumer are optional. Here are the most important once:

Option	Value type	Description
active	Boolean	Set to false if you want to have the given topic defined but not consumed. Helpful when working with topics via admin API
consumer	Class	Name of a consumer class that we want to use to consume messages from a given topic
deserializer	Class	Name of a deserializer that we want to use to deserialize the incoming data
manual_offset_management	Boolean	Should Karafka automatically mark messages as consumed or not
long_running_job	Boolean	Converts this topic consumer into a job that can run longer than max.poll.interval.ms
virtual_partitions	Hash	Allows you to parallelize the processing of data from a single partition.
dead_letter_queue	Hash	Provides a systematic way of dealing with persistent consumption errors.

class KarafkaApp < Karafka::App
  setup do |config|
    # ...
  end

  routes.draw do
    consumer_group :videos_consumer do
      topic :binary_video_details do
        consumer Videos::DetailsConsumer
        deserializer Serialization::Binary::Deserializer.new
      end

      topic :new_videos do
        consumer Videos::NewVideosConsumer
      end
    end

    topic :events do
      # Set to false because not for consumption.
      # Only inspection via admin API.
      active false
      deserializer EventsDeserializer.new
    end
  end
end