Class: Karafka::Routing::SubscriptionGroupsBuilder

Inherits:
Object
  • Object
show all
Defined in:
lib/karafka/routing/subscription_groups_builder.rb

Overview

rdkafka allows us to group topics subscriptions when they have same settings. This builder groups topics from a single consumer group into subscription groups that can be subscribed with one rdkafka connection. This way we save resources as having several rdkafka consumers under the hood is not the cheapest thing in a bigger system.

In general, if we can, we try to subscribe to as many topics with one rdkafka connection as possible, but if not possible, we divide.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ SubscriptionGroupsBuilder

Initializes the subscription groups builder



28
29
30
 
# File 'lib/karafka/routing/subscription_groups_builder.rb', line 28

def initialize
  @position = -1
end

Instance Method Details

#call(topics) ⇒ Array<SubscriptionGroup>

Returns all subscription groups we need in separate threads.

Parameters:

Returns:



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
# File 'lib/karafka/routing/subscription_groups_builder.rb', line 35

def call(topics)
  topics
    .map { |topic| [checksum(topic), topic] }
    .group_by(&:first)
    .values
    .map { |value| value.map(&:last) }
    .flat_map { |value| expand(value) }
    .map { |grouped_topics| SubscriptionGroup.new(@position += 1, grouped_topics) }
    .tap do |subscription_groups|
      subscription_groups.each do |subscription_group|
        subscription_group.topics.each do |topic|
          topic.subscription_group = subscription_group
        end
      end
    end
end