Skip to content

Assignments Tracking

The Karafka Assignments Tracker is a feature designed to keep track of the active assignments within a Karafka process. It serves as a tool for understanding the current assignments of topics partitions from a Kafka cluster. This tracker is handy in environments with dynamic partition reassignments, such as those involving multiple consumer groups or subscription groups.

Usage

To access your process's current assignments, you only need to execute Karafka::App.assignments method from any place in the process. This thread-safe method will return a Hash object where keys correspond to the assigned topics and values are arrays with IDs of the partitions assigned for a corresponding topic.

It is worth pointing out that the topics are Karafka::Routing::Topic objects and not just topic names. This is done that way so the system supports providing you insights about the same topic if it is being consumed in multiple consumer groups simultaneously. You can use it from any place in your code.

def consume
  Karafka::App.assignments.each do |topic, partitions|
    puts "In #{topic.name} I'm currently assigned to partitions: #{partitions.join(', ')}"
  end
end

Example Use Cases

  • Building Assignments Aware Schedulers: A custom scheduler that takes into account assignments can significantly improve efficiency and performance. Such a scheduler optimizes resource utilization and reduces latency by making intelligent scheduling decisions based on real-time cluster state.

  • Dynamic Resources Allocation: In scenarios where workloads fluctuate, the tracker can dynamically redistribute tasks based on the current partition assignments or apply specific processing strategies, ensuring optimal resource utilization.

  • Monitoring and Alerting: Developers can monitor partition assignments to detect anomalies (e.g., imbalanced loads, partition loss) and trigger alerts for proactive issue resolution.

  • Consumer Group Management: For applications involving multiple consumer groups, the tracker provides visibility into how partitions are distributed across different groups, aiding debugging and performance tuning.

  • Performance Tuning: Observing partition traffic can help identify hotspots or bottlenecks in data flow, enabling fine-tuning partition sizes or numbers to optimize performance.

  • Dynamic Consumer Scaling: For systems that dynamically scale consumers based on load, knowledge of partition assignments is necessary to redistribute partitions among the new set of consumers efficiently.