# Karafka Pro Features Compatibility Karafka provides several features that can work together. Unless explicitly stated otherwise, Karafka Pro features should work with each other without any limitations. When the routing validation rejects a combination, its section below says so. ## Karafka Pro Features Compatibility / Long Running Jobs + Virtual Partitions Long-Running Jobs work together with [Virtual Partitions](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Consumer-Groups-Virtual-Partitions.md). All the Virtual Partitions consumers will respond to `#revoked?` if the partition is lost, similar to regular consumers. ## Karafka Pro Features Compatibility / Enhanced Active Job + Virtual Partitions Virtual Partitions **can** be used with Active Job without any limitations. The only thing worth keeping in mind is that the message payload for Active Job contains serialized job details and should not be deserialized in the partitioner. The recommended approach is to use the Enhanced Active Job headers support to add a key that can be used for partitioning: ```ruby class Job < ActiveJob::Base queue_as :jobs karafka_options( dispatch_method: :produce_async, partitioner: ->(job) { job.arguments.first[0] } ) end class KarafkaApp < Karafka::App routes.draw do active_job_topic :jobs do virtual_partitions( partitioner: ->(job) { job.key } ) end end end ``` With Virtual Partitions, Karafka marks each job as consumed after it finishes and commits the highest offset up to which all the Virtual Partitions finished their jobs. ## Karafka Pro Features Compatibility / Enhanced Dead Letter Queue + Virtual Partitions Virtual Partitions can be used together with the Dead Letter Queue. This can be done due to Virtual Partitions' ability to collapse upon errors. The only limitation when combining Virtual Partitions with the Dead Letter Queue is the minimum number of retries. It needs to be set to at least `1`, and the routing validation rejects `max_retries: 0`: ```ruby routes.draw do topic :orders_states do consumer OrdersStatesConsumer virtual_partitions( partitioner: ->(message) { message.headers['order_id'] } ) dead_letter_queue( topic: 'dead_messages', # Minimum one retry because VPs needs to switch to the collapsed mode max_retries: 1 ) end end ``` ## Karafka Pro Features Compatibility / Virtual Partitions + Transactions Due to the Virtual Partitions' nature, message production transactions work entirely as expected. However, transactions involving offset storage operate in a simulated mode. This means that even if `#mark_as_consumed` is used within a transaction, it does not become part of the transaction itself. Instead, it is committed right after the transaction successfully ends. This creates an edge case: there could be inconsistencies if a consumer is killed or loses its assignment right after the Kafka transaction completes but before the consumer offset is sent to Kafka. This behavior aligns with the principles of the underlying Virtual Offset Management system. This system is crafted to handle offsets in a way distinct from Kafka's native offset handling due to the underlying parallelization process. As a result, certain operations, like `#mark_as_consumed`, are executed outside the main transaction scope, which is a direct consequence of the design and functionality of the Virtual Offset Management. The transactional behavior aligns with standard expectations in a collapsed virtual partition flow scenario. The associated offset is included in the transaction when a message is marked as consumed. ```ruby class VirtualPartitionedEventsConsumer < ApplicationConsumer def consume transaction do produce_async(topic: 'totals', payload: messages.payloads.sum(&:count).to_s) # if this topic uses virtual partitions this will NOT be part of the transaction and will be # executed right after the transaction has ended. # # In case `#collapsed?` would be true, this will behave like a regular transaction mark_as_consumed messages.last end end end ``` ## Karafka Pro Features Compatibility / Virtual Partitions + Offset Metadata Storage In Karafka's Virtual Partitions, the offset_metadata_strategy setting, configurable during routing, dictates whether the system should use the most recent (`:current`) or the exact (`:exact`) metadata associated with a materialized offset, a choice crucial for aligning processing logic with data consistency requirements. For detailed usage and configuration, refer to [this](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Consumer-Groups-Offset-Metadata-Storage.md#interaction-with-virtual-partitions) Karafka documentation section. ## Karafka Pro Features Compatibility / Routing Patterns + Dead Letter Queue While Karafka's Routing Patterns feature integrates with the Dead Letter Queue (DLQ) mechanism, developers are advised to exercise caution. Specifically, there is a potential issue where an imprecisely crafted regular expression could inadvertently match both primary and DLQ topics. This misconfiguration might result in messages from the DLQ being consumed in an unexpected loop, especially if the DLQ topic is not explicitly targeted for consumption. You can read more about this issue [here](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Routing-Patterns.md#dlq-accidental-auto-consumption). ## Karafka Pro Features Compatibility / Messages At Rest Encryption + Custom Headers Deserializer and Encryption When using Karafka's encryption features, encryption may not work as expected if you use a custom headers deserializer. Custom deserialization of headers can alter how encryption headers are processed, potentially leading to issues in correctly encrypting or decrypting messages. In cases where custom headers deserialization is necessary, it is recommended to consult with Karafka Pro support for guidance to ensure that encryption functionalities are properly integrated and maintained within your application. ## Karafka Pro Features Compatibility / Adaptive Iterator + Long-Running Jobs Adaptive Iterator should not be used with Long-Running Jobs because their operating principles are fundamentally incompatible. Long-Running Jobs have their own resource lifecycle management, while Adaptive Iterator dynamically adjusts batch sizes based on processing speed. This combination leads to conflicting resource allocation, disrupted timeout mechanisms, and inconsistent offset commitment patterns. The routing validation rejects this combination. ## Karafka Pro Features Compatibility / Adaptive Iterator + Virtual Partitions [Adaptive Iterator](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Consumer-Groups-Adaptive-Iterator.md) cannot be used with [Virtual Partitions](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Consumer-Groups-Virtual-Partitions.md). The routing validation rejects a topic that has both enabled. ## Karafka Pro Features Compatibility / Direct Assignments + Routing Patterns [Direct Assignments](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Direct-Assignments.md) cannot be used with [Routing Patterns](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Routing-Patterns.md). The routing validation rejects a topic that has both enabled. ## Karafka Pro Features Compatibility / Direct Assignments + Regular Subscriptions A single consumer group cannot mix topics that use [Direct Assignments](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Direct-Assignments.md) with topics that use regular subscriptions. Either all the topics of a consumer group use Direct Assignments, or none of them do. The routing validation rejects a consumer group that mixes them. Place the other topics in a separate consumer group. ## Karafka Pro Features Compatibility / Direct Assignments + Swarm When you use [Direct Assignments](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Direct-Assignments.md) with a list of partitions and allocate partitions to [Swarm](https://karafka.io/docs/Pro-Features-Compatibility/Pro-Enhanced-Swarm-Multi-Process.md#direct-assignments) nodes with a `nodes:` hash of partition arrays (for example `{ 0 => [0], 1 => [1, 2] }`), the routing validation checks that each assigned partition is allocated to exactly one node. It rejects: - a partition that is assigned but not allocated to any node, - a partition that is allocated to a node but not assigned, - a partition that is allocated to more than one node. The validation does not check a `nodes:` hash with Range values. --- *Last modified: 2026-10-31 20:24:56*