Features Compatibility
Karafka provides several features that can work together. Below you can find a comprehensive description of combinations that are and are not allowed.
Long-Running Jobs
Usage with Virtual Partitions
Long-Running Jobs work together with Virtual Partitions. All the Virtual Partitions will respond to #revoked?
if the partition is lost.
There is only one thing you need to keep in mind:
It is not allowed to use manual offset management with Virtual Partitions unless Virtual Partitions operate in the collapsed
mode. Hence you need to set shutdown_timeout
to a value that will compensate for that.
Using with Enhanced Dead Letter Queue
Long-Running Jobs work with the Enhanced Dead Letter Queue feature without additional changes.
Virtual Partitions
Usage with Long-Running Jobs
Virtual Partitions can be used with Long-Running Jobs. There are no special procedures.
routes.draw do
topic :orders_states do
consumer OrdersStatesConsumer
long_running_job true
virtual_partitions(
partitioner: ->(message) { message.headers['order_id'] }
)
end
end
Usage with Enhanced Active Job
Virtual Partitions can be used with Active Job without any limitations. The only thing worth keeping in mind is that the message payload for Active Job contains serialized job details and should not be deserialized in the partitioner.
The recommended approach is to use the Enhanced Active Job headers support to add a key that can be used for partitioning:
class Job < ActiveJob::Base
queue_as :jobs
karafka_options(
dispatch_method: :produce_async,
partitioner: ->(job) { job.arguments.first[0] }
)
end
class KarafkaApp < Karafka::App
routes.draw do
active_job_topic :jobs do
virtual_partitions(
partitioner: ->(job) { job.key }
)
end
end
end
Using with Enhanced Dead Letter Queue
Virtual Partitions can be used together with the Dead Letter Queue. This can be done due to Virtual Partitions' ability to collapse upon errors.
The only limitation when combining Virtual Partitions with the Dead Letter Queue is the minimum number of retries. It needs to be set to at least 1
:
routes.draw do
topic :orders_states do
consumer OrdersStatesConsumer
virtual_partitions(
partitioner: ->(message) { message.headers['order_id'] }
)
dead_letter_queue(
topic: 'dead_messages',
# Minimum one retry because VPs needs to switch to the collapsed mode
max_retries: 1
)
end
end
Enhanced ActiveJob
Usage with Virtual Partitions
For the Enhanced Active Job adapter to work with Virtual Partitions, you need to update your karafka.rb
and use the virtual_partitions
settings in the Active Job topic section:
class KarafkaApp < Karafka::App
setup do |config|
# ...
end
routes.draw do
active_job_topic :default do
virtual_partitions(
partitioner: ->(job) { job.key }
)
end
end
end
Your partitioner will accept the wrapping message of a job. Since the payload is the serialized job, you will not have access to the payload. We recommend using either key or headers metadata for virtual partitioning.
Please keep in mind that with Virtual Partitions, the offset will be committed after all the Virtual Partitions work is done. There is no "per job" marking as processed.
Usage with Long-Running Jobs
For the Enhanced Active Job adapter to work with Long-Running Jobs, you need to update your karafka.rb
and use the long_running_job
setting in the Active Job topic section:
class KarafkaApp < Karafka::App
setup do |config|
# ...
end
routes.draw do
active_job_topic :default do
long_running_job true
end
end
end
Using with Enhanced Dead Letter Queue
The Active Job Adapter can be used with the Enhanced Dead Letter Queue feature.
Messages At Rest Encryption
Messages at rest encryption works with all other Pro features without any changes required.