Configuration
Karafka contains multiple configuration options. To keep everything organized, all the configuration options were divided into two groups:
-
karafka
options - options directly related to the Karafka framework and its components. -
librdkafka
options - options related to librdkafka
To apply all those configuration options, you need to use the #setup
method from the Karafka::App
class:
class KarafkaApp < Karafka::App
setup do |config|
config.client_id = 'my_application'
# librdkafka configuration options need to be set as symbol values
config.kafka = {
'bootstrap.servers': '127.0.0.1:9092'
}
end
end
Note: Karafka allows you to redefine some of the settings per each topic, which means that you can have a specific custom configuration that might differ from the default one configured at the app level. This allows you for example, to connect to multiple Kafka clusters.
Note: kafka client.id
is a string passed to the server when making requests. This is to track the source of requests beyond just IP/port by allowing a logical application name to be included in server-side request logging. Therefore the client_id
should be shared across multiple instances in a cluster or horizontally scaled application but distinct for each application.
Karafka configuration options
A list of all the karafka configuration options with their details and defaults can be found here.
librdkafka driver configuration options
A list of all the configuration options related to librdkafka
with their details and defaults can be found here.
External components configurators
For additional setup and/or configuration tasks, you can use the app.initialized
event hook. It is executed once per process, right after all the framework components are ready (including those dynamically built). It can be used, for example, to configure some external components that need to be based on Karafka internal settings.
class KarafkaApp < Karafka::App
# Setup and other things...
# Once everything is loaded and done, assign Karafka app logger as a MyComponent logger
# @note This example does not use config details, but you can use all the config values
# to setup your external components
monitor.subscribe('app.initialized') do
MyComponent::Logging.logger = Karafka::App.logger
end
end
# Or if you prefer, you can do it from the outside of the app
Karafka.monitor.subscribe('app.initialized') do
MyComponent::Logging.logger = Karafka::App.logger
end
Environment variables settings
There are several env settings you can use with Karafka. They are described under the Env Variables section of this Wiki.
Messages compression
Kafka lets you compress your messages as they travel over the wire. By default, producer messages are sent uncompressed.
Karafka producer (WaterDrop) supports following compression types:
gzip
zstd
lz4
snappy
You can enable the compression by using the compression.codec
and compression.level
settings:
setup_karafka do |config|
config.kafka = {
# Other kafka settings...
'compression.codec': 'gzip',
'compression.level': '12'
}
end
Note: In order to use zstd
, you need to install libzstd-dev
:
apt-get install -y libzstd-dev