# WaterDrop Variants WaterDrop variants can manage different configuration settings per topic using the same Kafka producer with shared TCP connections. This feature allows optimal utilization of a producer's TCP connections while enabling tailored dispatch requirements for individual topics. Variants are beneficial when working with topics with varying levels of importance or different throughput and latency requirements. ## WaterDrop Variants / Creating and Using Variants To leverage variants in WaterDrop, you initialize a standard producer with default settings that apply broadly to all topics for which you intend to produce messages. Then, you can create variants of this producer with configurations specific to particular topics. These variants allow for topic-specific adjustments without needing multiple producer instances, thus conserving system resources and maintaining high performance. Variants are created using the `#with` and `#variant` methods. It is critical in enabling topic-specific configurations through variants while using a single producer instance. The `#with` and `#variant` methods are designed to accept two types of arguments: - `max_wait_timeout`: This is a root-scoped setting. - `topic_config` hash: This is where all topic-specific configurations are defined. Attributes placed inside the `topic_config` hash during variant creation are referred to as `topic_config` scoped. Conversely, settings like `max_wait_timeout`, which reside outside the `topic_config hash`, are considered root-scoped. Here's a simple example to demonstrate how to define and use variants with WaterDrop: ```ruby # Initialize the main producer with common settings producer = WaterDrop::Producer.new do |config| config.kafka = { 'bootstrap.servers': 'localhost:9092', 'acks': '2' # Default acknowledgment setting for medium-importance topics } end # Create variants with specific settings low_importance = producer.with(topic_config: { acks: 1 }) high_importance = producer.with(topic_config: { acks: 'all' }) # Use variants like regular producers low_importance.produce_async(topic: 'low_priority_events', payload: event.to_json) high_importance.produce_async(topic: 'critical_events', payload: event.to_json) ``` ## WaterDrop Variants / Configurable Settings Variants allow you to modify several Kafka and producer-specific settings to better suit the characteristics of different topics:
Scope Attribute Description
root max_wait_timeout Controls how long the producer waits for the dispatch result before raising an error.
topic_config acks
request.required.acks 		Determines the number of broker acknowledgments required before considering a message delivery successful.
topic_config compression.codec
compression.type 		Specifies the type of codec used for compression (e.g., none, gzip, snappy, lz4, zstd).
topic_config compression.level Determines the compression level for the selected codec, affecting both the compression ratio and performance.
topic_config delivery.timeout.ms
message.timeout.ms 		Limits the time a produced message waits for successful delivery. A time of 0 is infinite.
topic_config partitioner Defines partitioner to use for distribution across partitions within a topic.
topic_config request.timeout.ms The ack timeout of the producer request in milliseconds.
!!! info "Additional Configuration Attributes Details" For a more comprehensive list of configuration settings supported by librdkafka, please visit the [Librdkafka Configuration](https://karafka.io/docs/Librdkafka-Configuration.md) page. ## WaterDrop Variants / Inspecting the Active Variant When you dispatch through a variant, the variant in effect is tracked per dispatch and is fiber-local. The `#current_variant` method lets you read the variant that is active for the dispatch currently running on the present fiber. When called outside of a variant-wrapped dispatch, it returns the producer's default variant. This is primarily useful for [middleware](https://karafka.io/docs/WaterDrop-Middleware.md) and instrumentation listeners that run synchronously within a dispatch and need to read the effective per-dispatch settings, such as `topic_config`, `max_wait_timeout`, or whether the default variant is in use via `default?`: ```ruby high_importance = producer.with(topic_config: { acks: 'all' }) # Inside synchronous middleware or instrumentation running during the dispatch: variant = producer.current_variant variant.default? # => false when dispatching through a variant variant.topic_config # => { acks: 'all' } variant.max_wait_timeout # => the variant's root-scoped timeout high_importance.produce_async(topic: 'critical_events', payload: event.to_json) ``` !!! warning "Fiber-Local and Scoped to a Single Dispatch" `#current_variant` is fiber-local and scoped to a single dispatch; it does not represent a producer-wide setting. It is not meaningful from asynchronous delivery callbacks (such as `message.acknowledged`), which run on the poller thread - a different fiber - where it always returns the default variant rather than the variant the acknowledged message was dispatched with. ## WaterDrop Variants / Edge-Cases and Details When using variants in WaterDrop, there are specific edge cases and operational nuances that you should be aware of to ensure optimal performance and behavior: - **Buffering Behavior Across Variants**: It is crucial to understand that while `topic_config` specific settings are preserved per message, the `max_wait_timeout` applied during the flush operation will correspond to the variant that initiates the flushing. This means that messages from other variants that were buffered may be dispatched using the `max_wait_timeout` of the variant currently flushing the data. Since variants share a single producer buffer, this can affect how messages are processed. - **Inconclusive Error Messages**: Redefining `max_wait_timeout` without aligning it with other librdkafka settings can lead to inconclusive error. This issue arises because the timeout settings may not synchronize well with other operational parameters, potentially leading to errors that are difficult to diagnose. For a deeper understanding of this issue and how it might affect your Kafka operations, refer to the [Error Handling](https://karafka.io/docs/WaterDrop-Error-Handling.md) documentation. - **Immutable acks for Idempotent and Transactional Producers**: When working with idempotent or transactional producers, it is important to note that the `acks` setting is immutable and automatically set to `all`. This configuration cannot be altered through variants, as ensuring exactly-once semantics requires a fixed acknowledgment policy. Attempting to change the acks setting for these producers will result in an error. These details are critical in effectively managing and troubleshooting your Kafka message production environment, especially when utilizing the flexibility of variants for different topic configurations. ## WaterDrop Variants / Conclusion Variants address the need for dynamic, topic-specific configurations in applications interacting with Kafka. By enabling variations per topic within a single producer, WaterDrop helps streamline resource usage and enhance message dispatch efficiency, making it an essential tool for sophisticated Kafka-based messaging systems. ## WaterDrop Variants / See Also - [Multi-Cluster Setup](https://karafka.io/docs/Infrastructure-Multi-Cluster-Setup.md) - For managing multiple Kafka clusters - [Producing Messages](https://karafka.io/docs/Basics-Producing-Messages.md) - For message production techniques and patterns --- *Last modified: 2026-06-15 19:42:51*