Upgrading to WaterDrop 2.9

Default Polling Mode Changed from :thread to :fd

WaterDrop 2.9 switches the default polling mode from :thread to :fd. The FD mode uses a single Ruby thread with IO.select for efficient multiplexing, providing higher throughput, lower memory usage, and fewer threads compared to the thread-based mode.

If you experience any issues with the new default, you can revert to the previous behavior:

producer = WaterDrop::Producer.new

producer.setup do |config|
  config.deliver = true
  config.polling.mode = :thread
  # ...
end

Experiencing Issues?

If you experience any issues with the :fd polling mode, please open an issue on the WaterDrop GitHub repository. Your feedback will help improve the FD mode before the :thread mode is removed.

Deprecation Timeline

The :thread polling mode will be deprecated in WaterDrop 2.10 and removed in 2.11. If you need to revert, plan to migrate to :fd before 2.11.

Statistics Decorator Scope Reduced

The built-in statistics decorator now only decorates keys used by the Karafka official metrics listeners (tx, txretries, txerrs, rxerrs) and skips unused subtrees (topics, broker window stats). This reduces decoration cost by ~425x on large clusters.

If you rely on other _d or _fd delta keys in custom instrumentation, you need to provide a custom decorator:

producer = WaterDrop::Producer.new

producer.setup do |config|
  config.deliver = true

  # Provide a custom decorator that includes the keys you need
  config.statistics_decorator = ::Karafka::Core::Monitoring::StatisticsDecorator.new(
    only_keys: %w[tx txretries txerrs rxerrs your_custom_key],
    excluded_keys: %w[]
  )
  # ...
end

If you do not use custom statistics instrumentation beyond the built-in listeners, no action is needed.

Default Timeout Values Increased 3x

Default timeout values have been upscaled 3x, closer to librdkafka defaults, to prevent intermediate timeouts during node recovery:

Config	Previous Default	New Default
kafka message.timeout.ms	50000 ms (50 seconds)	150000 ms (150 seconds)
kafka transaction.timeout.ms	55000 ms (55 seconds)	165000 ms (165 seconds)
root max_wait_timeout	60000 ms (60 seconds)	180000 ms (180 seconds)

If you want to retain previous timeout behavior, set them explicitly:

producer = WaterDrop::Producer.new

producer.setup do |config|
  config.deliver = true
  config.max_wait_timeout = 60_000

  config.kafka = {
    'bootstrap.servers': 'localhost:9092',
    'message.timeout.ms': 50_000,
    'transaction.timeout.ms': 55_000
  }
end

These higher defaults provide more resilience during broker recovery scenarios and reduce the likelihood of premature timeout errors in production environments.

---

Last modified: 2026-04-08 11:25:57