Skip to content

Upgrading to WaterDrop 2.10

No Action Needed For Most Users

Unless you have custom instrumentation that subscribes to the statistics.emitted event on your WaterDrop producers, there is nothing you need to change when upgrading to WaterDrop 2.10. The built-in Karafka Web UI and Datadog listeners continue to work out of the box when they are wired up before first producer use, which is the standard setup. If you do not use statistics.emitted at all, you can safely upgrade without any code changes.

Tombstone API

WaterDrop 2.10 introduces a dedicated tombstone API for producing tombstone records (nil-payload messages used for key deletion in log-compacted topics). The following methods are available on both producers and variants:

  • #tombstone_sync(message) - Produces a tombstone and waits for delivery confirmation
  • #tombstone_async(message) - Produces a tombstone asynchronously
  • #tombstone_many_sync(messages) - Produces multiple tombstones synchronously
  • #tombstone_many_async(messages) - Produces multiple tombstones asynchronously

Each tombstone message requires a topic, a key (non-empty String), and a partition (non-negative Integer). The payload is always set to nil. See the Usage documentation for details and examples.

Statistics Emission Is Now Opt-In At Client Build Time

In previous versions of WaterDrop, librdkafka statistics were always collected on the interval defined by statistics.interval.ms (5 seconds by default in WaterDrop), even when no one was listening. The statistics payload was parsed from JSON, materialized as a Ruby hash, and passed through the statistics decorator on every tick - regardless of whether any subscriber was going to consume the result.

As of WaterDrop 2.10, statistics emission is opt-in and is decided once, when the underlying rdkafka client is lazily constructed for a given producer:

  • If at least one listener is subscribed to statistics.emitted at the moment the rdkafka client is built, statistics work exactly as before.
  • If no listener is subscribed at that moment, WaterDrop forces statistics.interval.ms to 0 regardless of what you configured, does not register the statistics callback on the rdkafka client, and skips all the associated work (JSON parsing, hash materialization, decoration) on the hot path.

This change removes a substantial amount of allocations and CPU work from producers that never needed statistics in the first place, while keeping the behavior fully transparent for producers that do.

What This Means In Practice

If your application already subscribes listeners to statistics.emitted during startup (for example through the built-in Karafka Web UI instrumentation, the Datadog metrics listener, or your own custom instrumentation), and those subscriptions happen before the first message is produced, you do not need to change anything. The statistics callback will be registered as usual and your listeners will keep receiving events.

If you are not using statistics.emitted at all, you will simply benefit from reduced overhead without any code changes.

You only need to take action if you are subscribing to statistics.emitted lazily, that is, after a producer has already been used and its underlying rdkafka client has been built. Late subscriptions of this kind were never a recommended pattern, but previous versions of WaterDrop would silently start delivering events on the next tick. In 2.10 this will no longer work, and an error will be raised as described in the next section.

Making Sure Your Listeners Are Subscribed In Time

To guarantee that statistics are enabled, subscribe your listeners during your application's startup sequence, before any producer is used:

producer = WaterDrop::Producer.new do |config|
  config.deliver = true
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
end

# Subscribe BEFORE the first produce call / before the rdkafka client is built
producer.monitor.subscribe('statistics.emitted') do |event|
  MyMetricsBackend.report(event[:statistics])
end

# First use - this is where the underlying rdkafka client is constructed
# and where WaterDrop decides whether statistics are enabled for its lifetime.
producer.produce_sync(topic: 'events', payload: 'hello')

The same applies to listener objects that respond to on_statistics_emitted:

producer.monitor.subscribe(MyStatisticsListener.new)

producer.produce_sync(topic: 'events', payload: 'hello')

WaterDrop::Errors::StatisticsNotEnabledError On Late Subscriptions

Previously, if you subscribed to statistics.emitted after the rdkafka client had already been built without statistics enabled, the subscription would succeed but no events would ever be delivered. This "silent nothing" failure mode was easy to miss and hard to debug.

In WaterDrop 2.10, attempting to subscribe to statistics.emitted on a monitor whose underlying client was built without statistics enabled raises WaterDrop::Errors::StatisticsNotEnabledError immediately. This applies both to block-based subscriptions and to listener objects that respond to on_statistics_emitted.

producer = WaterDrop::Producer.new do |config|
  config.deliver = true
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
end

# Nothing subscribed to statistics.emitted yet, so when the client is built
# below, statistics are disabled for this producer's lifetime.
producer.produce_sync(topic: 'events', payload: 'hello')

# In 2.10 this raises WaterDrop::Errors::StatisticsNotEnabledError
producer.monitor.subscribe('statistics.emitted') do |event|
  MyMetricsBackend.report(event[:statistics])
end

The fix is to move the subscription earlier in the boot flow, before the first use of the producer, as shown in the previous section.

Why This Is Raised

The error is intentional and actionable: it pinpoints exactly where the timing mistake happens instead of letting your instrumentation go silently dark. If you hit this error after upgrading, it means that in previous versions your subscription was already non-functional - the events were being emitted by librdkafka but your listener was registered too late to matter in a meaningful way for the first producer uses, and with 2.10 it would not be wired up at all. Fixing the subscription order restores the intended behavior and brings full statistics back to your producer.

Last modified: 2026-04-14 15:44:51