Karafka is a framework for producing and consuming messages using Kafka. It is built out of a few components:
- Karafka (Consumer) - responsible for consumption messages from Kafka
- WaterDrop (Producer) - messages producer integrated with Karafka out of the box
- Karafka-Web (UI) - User interface for the Karafka framework
- Karafka-Rdkafka (Driver) - A customized fork of
rdkafka-rubyproviding additional functionalities and extended stability
The producer can run in any Ruby process and allows you to produce messages to Kafka.
# Fast, non-blocking, recommended Karafka.producer.produce_async(topic: 'events', payload: Events.last.to_json) # Slower, blocking Karafka.producer.produce_sync(topic: 'events', payload: Events.last.to_json) Karafka.producer.class #=> WaterDrop::Producer
Karafka uses WaterDrop to produce messages. It is a standalone Karafka framework component that can also be used in applications that only produce messages.
Please refer to WaterDrop README for more details.
Consumer / server
Each Karafka server process pulls messages from Kafka topics and processes them. The server will instantiate consumers and deliver you messages from desired topics. Everything else is up to your code.
Example consumer printing payload of fetched messages:
class PrintingConsumer < ApplicationConsumer def consume messages.each do |message| puts message.payload end end end
Karafka Web UI is a user interface for the Karafka framework. The Web UI provides a convenient way for developers to monitor and manage their Karafka-based applications, without the need to use the command line or third party software. It does not require any additional database beyond Kafka itself.
Karafka uses its fork of the
rdkafka-ruby. It is done to ensure that each release is complete, stable, and tested against the Karafka ecosystem. Providing our driver layer ensures that upgrades are safe and reliable.