Home
About Karafka
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Supports parallel processing in multiple threads (also for a single topic partition work)
- Has ActiveJob backend support (including ordered jobs)
- Automatically integrates with Ruby on Rails
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box StatsD/DataDog monitoring with a dashboard template.
Note: All of the documentation here refers to Karafka
2.0.0or higher. If you are looking for the documentation for Karafka1.4, please click here.
Basics
- Getting Started
- The Basics
- Configuration
- Setting up Kafka
- Producing messages
- Consuming messages
- Testing
Production usage
- Development vs Production
- Signals
- Deployment (systemd, Docker, AWS MSK)
- Monitoring and logging
- Error handling and back off policy
- Administration (topics management and cluster info)
Advanced
- Routing
- Active Job
- Auto reload of code changes in development
- CLI
- Integrating with Ruby on Rails and other frameworks
- Concurrency and multithreading
- Deserialization
- Manual offset management (checkpointing)
- Consumer mappers
- WaterDrop reconfiguration
- Exit codes
- Articles and other references
- FAQ
- Versions Lifecycle and EOL
- Problems, Troubleshooting and Debugging