Home
About Karafka
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Supports parallel processing in multiple threads (also for a single topic partition work)
- Has ActiveJob backend support (including ordered jobs)
- Automatically integrates with Ruby on Rails
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box StatsD/DataDog monitoring with a dashboard template.
Note: All of the documentation here refers to Karafka 2.0.16 or higher. If you are looking for the documentation for Karafka 1.4, please click here.
Basics
- Getting Started
- Components
- Configuration
- Setting up Kafka
- Producing messages
- Consuming messages
- Testing
Production usage
- Development vs Production
- Signals
- Deployment
- Monitoring and logging
- Error handling and back off policy
- Administration (topics management and cluster info)
Advanced
- Routing
- Active Job
- Dead Letter Queue
- Auto reload of code changes in development
- CLI
- Integrating with Ruby on Rails and other frameworks
- Concurrency and multithreading
- Deserialization
- Offset management (checkpointing)
- Pausing and throttling
- Consumer mappers
- WaterDrop reconfiguration
- Exit codes
- Embedding
- Articles and other references
- FAQ
- Versions Lifecycle and EOL
- Problems, Troubleshooting and Debugging
Karafka Pro
Features and enhancements
- Features Compatibility
- Virtual Partitions
- Long-Running Jobs
- Enhanced Dead Letter Queue
- Enhanced Active Job
- Enhanced Scheduler
- Messages At Rest Encryption
Upgrade notes
It is recommended to do one major upgrade at a time.