Home
About Karafka
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Has a built-in Web UI providing a convenient way to monitor and manage Karafka-based applications.
- Supports parallel processing in multiple threads (also for a single topic partition work)
- Automatically integrates with Ruby on Rails
- Has ActiveJob backend support (including ordered jobs)
- Has a seamless Dead Letter Queue functionality built-in
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box StatsD/DataDog monitoring with a dashboard template.
Basics
- Getting Started
- Components
- Configuration
- Setting up Kafka
- Producing messages
- Consuming messages
- Web UI
- Testing
- FAQ
Web UI
Production usage
- Development vs Production
- Signals and states
- Deployment
- Monitoring and logging
- Error handling and back off policy
Advanced
- Routing
- Active Job
- Dead Letter Queue
- Auto reload of code changes in development
- Topics management and administration
- CLI
- Integrating with Ruby on Rails and other frameworks
- Concurrency and multithreading
- Deserialization
- Offset management (checkpointing)
- Pausing and throttling
- Consumer mappers
- WaterDrop reconfiguration
- Exit codes
- Embedding
- Env Variables
- Articles and other references
- Versions Lifecycle and EOL
- Problems, Troubleshooting and Debugging
Karafka Pro
- Build vs. Buy
- Purchase Karafka Pro
- Getting Started
- Rotating credentials
- Pro FAQ
- Pro Support
- Security
Features and enhancements
- Features list
- Features Compatibility
- Virtual Partitions
- Long-Running Jobs
- Enhanced Dead Letter Queue
- Enhanced Active Job
- Enhanced Scheduler
- Messages At Rest Encryption
- Enhanced Web UI
Upgrade notes
It is recommended to do one major upgrade at a time.