Home
Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
- Has a built-in Web UI providing a convenient way to monitor and manage Karafka-based applications.
- Supports parallel processing in multiple threads (also for a single topic partition work) and processes.
- Automatically integrates with Ruby on Rails
- Has ActiveJob backend support (including ordered jobs)
- Has a seamless Dead Letter Queue functionality built-in
- Supports in-development code reloading
- Is powered by librdkafka (the Apache Kafka C/C++ client library)
- Has an out-of the box AppSignal and StatsD/DataDog monitoring with dashboard templates.
Basics
- Getting Started
- Components
- Configuration
- Producing Messages
- Consuming Messages
- Web UI
- Testing
- FAQ
- Support
Web UI
- About
- Getting Started
- Configuration
- Transactions
- Features
- Tagging
- Multi App Mode
- Single Process Setup
- Development vs Production
- Data Management
- Operational Cost Breakdown
- Components
WaterDrop
- About
- Getting Started
- Configuration
- Usage
- Error Handling
- Monitoring and Logging
- Transactions
- Testing
- Middleware
- Labeling
- Variants
- Custom Partitioners
- Idempotence and Acknowledgements
Production Usage
- Development vs Production
- Signals and states
- Deployment
- Monitoring and Logging
- Error Handling and back off policy
Advanced
- Upgrading
- Routing
- Active Job
- Dead Letter Queue
- Declarative Topics
- Admin API
- ACLs API
- Configs API
- Auto reload of code changes in development
- CLI
- Integrating with Ruby on Rails and other frameworks
- Concurrency and multithreading
- Deserialization
- Offset management (checkpointing)
- Pausing, Seeking and Rate-Limiting
- Inline Insights
- WaterDrop reconfiguration
- Exit codes
- Embedding
- Swarm / Multi Process
- Multi-Cluster Setup
- Env Variables
- Assignments Tracking
- Active Record Connections Management
- Forking
- Resources Management
- Latency and Throughput
- Articles and other references
- Versions Lifecycle and EOL
- Problems, Troubleshooting and Debugging
- Software Bill of Materials (SBOM)
Karafka Pro
- Build vs. Buy
- Purchase Karafka Pro
- Getting Started
- Rotating credentials
- Pro FAQ
- Pro Support
- Security
- HIPAA, PHI, PII Support
- Enterprise
- Enterprise Workshop Session
- Enterprise License Setup
Features and Enhancements
- Features List
- Features Compatibility
- Transactions
- Offset Metadata Storage
- Virtual Partitions
- Delayed Topics
- Long-Running Jobs
- Non-Blocking Jobs
- Adaptive Iterator
- Periodic Jobs
- Expiring Messages
- Routing Patterns
- Rate Limiting
- Filtering API
- Scheduling API
- Iterator API
- Cleaner API
- Granular Backoffs
- Direct Assignments
- Multiplexing
- Piping
- Recurring Tasks
- Scheduled Messages
- Messages At Rest Encryption
- Enhanced Swarm / Multi Process
- Enhanced Dead Letter Queue
- Enhanced Active Job
- Enhanced Reliability
- Enhanced Inline Insights
- Enhanced Web UI
Librdkafka
Kafka
Upgrade Notes
It is recommended to do one major upgrade at a time.
- Upgrading to 2.4
- Upgrading to 2.3
- Upgrading to 2.2
- Upgrading to 2.1
- Upgrading to 2.0
- Upgrading to 1.4
- Upgrading to 1.3
- Upgrading to 1.2
- Upgrading to 1.1