Enhanced Reliability

Enhanced Execution Stability

Karafka Pro comes with Enhanced Execution Stability improvements that significantly reduce the risks associated with involuntary revocations, which occur when partition ownership is transferred among consumers.

This mechanism minimizes the number of jobs in flight during such reassignments, thereby preventing potential race conditions and ensuring a stable, predictable execution environment, even during high-volume data processing and dynamic consumer scenarios.

Enhanced Scheduler

Karafka Pro comes shipped with an Enhanced Scheduler.

The default scheduler schedules work in a FIFO (First-In, First-Out) order.

The Enhanced Scheduler uses a non-preemptive LJF (Longest Job First) algorithm.

This scheduler is designed to optimize execution times, especially on jobs that perform IO operations. When computing the order, it considers the potential time cost of executing jobs based on the in-process p95 time and number of messages.

For IO intense jobs, where the number of jobs exceeds the number of threads, this can provide gains up to 20%.

Enhanced Memory Utilization

Karafka Pro provides the Cleaner API. This feature is specifically designed to optimize memory management for message payloads, especially those exceeding 10KB.

This mechanism allows users to achieve memory savings of up to 80%, ensuring that applications run more efficiently and with reduced risk. Moreover, the memory usage patterns become substantially more stable with the Cleaner API, offering robust protection against unexpected out-of-memory exceptions.

For a deeper dive into the nuances and technicalities of this feature, please refer to the dedicated Cleaner API documentation.

Enhanced Supervision in Swarm Mode

Karafka Pro enhances the stability and performance of distributed systems with its Enhanced Supervision for Swarm Nodes. This feature targets the detection and graceful management of hanging or bloated worker nodes within the swarm. By monitoring and identifying nodes that are either unresponsive or consuming excessive memory, Karafka Pro ensures that such nodes are shut down and restarted gracefully. This process not only preserves the integrity of ongoing tasks but also optimizes system resources by preventing memory leaks and ensuring efficient allocation of processing power.

For more detailed information on how Enhanced Supervision for Swarm Nodes works and how to implement it in your Karafka Pro setup, please refer to the Enhanced Swarm documentation.