Enhanced Inline Insights
Inline Insights is a feature of Karafka that provides a way to enhance your data processing capabilities by allowing your consumers to adjust their actions based on real-time metrics.
Not all applications and services need insights at all times. However, for critical applications, these insights can be crucial. With the
required option, developers can ensure that their consumers only process the data when the associated insights metrics are available.
class KarafkaApp < Karafka::App setup do |config| # ... end routes.draw do topic :orders_states do consumer OrdersStatesConsumer # Ensures that `#insights` are always present during processing inline_insights(required: true) end end end
How does it work?
required option is activated in Karafka Pro's Inline Insights, it automatically verifies the data processing workflow. Karafka immediately checks to ascertain if the corresponding insights are available after receiving messages for a particular topic partition. If these insights are unavailable, Karafka temporarily pauses processing for that specific topic partition rather than proceeding with potentially incomplete data processing. This pause continues until the necessary
#insights become available. With this feature in place, developers are spared the chore of manually verifying the presence of insights using the
#insights? method, as the system intelligently and autonomously ensures that every piece of data is paired with its analytical insights before processing.
In sum, the
required option in Karafka Pro's Inline Insights is an invaluable asset for developers aiming to bolster and assure the reliability of their metrics availability.