Getting Started
Karafka Web UI is a user interface for the Karafka framework. The Web UI provides a convenient way for developers to monitor and manage their Karafka-based applications, without the need to use the command line or third party software. It does not require any additional database beyond Kafka itself.
The interface, amongst others, displays:
- real-time aggregated metrics,
- real-time information on resources usage,
- errors details,
- performance statistics,
- trends
- allows for Kafka topics data exploration
- routing and system information
- status of Web UI integration within your application
Karafka Web UI is shipped as a separate gem with minimal dependencies.
To use it:
-
Make sure Apache Kafka is running. You can start it by following instructions from here.
-
Add Karafka Web-UI to your
Gemfile
:
bundle add karafka-web
- Run the following command to install the karafka-web in your project:
# For production you should add --replication-factor N
# Where N is the replication factor you want to use in your cluster
bundle exec karafka-web install
By default, Karafka uses three topics with the following names:
karafka_consumers_states
karafka_consumers_reports
karafka_errors
If you have the auto.create.topics.enable
set to false
or problems running the install command, create them manually. The recommended settings are as followed:
Topic name | Partitions count | Replication factor | Settings |
---|---|---|---|
karafka_consumers_states | 1 |
Aligned with your company policy | # We are only interested in the current materialized state cleanup.policy : compact |
karafka_consumers_reports | 1 |
Aligned with your company policy | # Keep reports for 7 days retention.ms : 7 * 86_400_000 |
karafka_errors | OSS: 1 Pro: as many as you need |
Aligned with your company policy | # Keep errors details for 3 months retention.ms : 3 * 31 * 86_400_000 |
Note: Karafka Web UI topics are not managed via the Ddeclarative topics API. It is done that way, so your destructive infrastructure changes do not break the Web UI. If you want to include their management in your declarative topic's code, you can do so by defining their configuration manually in your routing setup. Injected routing can be found here.
- Mount the Web interface in your Ruby on Rails application routing:
require 'karafka/web'
Rails.application.routes.draw do
# other routes...
mount Karafka::Web::App, at: '/karafka'
end
Or use it as a standalone Rack application by creating karafka_web.ru
rackup file with the following content:
# Require your application code here and then...
require_relative 'karafka.rb'
run Karafka::Web::App
- Enjoy Karafka Web UI.
If you do everything right, you should see this in your browser:
Multi-App / Multi-Tenant configuration
Karafka Web-UI can be configured to monitor and report data about many applications to a single dashboard.
Please visit the Web UI Multi-App documentation page to learn more about it.
Authentication
Karafka Web UI is "just" a Rack application, and it can be protected the same way as any other. For Ruby on Rails, in case you use Devise, you can just:
authenticate :user, lambda { |user| user.admin? } do
mount Karafka::Web::App, at: '/karafka'
end
or in case you want the HTTP Basic Auth, you can wrap the Web UI with the Basic Auth callable:
Rails.application.routes.draw do
with_dev_auth = lambda do |app|
Rack::Builder.new do
use Rack::Auth::Basic do |username, password|
username == 'username' && password == 'password'
end
run app
end
end
mount with_dev_auth.call(Karafka::Web::App), at: 'karafka'
end
You can find an explanation of how that works here.
Troubleshooting
As mentioned above, the initial setup requires you to run bundle exec karafka-web install
once so Karafka can build the initial data structures needed. Until this happens, upon accessing the Web UI, you may see a 404 error.
Before reporting an issue, please make sure that:
- You have visited the Karafka Web status page
- All the topics required by Karafka Web exist
- Use
bundle exec karafka-web install
to create missing topics - You have a working connection with your Kafka cluster
- The resource you requested exists
If you were looking for a given process or other real-time information, the state might have changed, and the information you were looking for may no longer exist.
Web UI status page
The Karafka Web UI status page allows you to check and troubleshoot the state of your Karafka Web UI integration with your application.
It can help you identify and mitigate problems that would cause the Web UI to malfunction or misbehave. If you see the 404
page or have issues with Karafka Web UI, this page is worth visiting.
You can read more about it here.
Resetting the Web UI state
If you want to reset the overall counters without removing the errors collection, you can run the bundle exec karafka-web install
again.
If you want to fully reset the Web UI state, you can run the bundle exec karafka-web reset
command. This command will remove all the Web UI topics and re-create them with an empty state.
Uninstalling the Web UI
If you want to remove Karafka Web UI, you need to:
- Remove all the Web app routes from your routing.
- Run
bundle exec karafka-web uninstall
. - Remove
karafka-web
from yourGemfile
.
And that is all.
statistics.interval.ms
alignment
Karafka uses its internal state knowledge and librdkafka
metrics to report the states. This means that the statistics.interval.ms
needs to be enabled and should match the reporting interval.
Note: Both are enabled by default, and both report every 5 seconds, so unless you altered the defaults, you should be good.
Message-producing permissions for consumers
Karafka Web-UI uses Karafka.producer
to produce state reports out of processes. This means that you need to make sure that the default Karafka.producer
can deliver messages to the following topics:
karafka_consumers_states
karafka_errors
Without that, Karafka will not be able to report anything.