Development vs Production
There are unavoidable differences when working in dev (development
and test
environments) and production
that may impact your work experience:
- Karafka
reload
mode should not be used in production. - You should create all the topics you need upfront in your production Kafka cluster.
- You should not use
allow.auto.create.topics
set totrue
in production. - Once you stabilize your topics list, you should consider locking it in the development. It is a common thing to make typos and forget about them.
- Please consider that the topics created using
allow.auto.create.topics
always have only one partition. This is ok for development, but you should create more partitions upfront for the production environment. - Remember that rolling deployment of
N
processes will createN
rebalances. This may affect performance and stability. - Creating the topic manually or by sending the first message and then starting a consumer is recommended. Karafka refreshes cluster metadata information, but it may take over five minutes to notice a new topic.
- In the development the
topic.metadata.refresh.interval.ms
is set by default to 5 seconds. This means that a topic that did not exist the moment you started Karafka will be discovered fast. It is not recommended in production, and the default is 5 minutes. - It is recommended to use
cooperative-sticky
rebalance strategy via kafkapartition.assignment.strategy
for production.