Class: Karafka::Web::Management::Migrator

Inherits:
Object
  • Object
show all
Includes:
Processing::Consumers::Aggregators
Defined in:
lib/karafka/web/management/migrator.rb

Overview

Note:

We do not migrate the consumers reports for the following reasons: - if would be extremely hard to migrate them as they are being published and can be still published when the migrations are running - we would have to run migrations on each message - we already have a mechanism in the processing consumer that skips outdated records for rolling migrations - those records are short-lived and the expectation is for the user not to run old and new consumers together for an extensive period of time

Note:

It will raise an error if we try to run migrations but the schemas we want to operate are newer. This will prevent us from damaging the data and ensures that we only move forward with the migrations. This can happen in case of a rolling upgrade, where old instance that is going to be terminated would get a temporary assignment with already migrated state.

Migrator used to run migrations on the states topics There are cases during upgrades, where extra fields may be added and other data, so in order not to deal with cases of some information missing, we can just migrate the data and ensure all the fields that we require after upgrade are present

Migrations are similar to the once that are present in Ruby on Rails conceptually.

We take our most recent state and we can alter it “in place”. The altered result will be passed to the consecutive migrations and then republished back to Kafka. This allows us to manage Web UI aggregated data easily.

Instance Method Summary collapse

Instance Method Details

#call ⇒ Object

Note:

To simplify things we always migrate and update all the topics data even if only part was migrated. That way we always ensure that all the elements are up to date

Picks needed data from Kafka, alters it with migrations and puts the updated data back into Kafka. This ensures, that our Web UI topics that hold aggregated data are always aligned with the Web UI expectations



43
44
45
46
47
 
# File 'lib/karafka/web/management/migrator.rb', line 43

def call
  ensure_migrable!
  # If migrating returns `false` it means no migrations happened
  migrate && publish
end