[//]: # (This file is auto-generated by bin/refresh_librdkafka)
[//]: # (Do not edit manually - changes will be overwritten)

# Karafka Integration Tests Catalog

<style>
  .md-grid {
    max-width: 100%;
  }
</style>

!!! info "Auto-Generated Documentation"
    This catalog is automatically generated from the Karafka repository's integration test suite.

    It provides an overview of all integration tests and their purposes to help developers understand the framework's testing coverage and find relevant test examples.

!!! warning "Karafka Framework Integration Tests Only"
    This catalog exclusively covers **Karafka framework integration tests** and does not include tests from other components in the ecosystem such as:

    - **karafka-rdkafka** (Ruby bindings for librdkafka)
    - **WaterDrop** (Kafka producer library)
    - **Karafka Web UI** (monitoring and management interface)
    - **Other supporting libraries**

    Each component maintains its own separate test suite with different testing approaches and structures.
    This page serves as a **Karafka-specific reference** to help developers understand the framework's capabilities, use cases, and edge case handling through real integration scenarios.

    These integration tests are particularly valuable because they demonstrate **user-facing APIs and workflows** rather than internal implementation details, making them excellent examples for understanding how to work with Karafka in practice.
    For comprehensive testing documentation of other components, please refer to their respective repositories and documentation.

| File Path | Description |
|-----------|-------------|
| `admin/acl_flow_spec.rb` | This integration spec illustrates the all the basic ACL flows. Detailed ACL API specs are in the unit RSpec specs. |
| `admin/configs_flow_spec.rb` | This integration spec illustrates the all the basic Admin configs flows. Detailed Configs API specs are in the unit RSpec specs. |
| `admin/consumer_group_operations_spec.rb` | Karafka should handle consumer group operations with active consumers correctly |
| `admin/consumer_groups/delete_existing_spec.rb` | We should be able to remove consumer group and start from beginning using the Admin API |
| `admin/consumer_groups/kip_848_admin_operations_spec.rb` | Karafka Admin API should work correctly with KIP-848 consumer groups This tests that AdminClient operations like describe work with the new protocol |
| `admin/consumer_groups/seek_consumer_group/few_partitions_few_topics_spec.rb` | We should be able to move the offset to requested location per topic partition |
| `admin/consumer_groups/seek_consumer_group/mixed_time_and_real_offsets_spec.rb` | We should be able to mix moving of the same topic data and use different type of location in between partitions. |
| `admin/consumer_groups/seek_consumer_group/time_spec.rb` | We should be able to move the offset to where we want on one partition directly by the time reference instead of an offset |
| `admin/consumer_groups/seek_consumer_group/time_total_future_spec.rb` | We should be able to move the offset to where we want on one partition directly by the time reference instead of an offset |
| `admin/consumer_groups/seek_consumer_group/time_total_past_spec.rb` | We should be able to move the offset way beyond in time and should just select first offset |
| `admin/consumer_groups/seek_consumer_group/to_earliest_latest_par_partition_spec.rb` | We should be able to move the offset per partition using `:earliest` and `:latest` |
| `admin/consumer_groups/seek_consumer_group/to_earliest_on_topic_spec.rb` | We should be able to move the the earliest offset by providing :earliest as the seek value on the topic level |
| `admin/consumer_groups/seek_consumer_group/to_earliest_on_topic_via_false_spec.rb` | We should be able to move the the earliest offset by providing `false` as the seek value on the topic level |
| `admin/consumer_groups/seek_consumer_group/to_earliest_string_on_topic_spec.rb` | We should be able to move the the earliest offset by providing 'earliest' as the seek value on the topic level |
| `admin/consumer_groups/seek_consumer_group/to_latest_spec.rb` | We should be able to move the the latest offset by providing :latest as the seek value on the topic level |
| `admin/consumer_groups/seek_consumer_group/topic_spec.rb` | We should be able to move the offset on a topic level and it should move where expected |
| `admin/consumer_groups/seek_consumer_group/topic_time_total_past_multi_partition_all_spec.rb` | We should be able to move the offset way beyond in time for all partitions at once |
| `admin/consumer_groups/seek_consumer_group/topic_time_total_past_multi_partition_spec.rb` | We should be able to move the offset way beyond in time for all partitions at once |
| `admin/consumer_groups/seek_consumer_group/topic_time_total_past_spec.rb` | We should be able to move the offset way beyond in time and should just select first offset This should work when defining time on a topic level |
| `admin/consumer_groups/trigger_rebalance_spec.rb` | Karafka should be able to trigger a rebalance for a consumer group using Admin API |
| `admin/copy_consumer_group/when_previous_and_new_cgs_exist_spec.rb` | When we have old cg and new cg and topics with offsets to migrate, it should work |
| `admin/copy_consumer_group/when_previous_cg_does_not_exist_spec.rb` | When the previous consumer group does not exist, it should not crash |
| `admin/copy_consumer_group/with_non_existing_topics_spec.rb` | When trying to migrate with non-existing topics, it should migrate those in use (none) |
| `admin/lag_calculation_edge_cases_spec.rb` | Karafka should handle lag calculation edge cases correctly |
| `admin/metadata_refresh_during_changes_spec.rb` | Karafka should handle metadata refresh correctly during topic changes |
| `admin/multi_cluster_operations_spec.rb` | Karafka Admin should support operations on multiple Kafka clusters by allowing custom kafka configuration per admin instance. This test verifies that: 1. An admin instance with custom kafka config can connect to a specific cluster 2. The custom config overrides the default config 3. Operations work correctly with the custom cluster |
| `admin/post_fork_behaviour_spec.rb` | When working with Admin API in post-forks we should have no issues. @see https://github.com/ffi/ffi/issues/1114 |
| `admin/read_lags_with_offsets/inactive_topics_lags_spec.rb` | When configured with inactive visible, we should get their lags. |
| `admin/read_lags_with_offsets/multi_cg_setup_spec.rb` | A set of cases that can be easily tested together to save time |
| `admin/read_lags_with_offsets/simple_cases_spec.rb` | A set of cases that can be easily tested together to save time |
| `admin/read_topic/with_custom_deserializer_spec.rb` | When using read_topic with topic that is inactive, the deserializer resolution should work and properly defined deserializer should be used |
| `admin/read_topic/with_default_for_non_routed_topic_spec.rb` | When using read_topic with a topic that is not part of the routing, it should use the defaults deserializers defined if they are present instead of the "total" framework defaults |
| `admin/read_topic/with_defaults_but_overwritten_in_routes_spec.rb` | When having defaults but also having explicit definitions, defaults should not overwrite explicit for admin |
| `admin/rename_consumer_group/when_previous_and_new_cgs_exist_spec.rb` | When we have old cg and new cg and topics with offsets to migrate, it should work |
| `admin/rename_consumer_group/when_previous_cg_does_not_exist_spec.rb` | When the previous consumer group does not exist, it should return false |
| `admin/rename_consumer_group/with_non_existing_topics_spec.rb` | When trying to migrate with non-existing topics, it should migrate those in use (none) |
| `admin/rename_consumer_group/without_old_removal_spec.rb` | When we have old cg and new cg and topics with offsets to migrate, it should work When we indicate that old group should stay, it should not be removed. |
| `admin/replication/plan_basic_spec.rb` | Karafka should generate basic replication plans with correct structure |
| `admin/replication/plan_error_handling_spec.rb` | Karafka should validate replication plan parameters and provide clear errors |
| `admin/replication/plan_kafka_tool_compatibility_spec.rb` | Karafka-generated JSON should work with kafka-reassign-partitions.sh This is a TRUE INTEGRATION TEST that validates our JSON is compatible with Kafka's official tooling by actually executing kafka-reassign-partitions.sh |
| `admin/replication/plan_manual_assignment_spec.rb` | Karafka should support manual broker assignment for replication plans |
| `admin/replication_info_fetch_spec.rb` | We should be able to fetch and use info about replication, in sync etc. |
| `admin/replication_multi_broker_sequential/concurrent_operations_spec.rb` | Karafka should handle concurrent RF changes on multiple topics |
| `admin/replication_multi_broker_sequential/increase_replication_factor_spec.rb` | Karafka should increase topic replication factor from 1 to 3 using kafka-reassign-partitions |
| `admin/replication_multi_broker_sequential/insufficient_brokers_error_spec.rb` | Karafka should raise errors for invalid replication factor values |
| `admin/replication_multi_broker_sequential/manual_broker_assignment_spec.rb` | Karafka should respect manual broker assignment when increasing replication factor |
| `admin/replication_multi_broker_sequential/multi_partition_topic_spec.rb` | Karafka should increase replication factor for multi-partition topics |
| `admin/replication_multi_broker_sequential/rebalance_replicas_spec.rb` | Karafka should rebalance replicas across brokers without changing replication factor |
| `admin/replication_multi_broker_sequential/verify_isr_after_change_spec.rb` | Karafka should have valid ISR after replication factor change |
| `admin/seeking_operations_active_spec.rb` | Karafka should handle seeking operations during active consumption correctly |
| `admin/topic_operations_nonexistent_spec.rb` | Karafka should handle administrative operations on non-existent topics gracefully |
| `cli/cli_start_spec.rb` | Karafka CLI should work and should just run help without command |
| `cli/cli_unknown_spec.rb` | Karafka CLI should work and should fail with unknown command error |
| `cli/declaratives/align/aliased_aligned_spec.rb` | karafka topics align should do nothing when no changes needed despite changes being defined but then redefined by an alias |
| `cli/declaratives/align/aligned_spec.rb` | karafka topics align should do nothing when no changes needed |
| `cli/declaratives/align/existing_spec.rb` | karafka topics align should be able to align declarative topics that have their settings changed |
| `cli/declaratives/align/incorrectly_spec.rb` | karafka topics align should crash if we want to align something with incorrect changes Please note this is NOT transactional. It will crash and exit 1. |
| `cli/declaratives/align/nothing_spec.rb` | karafka topics align should do nothing when no declarative topics |
| `cli/declaratives/create/empty_spec.rb` | karafka topics create should work and not fail when no topics are defined |
| `cli/declaratives/create/new_spec.rb` | karafka topics create should create topics when defined in routing |
| `cli/declaratives/create/new_with_exit_code_spec.rb` | karafka topics create should exit with 2 when new topics are created and detailed exit codes are turned on |
| `cli/declaratives/create/non_empty_but_opted_out_spec.rb` | karafka topics create should not create topics when defined in routing but with opted out config setup |
| `cli/declaratives/create/skip_existing_spec.rb` | karafka topics create should skip topics that already exist |
| `cli/declaratives/delete/empty_spec.rb` | karafka topics delete should work and not fail when no topics are defined |
| `cli/declaratives/delete/existing_spec.rb` | karafka topics delete should delete existing topics defined in routes |
| `cli/declaratives/delete/existing_with_exit_code_spec.rb` | karafka topics delete with detailed exit code should exit with 2 when cli runs with detailed flag. We check this in the exit code definitions for integrations |
| `cli/declaratives/delete/opted_out_spec.rb` | karafka topics delete should not delete existing topics defined in routes but opted out |
| `cli/declaratives/delete/skip_non_existing_spec.rb` | karafka topics delete should skip non-existing topics defined in routes and nothing should break |
| `cli/declaratives/delete/skip_not_defined_spec.rb` | karafka topics delete should never delete topics that are not defined in the routes |
| `cli/declaratives/ignore_non_primary_cluster_spec.rb` | In case of using a custom `#kafka` setup for a topic, we should consider it a secondary and we should not run anything against it |
| `cli/declaratives/migrate/existing_more_partitions_spec.rb` | karafka topics migrate should repartition topics when defined in routing, existing and not enough partitions |
| `cli/declaratives/migrate/existing_no_change_spec.rb` | karafka topics migrate should do nothing when all stays the same |
| `cli/declaratives/migrate/existing_no_topic_spec.rb` | karafka topics migrate should create topics when defined in routing and not existing |
| `cli/declaratives/migrate/inactive_spec.rb` | karafka topics migrate should work with topics that are not actively consumed but are managed using Karafka |
| `cli/declaratives/migrate/opted_out_spec.rb` | karafka topics migrate should not create topics when defined in routing and not existing but also not active. Should also not run alignments when there are no declarative topics |
| `cli/declaratives/plan/when_changes_with_detailed_exit_code_spec.rb` | Declaratives should work when multiple things are changing |
| `cli/declaratives/plan/when_downscaling_partitions_spec.rb` | When declarative topic exists but has more partitions than declared, we should just inform that such change is not supported and will be ignored |
| `cli/declaratives/plan/when_downscaling_partitions_with_exit_code_spec.rb` | When declarative topic exists but has more partitions than declared, we should exit with 0 when detailed exit code is on because no changes will be applied |
| `cli/declaratives/plan/when_missing_topics_partitions_and_configs_spec.rb` | Declaratives should work when multiple things are changing |
| `cli/declaratives/plan/when_no_changes_with_detailed_exit_code_spec.rb` | When declarative topics exist but no changes needed it should exit with 0 when detailed exit code is enabled |
| `cli/declaratives/plan/when_no_topics_at_all_spec.rb` | When no declarative topics, nothing should break |
| `cli/declaratives/plan/when_no_topics_at_all_with_exit_code_spec.rb` | When no declarative topics, we should exit with 0 on detailed exit code |
| `cli/declaratives/plan/when_same_config_under_aliases_no_change_spec.rb` | When declarative topics exist and there is same attribute defined multiple times with the last synonym (or proper name) having default, Karafka should not suggest any migration as the last value is default that is already in use |
| `cli/declaratives/plan/when_same_config_under_aliases_with_change_spec.rb` | When declarative topics exist with config request, it should work even if the same attribute is defined multiple times as long as the last value is a change Proper name should be used and not alias but value from last used (which is alias) |
| `cli/declaratives/plan/when_same_config_under_aliases_with_proper_change_spec.rb` | When declarative topics exist with config request, it should work even if the same attribute is defined multiple times as long as the last value is a change Proper name should be used and not alias and value from last used (which is alias) |
| `cli/declaratives/plan/when_with_config_spec.rb` | When declarative topics exist with config request, it should work |
| `cli/declaratives/plan/when_with_no_changes_needed_spec.rb` | When declarative topics exist but no changes needed as the config is the same, we should not print changes details |
| `cli/declaratives/plan/when_with_partitions_spec.rb` | When declarative topics exist with repartition request, it should work |
| `cli/declaratives/plan/when_without_changes_spec.rb` | When declarative topics exist but no changes needed, nothing should break |
| `cli/declaratives/repartition/empty_spec.rb` | karafka topics repartition should work and not fail when no topics are defined |
| `cli/declaratives/repartition/less_present_spec.rb` | karafka topics repartition should increase number of partitions if less than what we want |
| `cli/declaratives/repartition/more_present_spec.rb` | karafka topics repartition should do nothing, when there are already more partitions or equal number of them |
| `cli/declaratives/repartition/opted_out_spec.rb` | karafka topics repartition should not increase number of partitions if less than what we want but the topic is opted out |
| `cli/declaratives/reset/existing_spec.rb` | karafka topics reset should delete and re-create topics defined in routes |
| `cli/declaratives/strict_setup_requirement_spec.rb` | When we have strict_declarative_topics set to true, we should ensure all non-pattern definitions of topics have their declarative references |
| `cli/info_spec.rb` | Karafka CLI should work and should not fail |
| `consumption/at_most_once_on_error_spec.rb` | When marking as consumed before the error, message should be skipped as it should be considered consumed |
| `consumption/batch_processing/early_termination_spec.rb` | Karafka should handle batch processing with early termination scenarios |
| `consumption/batch_processing/empty_batch_handling_spec.rb` | Karafka should handle empty batches correctly in different processing strategies |
| `consumption/batch_processing/single_message_batches_spec.rb` | Karafka should handle single message batches correctly |
| `consumption/batch_processing/variable_batch_sizes_spec.rb` | Karafka should handle very small and very large batches efficiently |
| `consumption/compressions/using_gzip_spec.rb` | Karafka should be able to produce and consume messages compressed with gzip codec |
| `consumption/compressions/using_lz4_spec.rb` | Karafka should be able to produce and consume messages compressed with lz4 codec |
| `consumption/compressions/using_snappy_spec.rb` | Karafka should be able to produce and consume messages compressed with snappy codec |
| `consumption/concurrency/memory_visibility_spec.rb` | Karafka should handle memory visibility between processing threads correctly |
| `consumption/concurrency/shared_consumer_state_spec.rb` | Karafka should handle concurrent access to shared consumer state safely |
| `consumption/concurrency/shared_mutable_objects_spec.rb` | Karafka should handle message processing with shared mutable objects safely |
| `consumption/concurrency/thread_local_isolation_spec.rb` | Karafka should handle thread-local variable isolation properly |
| `consumption/consumer_initialization_with_preexisting_topics_spec.rb` | Karafka should initialize consumers properly when topics already have messages |
| `consumption/eofed/eof_offset_continuity_spec.rb` | Regression test to ensure correct offset handling when EOF is reached and subsequent messages are produced. Note: While librdkafka has a bug in rd_kafka_consume_batch() where offsets can be incorrectly advanced by 2 after EOF (see https://github.com/confluentinc/librdkafka/pull/5213), this bug does NOT affect Karafka as we use different APIs (rd_kafka_consumer_poll). However, the bug affects rd_kafka_position(), which we use. This test verifies that position tracking remains correct after EOF. Test scenario: 1. Produce 5 messages (offsets 0-4) 2. Consume all and reach EOF 3. Verify position is 5 (last consumed + 1) 4. Produce 1 more message (offset 5) 5. Verify all 6 messages are consumed with continuous offsets (0-5) |
| `consumption/eofed/error_recovery_spec.rb` | When eof is in use and `#eofed` crashes, it should emit an error but it should not cause any other crashes. Since `#eofed` does not deal with new data, it is not retried. It is up to the user to deal with any retry policies he may want to have eofed errors should not leak and processing should continue |
| `consumption/eofed/marking_as_consumed_spec.rb` | We should be able to mark as consumed from the `#eofed` |
| `consumption/eofed/post_consumption_spec.rb` | When eof is in use we may not get it as it may go with messages polling via `#consume` It may happen that eof goes via `#eofed` because of polling but we will correct this spec only when this would happen as it should not be frequent |
| `consumption/eofed/post_eof_state_change_spec.rb` | When we had an eof but then not, it should be reflected in the `#eofed?` status |
| `consumption/eofed/when_eof_reached_on_an_empty_topic_spec.rb` | When eof is in use and we get it on an empty topic, we should not use consume and start with `#eofed`. Shutdown should also work and respond to `#eofed?` |
| `consumption/eofed/when_eof_reached_on_multiple_empty_topics_spec.rb` | When eof is in use and we get it on an empty topic, we should not use consume and start with `#eofed` per each partition and topic independently |
| `consumption/eofed/without_eofed_on_spec.rb` | When enable.partition.eof is on but topic eofed is false, we should not run eofed |
| `consumption/error_on_first_latest_and_restart_spec.rb` | When starting from latest offset and having error on first run, Karafka has no offset to write as the first one, thus if restarted, it will against start from "latest". |
| `consumption/error_on_first_latest_marked_and_restart_spec.rb` | When starting from latest offset and having error on first run, Karafka has no offset to write as the first one, thus if restarted, it will against start from "latest". |
| `consumption/finishing_job_after_max_poll_exceeded_spec.rb` | While we cannot mark after max.poll.interval.ms kicked us out, Karafka should not break or stop in any way the job that was being executed |
| `consumption/from_never_existing_topic_spec.rb` | Karafka should work when subscribing to a topic that does not exist This should operate even if topic is not created as it may be in the future. Even if not created, we should still be able to operate, start and stop |
| `consumption/from_non_existing_topic_spec.rb` | Karafka in dev should pick up new topic creation fairly fast. It should not wait for 5 minutes |
| `consumption/inflight_topic_removal_with_auto_create_spec.rb` | When topic in use is removed, Karafka may issue an `unknown_partition` error but even if, it should re-create the topic and move on. |
| `consumption/inflight_topic_removal_without_auto_create_spec.rb` | When topic in use is removed, Karafka should emit an error |
| `consumption/inline_insights/from_earliest_spec.rb` | We should be able to get the insights and use them via the API when they are defined |
| `consumption/inline_insights/multi_topic_partition_spec.rb` | Inline Insights should ship correct partition and topic details and not mix them |
| `consumption/inline_insights/slow_with_refreshed_insights_spec.rb` | Inline insights should auto-refresh when new values are present during extensive processing |
| `consumption/inline_insights/without_being_enabled_spec.rb` | Karafka should not have the insights methods when insights are not enabled |
| `consumption/loop_with_messages_with_headers_spec.rb` | Karafka should be able to consume messages in a loop Messages can have headers that should be accessible to use |
| `consumption/max_message_size_mismatches_spec.rb` | Karafka should handle different message sizes properly |
| `consumption/message_content_edge_cases_spec.rb` | Karafka should handle various message content edge cases properly |
| `consumption/no_long_wait_with_eof_spec.rb` | With above settings it would never finish because would accumulate messages for so long Only if eof return works, will it finish fast enough |
| `consumption/of_a_tombstone_record_spec.rb` | Karafka should not have any problems of consuming of a tombstone record with the default deserializer |
| `consumption/one_consumer_group_two_topics_spec.rb` | Karafka should be able to consume two topics with same consumer group |
| `consumption/one_worker_many_topics_spec.rb` | Karafka should be able to consume multiple topics with one worker |
| `consumption/pausing_not_unlocking_spec.rb` | When running longer jobs, someone may try to pause processing prior to reaching poll interval to bypass the issue. This will not work and this spec illustrates this. |
| `consumption/producer_ping_pong_spec.rb` | Karafka should be able to easily consume and produce messages from consumer |
| `consumption/producing_with_aliases_spec.rb` | Karafka should be able to produce from consumers using the delegated aliased API |
| `consumption/rate_limited_spec.rb` | Karafka should be able to use pause to rate limit when consumption is tracked We can do it by using the pausing capabilities. While it is rather not recommended, but for the sake of demo and making sure things work as expected, we us it |
| `consumption/reaching_max_messages_count_spec.rb` | When we reach max messages prior to reaching max wait time, we should process that many messages without waiting max wait time |
| `consumption/reaching_max_wait_time_spec.rb` | When we have a max_wait_time and we did not reach the requested number of messages, we should wait for at most the max time we requested. We also should not wait shorter period of time, as the messages number is not satisfied. |
| `consumption/removing_a_topic_should_still_consume_rest_spec.rb` | Karafka should be able to consume messages after a no-longer used topic has been removed from a given consumer group. It should not cause any problems |
| `consumption/retrying_flow_spec.rb` | Non-errored flow should not be in the retrying state, while consecutive retries should |
| `consumption/simple_from_earliest_spec.rb` | Karafka should be able to consume all the data from beginning |
| `consumption/single_message_consumer_spec.rb` | With a custom per message consumer abstraction layer, we should be able to use "1.4 style" of using per-message consumers while retaining decent performance and warranties |
| `consumption/standard_saturation_reliability_spec.rb` | Non Pro Karafka handles oversaturated jobs on involuntary revocation as any others. That is, it does not provide extended processing warranties and will run those jobs as any others. This spec demonstrates and checks that. Extended saturation warranties are part of the Pro. |
| `consumption/strategies/aj_mom/backoff_and_retry_on_error_spec.rb` | When there is ActiveJob processing error, Karafka should backoff and retry |
| `consumption/strategies/aj_mom/big_batch_shutdown_early_break_spec.rb` | Karafka should be able to finish a big batch of jobs early when we decide to stop All the jobs from the batch should not be processed and we should early exit. |
| `consumption/strategies/aj_mom/dispatching_async_job_spec.rb` | Karafka should be able to dispatch jobs using async adapter |
| `consumption/strategies/aj_mom/dispatching_sync_job_spec.rb` | Karafka should be able to dispatch jobs using sync adapter |
| `consumption/strategies/aj_mom/failing_perform_later_spec.rb` | OSS Karafka should not support `#enqueue_at` |
| `consumption/strategies/aj_mom/invalid_configuration_attempt_spec.rb` | When there is a misconfiguration of karafka options on ActiveJob job class, it should raise an error |
| `consumption/strategies/aj_mom/multiple_jobs_same_topic_spec.rb` | Karafka should be able to handle multiple jobs with same topic |
| `consumption/strategies/aj_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `consumption/strategies/aj_mom/options_override_and_inheritance_spec.rb` | Karafka should use different partitioners and karafka options for jobs and not mutate in between |
| `consumption/strategies/aj_mom/parallel_current_attributes_coop_spec.rb` | Karafka should correctly assign and manage the current attributes in multiple threads |
| `consumption/strategies/aj_mom/processing_all_on_shutdown_spec.rb` | Karafka should finish processing all the jobs that it has from current messages batch before completely shutting down @note This behaviour is different with PRO AJ consumer, where Karafka waits only on the currently processed work (for LRJ and without LRJ) |
| `consumption/strategies/aj_mom/retry_on_flow_spec.rb` | Karafka should be able to handle retry_on with immediate retries in OSS as long as there is no jitter |
| `consumption/strategies/aj_mom/tagging_with_job_class_spec.rb` | Karafka should automatically assign the tag of job class that is executed |
| `consumption/strategies/aj_mom/with_envelope_and_current_attributes_spec.rb` | Karafka should correctly handle current attributes when using a custom envelope deserializer This ensures that current attributes persistence works with custom serialization formats |
| `consumption/strategies/aj_mom/with_envelope_spec.rb` | Karafka should be able to process ActiveJob jobs wrapped in a custom envelope This demonstrates a common pattern where all Kafka messages must follow a specific format with additional metadata wrapping the actual job payload Custom deserializer that wraps jobs in an envelope with metadata This extends the default Karafka ActiveJob deserializer |
| `consumption/strategies/aj_mom_dlq/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `consumption/strategies/aj_mom_dlq/offset_skip_on_non_recoverable_error_spec.rb` | When there is ActiveJob processing error that cannot recover, upon moving to DLQ, the offset should be moved as well. |
| `consumption/strategies/aj_mom_dlq/processing_on_non_recoverable_error_spec.rb` | When there is ActiveJob processing error that cannot recover, upon moving to DLQ, the offset should be moved as well and we should continue. |
| `consumption/strategies/default/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt |
| `consumption/strategies/default/constant_error_should_not_clog_others_spec.rb` | Karafka should process other partitions data using same worker in which a job failed Workers should not hang when a job within them fails but should be available for other jobs Workers should not be clogged by a failing job |
| `consumption/strategies/default/from_earliest_simple_routing_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) when simple routing is in use |
| `consumption/strategies/default/from_earliest_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) |
| `consumption/strategies/default/from_latest_with_non_persistent_error_spec.rb` | When we connect for the first time with cluster from a new consumer group and start consuming from earliest and an error occurs on a first message, we should pause and retry consumption until we can process this message. No messages should be skipped or ignored. |
| `consumption/strategies/default/kip_848/post_revocation_recovery_spec.rb` | Karafka should recover from expired timeout when post-recovery the processing is fast enough |
| `consumption/strategies/default/kip_848/revocation_on_timeout_spec.rb` | Test KIP-848 with regular (non-LRJ) consumption to ensure that when max.poll.interval.ms is exceeded, the consumer is properly kicked out and revoked callbacks are triggered |
| `consumption/strategies/default/kip_848/shutdown_revocation_spec.rb` | Test KIP-848 to verify that the #revoked callback is not called during shutdown |
| `consumption/strategies/default/kip_848/single_cg_mixed_spec.rb` | Karafka should allow for running a single consumer group with new protocol members joining. It should allow for transition. Note: it is not recommended to do so. |
| `consumption/strategies/default/many_topics_same_group_spec.rb` | Karafka should be able to subscribe and consume messages from as many topics as someone wants Here we consume from 100 topics |
| `consumption/strategies/default/multi_topic_ping_pong_spec.rb` | Karafka should be able to easily consume messages sent from one topic to another |
| `consumption/strategies/default/non_constant_error_and_other_topics_spec.rb` | When on one partition topic an error occurs, other topics should be processed and given partition should catch up on recovery after the pause timeout |
| `consumption/strategies/default/non_critical_error_constant_backoff_spec.rb` | When we have non-critical error happening couple times and we use constant backoff, Karafka should not increase the backoff time with each occurrence. |
| `consumption/strategies/default/non_critical_error_exponential_backoff_spec.rb` | When we have non-critical error happening couple times and we use exponential backoff, Karafka should increase the backoff time with each occurrence until max backoff. |
| `consumption/strategies/default/non_critical_error_pause_no_marking_spec.rb` | Karafka when no marking happens should restart whole batch |
| `consumption/strategies/default/non_critical_error_pause_position_marking_spec.rb` | Karafka even when a batch is fetched, should pause on message that failed as long as marking as consumed happened. It should not restart whole batch if marking happened |
| `consumption/strategies/default/non_critical_error_post_error_action_spec.rb` | When we have non-critical error we should be able to do some "post error" operations without breaking the backoff Karafka offers |
| `consumption/strategies/default/non_critical_error_recovery_spec.rb` | Karafka should be able to recover from non-critical error with same consumer instance and it also should emit an event with error details that can be logged |
| `consumption/strategies/default/of_many_partitions_with_error_spec.rb` | When consuming on multiple workers, when one receives a non-critical exception, others should continue processing and the one should be retried |
| `consumption/strategies/default/of_many_partitions_with_many_workers_spec.rb` | Karafka should use more than one thread to consume independent topics partitions |
| `consumption/strategies/default/of_many_topics_on_many_workers_spec.rb` | Karafka should use more than one thread to consume independent topics |
| `consumption/strategies/default/of_many_topics_with_different_settings_spec.rb` | Karafka should be able to consume multiple topics even when there are many subscription groups underneath due to non-homogeneous settings Usually configuration like this may not be optimal with too many subscription groups, nonetheless we should support it |
| `consumption/strategies/default/of_many_with_error_and_long_pause_spec.rb` | Karafka should pause and if pausing spans across batches, it should work and wait |
| `consumption/strategies/default/of_messages_with_headers_spec.rb` | Karafka should be able to work with messages that have headers and all headers should have string keys |
| `consumption/strategies/default/processing_with_delay_spec.rb` | With usage of `#pause` we should be able to ensure we always process messages with certain delay not to process messages that are "fresh" |
| `consumption/strategies/default/with_failed_transactions_spec.rb` | When data we try to consume comes from aborted transactions, it should not be visible by default |
| `consumption/strategies/default/with_offset_moved_backwards_directly_spec.rb` | Allow to mark in past. |
| `consumption/strategies/dlq/dispatch_instrumentation_spec.rb` | When DLQ delegation happens, Karafka should emit appropriate event. |
| `consumption/strategies/dlq/independent_constant_recovering_spec.rb` | When independent flag is on and the error is rolling, it should never go to the DLQ |
| `consumption/strategies/dlq/manual_dispatch_to_dlq_spec.rb` | When having the DLQ defined, we should be able to manually dispatch things to the DLQ and continue processing whenever we want. We can use this API to manually move stuff to DLQ without raising any errors upon detecting a corrupted message. |
| `consumption/strategies/dlq/multi_partition_source_target_flow_spec.rb` | When handling failing messages from a many partitions and there are many errors, standard DLQ will not provide strong ordering warranties inside DLQ. If you need strong ordered DLQ, please look into getting the Pro version. |
| `consumption/strategies/dlq/multi_partition_target_flow_spec.rb` | When handling failing messages from a single partition and there are many errors, standard DLQ will not provide strong ordering warranties inside DLQ. If you need strong ordered DLQ, please look into getting the Pro version. |
| `consumption/strategies/dlq/partial_manual_dlq_with_out_of_scope_crashes_spec.rb` | In case a manual DLQ dispatch is used on internal operations but the whole batch processing crashes, this can cause extensive reprocessing with small forward movement. Out of 100 messages produced, with this code we will end up with over a 1000 of failures because of how offsets are moving forward Note: This example illustrates a misuse of the DLQ flow and should not be used. |
| `consumption/strategies/dlq/with_custom_producer_spec.rb` | In the DLQ flow custom producer should be used when defined |
| `consumption/strategies/dlq/with_error_handling_pipeline_spec.rb` | We should be able to use DLQ in a pipeline where we handle each stage separately. |
| `consumption/strategies/dlq/with_external_each_patch_spec.rb` | This spec verifies that DLQ strategy works correctly when external libraries monkey-patch the Messages#each method (e.g., for tracing/instrumentation purposes). The DLQ strategy internally uses messages.raw.find to locate skippable messages, which bypasses the patched each method by accessing the underlying array directly. @see https://github.com/karafka/karafka/issues/2939 |
| `consumption/strategies/dlq/with_first_crashing_spec.rb` | When its the first message ever that is constantly crashing it should move as expected with the DLQ flow |
| `consumption/strategies/dlq/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic |
| `consumption/strategies/dlq/with_non_recoverable_error_without_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message and we don't want to do any retries, we should skip without retrying of processing. |
| `consumption/strategies/dlq/with_non_recoverable_first_message_spec.rb` | When dead letter queue is used and we first message out of all is broken, things should behave like for any other broken message |
| `consumption/strategies/dlq/with_non_recoverable_last_message_spec.rb` | When dead letter queue is used and we last message out of all is broken, things should behave like for any other broken message and we should pick up when more messages are present |
| `consumption/strategies/dlq/with_recoverable_error_on_retry_spec.rb` | When dead letter queue is used and we encounter recoverable message, we should not skip and not move it to the dead letter topic. Just retry. |
| `consumption/strategies/dlq/with_rolling_error_spec.rb` | Without the independent flag (default) Karafka will accumulate attempts on a batch in a rolling fashion when recoverable errors appear over and over again on a set of messages. |
| `consumption/strategies/dlq/with_sync_dispatch_spec.rb` | When using DLQ, it should work when dispatch in sync |
| `consumption/strategies/dlq/with_sync_marking_spec.rb` | When using DLQ, it should work when marking as consumed sync |
| `consumption/strategies/dlq/without_any_errors_spec.rb` | When dead letter queue is used and we don't encounter any errors, all should be regular. |
| `consumption/strategies/dlq/without_intermediate_marking_spec.rb` | When we do not perform any markings as consumed we should nonetheless skip one after another starting from the beginning of the batch on failed batches. This should allow us to move one by one slowly and should mark on successful batches. |
| `consumption/strategies/dlq/without_marking_spec.rb` | When using DLQ without post error marking, the committed offset should remain unchanged |
| `consumption/strategies/dlq_mom/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt |
| `consumption/strategies/dlq_mom/multi_partition_source_target_flow_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/multi_partition_target_flow_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_error_handling_pipeline_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_first_crashing_spec.rb` | When its the first message ever that is constantly crashing it should move as expected with the DLQ flow |
| `consumption/strategies/dlq_mom/with_marking_spec.rb` | When we do mark and user does not mark, we will not end up with an infinite loop. |
| `consumption/strategies/dlq_mom/with_non_recoverable_error_with_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_non_recoverable_error_without_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_non_recoverable_first_message_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_non_recoverable_last_message_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/with_recoverable_error_on_retry_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/without_any_errors_spec.rb` | Same as pure DLQ version until rebalance |
| `consumption/strategies/dlq_mom/without_intermediate_marking_spec.rb` | When we do not mark and user does not mark, we will end up with an infinite loop. This is expected and user should deal with this on his own. |
| `consumption/strategies/mom/manual_offset_with_error_every_bang_spec.rb` | When manual offset management is on, upon error Karafka should start again from the place it had in the checkpoint. If we checkpoint after each message is processed (here adding to array) it should not have any duplicates as the error happens before checkpointing |
| `consumption/strategies/mom/manual_offset_with_error_every_non_bang_spec.rb` | When manual offset management is on, upon error Karafka should start again from the place it had in the checkpoint. If we checkpoint after each message is processed (here adding to array) it should not have any duplicates as the error happens before checkpointing |
| `consumption/strategies/mom/manual_offset_with_error_interval_bang_spec.rb` | When manual offset management is on, upon error Karafka should start again from the place it had in the checkpoint, not from the beginning. We check here that this works well when we commit "from time to time", not every message |
| `consumption/strategies/mom/manual_offset_with_error_interval_non_bang_spec.rb` | When manual offset management is on, upon error Karafka should start again from the place it had in the checkpoint, not from the beginning. We check here that this works well when we commit "from time to time", not every message |
| `consumption/strategies/mom/with_manual_offset_and_error_spec.rb` | Using manual offset management under rebalance and error happening, we should start from the last place that we were, even when there were many batches down the road and no checkpointing |
| `consumption/strategies/mom/with_offset_moved_backwards_directly_spec.rb` | Allow to mark in past. |
| `consumption/time_bound_consumption_spec.rb` | Karafka should be able to consume messages for given amount of time (10 seconds) and then stop While Karafka is designed as a long-running process, it can be used as recurring job as well |
| `consumption/two_consumer_groups_one_active_spec.rb` | Karafka should be able to consume one topic when two consumer groups are defined but only one is active |
| `consumption/two_consumer_groups_same_topic_spec.rb` | Karafka should be able to consume same topic using two consumer groups |
| `consumption/when_initialization_happens_spec.rb` | Karafka should invoke the `#initialized` method and things like topic, partition and messages should not be nil |
| `consumption/with_array_headers_spec.rb` | Karafka should support KIP-82 and should be able to both read and write array headers |
| `consumption/with_branching_logic_on_retry_spec.rb` | We should be able to apply custom logic flow, when re-processing data after an error. |
| `consumption/with_cluster_info_based_routes_spec.rb` | We should be able to define settings and then use cluster info to get topics and subscribe to them automatically also matching via regexp |
| `consumption/with_error_for_seek_offset_tracking_spec.rb` | Karafka should publish seek offset upon error so we know offset of message that failed @note This may be a bit more complex when doing batch processing operations |
| `consumption/with_existing_and_not_existing_topics_in_same_sg_spec.rb` | When Karafka starts consuming in one SG existing topic and not existing topic and auto create is off, it should emit an error but at the same time should consume from existing topic |
| `consumption/with_existing_cg_on_a_new_topic_spec.rb` | When we have an existing CG that gets a new topic from latest, it should pick from latest. We postpone second connection but because it has 'latest' start, it should not poll any data because data was published only in the past |
| `consumption/with_future_messages_due_to_drift_spec.rb` | Karafka should be able to handle a case where the cluster and consumer times drifted Some metrics may not be 100% accurate and this should not happen often but may |
| `consumption/with_lazy_payload_deserialization_spec.rb` | Karafka when processing messages should deserialize only in case where we request payload to be deserialized even when iterating over all the objects. |
| `consumption/with_long_living_buffer_spec.rb` | Karafka should allow to create a long living buffer that we can use and fill as we go, beyond a single batch of data This will force several batches, so we won't end up with 1 huge as this is not what we want here |
| `consumption/with_many_partitions_spec.rb` | Karafka should be able to consume messages from thousands of partitions Please note, that creating a topic with that many partitions can take few seconds This number should not go higher because slow CIs can be overloaded |
| `consumption/with_max_messages_spec.rb` | Karafka should not return more messages than defined with `max_messages` |
| `consumption/with_mixed_performance_in_subscription_group_spec.rb` | When running work of a single consumer group from many subscription groups, upon shutdown we should not experience any rebalance problems as the shutdown should happen after all the work is done. |
| `consumption/with_mixed_subscription_groups_spec.rb` | When using separate subscription groups for every nth topic, we should have proper number of underlying clients. |
| `consumption/with_non_existing_topics_auto_create_spec.rb` | Karafka should be able to poll on non-existing topics when auto-create is on and should not raise an error about it |
| `consumption/with_non_existing_topics_no_auto_create_spec.rb` | Karafka should raise an error when it encounters non-existing topic and no auto topic creation We need to allow errors so we can track them and check the propagation |
| `consumption/with_on_after_consume_method_spec.rb` | Karafka has a `#after_consume` method. This method should not be used as part of the official API but we add integration specs here just to make sure it runs as expected. |
| `consumption/with_on_before_consume_method_spec.rb` | Karafka has a `#before_consume` method. This method should not be used as part of the official API but we add integration specs here just to make sure it runs as expected. |
| `consumption/with_producer_disconnect_error_spec.rb` | When Karafka producer (WaterDrop) disconnects it should reconnect automatically. @note Here we force the disconnect by setting the `connections.max.idle.ms` to a really low value. With librdkafka updates, planned disconnections no longer emit "all brokers down" errors, so we test that the producer continues working after forced disconnection. |
| `consumption/with_reset_topic_spec.rb` | When weird things happen with partition (it disappears), Karafka receives weird metadata, etc we should be able to detect this and gracefully shutdown For any reason, especially affected is MSK during rolling updates. |
| `consumption/with_separate_subscription_groups_spec.rb` | When using separate subscription groups, each should have it's own underlying client and should operate independently for data fetching. |
| `consumption/with_static_group_of_many_subscriptions_spec.rb` | Karafka should be able to use few subscription groups with static group memberships This requires us to inject extra postfix to group id per client and should happen automatically. |
| `consumption/with_sync_mark_sa_consumed_after_revoke_spec.rb` | When we've lost a topic and end up with an `unknown_member_id`, we should handle that gracefully while running `mark_as_consumed\!`. |
| `consumption/with_unwanted_offset_reset_spec.rb` | When offset reset error occurs it should be immediately raised when user explicitly wanted an error to occur. |
| `consumption/without_persistence_enabled_spec.rb` | Karafka should not use same consumer instance when consumer_persistence is set to false Each batch of data should be consumed with new instance |
| `consumption/worker_critical_error_behaviour_spec.rb` | Karafka should recover from critical errors that happened in the workers while consuming jobs. It should notify on a proper channel and do other stuff @note This test is a bit special as due to how Karafka operates, when unexpected issue happens in particular moments, it can bubble up and exit 2 |
| `consumption/wrapping/consume_spec.rb` | Karafka should allow for wrapping on consume |
| `consumption/wrapping/on_error_spec.rb` | Errors in the processing should not affect wrapping itself |
| `consumption/wrapping/wrapped_error_handling_spec.rb` | Errors happening in wrap should be handled gracefully and by no means should they leak out If there is a chance of error (like pool timeout), it should be handled in the consumer flow so the consumer can back off if needed |
| `deserialization/custom_default_deserializer_usage_spec.rb` | Karafka should be able to use custom default deserializer on messages after it is declared |
| `deserialization/custom_deserializer_usage_spec.rb` | Karafka should be able to use custom deserializers on messages after they are declared |
| `deserialization/custom_per_topic_spec.rb` | Karafka should allow to setup defaults and never use them as it uses per topic config |
| `deserialization/data_serialization_edge_cases_spec.rb` | Karafka should handle various data serialization edge cases properly |
| `deserialization/encoding_issues_spec.rb` | Karafka should handle various encoding issues gracefully |
| `deserialization/large_nested_objects_spec.rb` | Karafka should handle large nested objects and deep JSON structures |
| `deserialization/malformed_json_handling_spec.rb` | Karafka should handle malformed JSON gracefully without crashing |
| `deserialization/mixed_formats_single_topic_spec.rb` | Karafka should handle mixed serialization formats in a single topic |
| `deserialization/routing_default_deserializer_usage_spec.rb` | Karafka should be able to use routing default deserializers |
| `embedding/puma_single_poro/flow_spec.rb` | Karafka should run from a Puma server easily |
| `embedding/puma_with_worker_poro/flow_spec.rb` | Karafka should run from a Puma server easily |
| `embedding/quiet_spec.rb` | Karafka should be able to run as embedded code and should respond to quiet. |
| `embedding/sidekiq_poro/flow_spec.rb` | Test that Karafka can run embedded in Sidekiq without Redis This test does not cover all the cases because it uses the Sidekiq testing mode, but at least it ensures that we don't have any immediate crashing conflicts. |
| `embedding/simple_spec.rb` | Karafka should be able to run as embedded code. |
| `embedding/tag_injection_spec.rb` | When running Karafka using the embedding API, we should get appropriate tag attached automatically to the Karafka process |
| `examples_deserialization_spec.rb` | All our examples should be valid. |
| `installation/standalone_nested_boot_file_pristine/install_spec.rb` | Karafka install should run and after that, we should be able to run it without any problems It should be able to install in a nested location that had to be created |
| `installation/standalone_pristine/install_spec.rb` | Karafka install should run and after that, we should be able to run it without any problems |
| `installation/waterdrop_run_pristine/run_spec.rb` | WaterDrop should work without any extra require or anything and without Rails @see https://github.com/karafka/waterdrop/pull/485 |
| `installation/with_rails7_pristine/install_spec.rb` | Karafka install should run and after that, we should be able to run it without any problems |
| `instrumentation/aj_per_job_consumption_events_spec.rb` | Karafka should instrument on particular active job jobs and should include details allowing for correlation of jobs with topic, messages, etc |
| `instrumentation/app_status_lifecycle_flow_spec.rb` | Karafka when started and stopped should go through all the lifecycle stages States changes that are published |
| `instrumentation/assignment_and_revocation_logs_spec.rb` | Karafka should have the info level logging of assigned and revoked partitions |
| `instrumentation/assignments_tracker_revocation_detection_spec.rb` | This spec demonstrates using Fiber storage to pass context (topic, partition) to helper classes without explicit arguments, enabling them to check assignment status. |
| `instrumentation/before_schedule_jobs_tracking_spec.rb` | Karafka should instrument prior to each consumer being scheduled |
| `instrumentation/cg_info_on_maximum_poll_interval_exceeded_spec.rb` | When we exceed max.poll.interval the error should contain CG name |
| `instrumentation/client_events_poll_on_assignment_loss_spec.rb` | This spec verifies that the client.events_poll event is emitted when assignment loss is detected, and that the AssignmentsTracker properly clears assignments in response. |
| `instrumentation/consumer_seeking_spec.rb` | We should be able to instrument seeking |
| `instrumentation/consumers_runtime_tagging_spec.rb` | Karafka should allow us to tag consumers runtime operations and should allow us to track those tags whenever we need from any external location. |
| `instrumentation/consumption_event_vs_processing_spec.rb` | Karafka should publish same number of consumed events as batches consumed We also should track the assignments correctly |
| `instrumentation/custom_events_spec.rb` | Karafka should allow to use the framework event bus for internal events as long as those are registered. |
| `instrumentation/error_callback_multiple_subscription_groups_spec.rb` | Karafka should publish async errors from all the clients via a dedicated instrumentation hooks and they should not collide with each other. If they would, events would be published twice. |
| `instrumentation/error_callback_on_max_poll_interval_spec.rb` | Karafka should publish the max.poll.interval.ms violations into the errors pipeline |
| `instrumentation/error_callback_subscription_spec.rb` | Karafka should publish async errors from the client via a dedicated instrumentation hook |
| `instrumentation/event_ordering_guarantees_spec.rb` | Instrumentation events should maintain proper ordering guarantees and sequence during parallel processing and concurrent event emission. |
| `instrumentation/initialized_flow_spec.rb` | Karafka should publish correct events post-initializing the consumer instances When initialization fails, it should also publish error |
| `instrumentation/listener_lifecycle_flow_spec.rb` | Karafka should publish listener lifecycle events |
| `instrumentation/low_level_buffer_analysis_spec.rb` | We should be able to inject a low level monitor for ensuring, that there are no duplicates in the data received from rdkafka |
| `instrumentation/nil_missing_data_edge_cases_spec.rb` | Instrumentation should handle events with nil/missing data gracefully without crashing the monitoring system or causing data corruption. |
| `instrumentation/oauth_listener_producer_inheritance_spec.rb` | Karafka should propagate the oauth listener to the producer when oauth in use |
| `instrumentation/pausing_and_retry_callback_subscription_spec.rb` | We should be able to instrument on pausing and retry events |
| `instrumentation/pausing_and_retry_client_callbacks_spec.rb` | We should be able to get client level instrumentation on pausing and resuming |
| `instrumentation/performance_monitoring_load_spec.rb` | Performance monitoring should handle high-volume event emission without significant performance degradation or memory leaks during heavy load scenarios. |
| `instrumentation/post_errors_instrumentation_error_spec.rb` | If post-error instrumentation re-raises the error so it bubbles up to the worker, it will crash and karafka should stop. Since an endless bubbling error cannot be rescued, it will force Karafka to move into a forceful shutdown. |
| `instrumentation/post_mutation_messages_count_reporting_spec.rb` | When we mutate the messages batch size, it should not impact the instrumentation Alongside that, because we use auto-offset management, it also should not crash as it should use original messages references. |
| `instrumentation/process_tagging_spec.rb` | We should have abilities to tag process with whatever info we want |
| `instrumentation/producer_logger_on_forks_spec.rb` | After fork unused producer logger should work as expected Note: We check this because rdkafka had an issue where fork logger would not work at all. |
| `instrumentation/producer_monitor_overwrite_spec.rb` | You should be able to use a custom monitor for the producer |
| `instrumentation/revocation_before_schedule_event_spec.rb` | Karafka should trigger proper before schedule event for revocation |
| `instrumentation/statistics_callback_multiple_subscription_groups_spec.rb` | Karafka should publish async statistics from all the clients via a dedicated instrumentation hooks and they should not collide with each other. If they would, events would be published twice. |
| `instrumentation/statistics_callback_on_client_change_spec.rb` | Karafka should not only recover from critical errors that happened but it also should reload the underlying client and keep publishing statistics from the new librdkafka client |
| `instrumentation/statistics_callback_subscription_spec.rb` | Karafka should publish async errors from the client via a dedicated instrumentation hook |
| `instrumentation/statistics_callback_with_crashing_subscription_spec.rb` | Karafka should not hang or crash when we receive the statistics processing error. It should recover and be responsive. |
| `instrumentation/statistics_frozen_lag_on_paused_spec.rb` | When partition is paused for a longer period of time, its metadata won't be refreshed. This means that consumer-reported statistics lag will be frozen. This spec illustrates that. |
| `instrumentation/statistics_on_lag_with_many_sub_group_spec.rb` | Karafka when consuming messages, should report per topic partition consumer lag By using two subscription groups, we can make sure we have separate connections and that we fetch data in parallel and ship it as it goes, so one topic partition data is not causing a wait on other things ref https://github.com/edenhill/librdkafka/wiki/FAQ#how-are-partitions-fetched |
| `instrumentation/statistics_on_lag_with_one_sub_group_spec.rb` | Karafka when consuming messages, should report per topic partition consumer lag Since librdkafka fetches data in batches onto the queue, this can cause the lagged partition to run first without processing anything else despite good concurrency settings This can be solved either via subscription group distribution or by tuning the per partition data that goes into the buffer ref https://github.com/edenhill/librdkafka/wiki/FAQ#how-are-partitions-fetched |
| `instrumentation/statistics_publishing_on_long_poll_spec.rb` | Karafka should publish statistics even when a long blocking processing occurs and there are no processing workers available as it should happen from one of the main threads. |
| `instrumentation/statistics_publishing_on_long_processing_spec.rb` | Karafka should publish statistics even when a long blocking processing occurs and there are no processing workers available as it should happen from one of the main threads. |
| `instrumentation/subscription_group_loop_start_info_spec.rb` | Karafka should publish basic info when starting a subscription group This is useful for debugging |
| `instrumentation/vendors/appsignal/errors_flow_spec.rb` | Here we ensure that our error tracker works as expected |
| `instrumentation/vendors/appsignal/metrics_flow_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications We use a dummy client that will intercept calls that should go to Appsignal and check basic metrics presence |
| `instrumentation/vendors/appsignal/with_original_client_pristine/with_proper_tracing_spec.rb` | This spec ensures, we do not use by accident ActiveSupport methods when working with listeners @see https://github.com/karafka/karafka/pull/1624 |
| `instrumentation/vendors/datadog/logger_flow_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications We use a dummy client that will intercept calls that should go to DataDog |
| `instrumentation/vendors/datadog/metrics_flow_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications We use a dummy client that will intercept calls that should go to DataDog and check basic metrics presence |
| `instrumentation/vendors/datadog/metrics_flow_with_distributions_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications We use a dummy client that will intercept calls that should go to DataDog and check basic metrics presence. Listener is instantiated with `use_distributions` set to `true`, replacing all histogram metrics with distribution metrics. |
| `instrumentation/vendors/datadog/metrics_with_eofed_spec.rb` | DD instrumentation should work with eof |
| `instrumentation/vendors/datadog/revoked_error_logger_spec.rb` | Here we subscribe to our listener and make sure it properly handles consumer.revoked.error This error type was previously not explicitly handled and would have raised UnsupportedCaseError We use a dummy client that will intercept calls that should go to DataDog This test triggers a rebalance by exceeding the poll interval, which causes partition revocation |
| `instrumentation/vendors/datadog/shutdown_error_logger_spec.rb` | Here we subscribe to our listener and make sure it properly handles consumer.shutdown.error This error type was previously not explicitly handled and would have raised UnsupportedCaseError We use a dummy client that will intercept calls that should go to DataDog |
| `instrumentation/vendors/datadog/with_excluded_librdkafka_metrics_spec.rb` | This test illustrates the metric exclusion strategy from the wiki: By using exclude_rd_kafka_metrics, users can keep most default metrics while excluding specific high-cardinality ones (like certain network latency percentiles) without having to manually recreate the entire default metrics list. This reduces DataDog costs while maintaining most observability. |
| `instrumentation/vendors/datadog/with_original_client_pristine/with_proper_ddtrace_spec.rb` | This spec ensures, we do not use by accident ActiveSupport methods when working with listeners @see https://github.com/karafka/karafka/pull/1624 |
| `instrumentation/vendors/datadog/with_reduced_librdkafka_metrics_spec.rb` | This test illustrates the cost reduction strategy from the wiki: By customizing rd_kafka_metrics to only include essential consumer lag metrics, we skip high-cardinality network latency and connection metrics that would otherwise increase DataDog costs without providing proportional observability value. |
| `instrumentation/vendors/datadog/with_reduced_metrics_and_frequency_flow_spec.rb` | We should be able to sub-class and limit the operational cost of DD listener if needed |
| `instrumentation/vendors/datadog_native_pristine/consume_with_tracing_spec.rb` | Karafka should integrate with Datadog native tracing and enable distributed tracing for message consumption |
| `instrumentation/vendors/kubernetes/consuming_liveness_parallel_regular_flow_spec.rb` | When consuming using multiple subscription groups and all of them are within time limits, we should never get 500 |
| `instrumentation/vendors/kubernetes/consuming_liveness_parallel_timeout_flow_spec.rb` | When consuming using multiple subscription groups and only one hangs, k8s listener should be able to detect that. |
| `instrumentation/vendors/kubernetes/consuming_liveness_timeout_flow_spec.rb` | When consuming takes more time then expected, we should see that in the status |
| `instrumentation/vendors/kubernetes/fenced_out_criticality_spec.rb` | When fenced out by a new instance, kubernetes listener should report this as a 500 |
| `instrumentation/vendors/kubernetes/liveness_all_good_spec.rb` | When all good, all should be ok |
| `instrumentation/vendors/kubernetes/non_existing_until_running_spec.rb` | If Karafka is configured but not started, the liveness probing should not work |
| `instrumentation/vendors/kubernetes/polling_liveness_timeout_flow_spec.rb` | When the polling happens less frequently than expected, we should get a timeout indication out of the probing |
| `instrumentation/vendors/kubernetes/swarm_liveness_all_good_spec.rb` | When all good, all should be ok in swarm |
| `instrumentation/vendors/kubernetes/swarm_liveness_not_controlling_often_spec.rb` | When we expect to control nodes more often that it happens, it should be reflected |
| `instrumentation/with_hammered_assignments_tracker_spec.rb` | AssignmentsTracker should remain stable and not crash when inspected concurrently from multiple consumer groups during message processing |
| `instrumentation/with_offset_querying_spec.rb` | Karafka when started and stopped should go through all the lifecycle stages |
| `instrumentation/with_replaced_monitor_spec.rb` | Karafka should allow for a monitor that can be used to run wrapped handling, as long as there is a delegation back to Karafka monitor afterwards. |
| `instrumentation/with_watermarks_querying_spec.rb` | Karafka should be able to query watermark offsets for multiple topics and partitions using a single consumer instance |
| `lags/consumption_delayed_spec.rb` | Karafka should correctly report consumption_lag when there is a delay in between publishing messages and their consumption |
| `lags/consumption_real_time_spec.rb` | Karafka should correctly report consumption_lag when we consume messages fast and it should never be bigger than couple hundred ms with the defaults for integration specs |
| `lags/processing_fast_spec.rb` | When processing data fast, the processing lag should not be big and things should be processed almost real time |
| `lags/processing_not_enough_threads_spec.rb` | In case there are not enough threads to parallelize work from multiple topics/partitions we can expect a processing lag, as work will wait in a queue to be picked up once resources are available |
| `lags/processing_slow_spec.rb` | When processing data slowly from a single partition of a single topic, we do not fetch more data from Kafka, thus the processing lag should not be big as there is no more data enqueued We keep the time numbers a bit higher than they could be as when running on the CI, sometimes there are small lags |
| `offset_management/commit_offset_after_exceeding_max_poll_spec.rb` | When we exceed max poll interval but did not poll yet, the async and sync offset commits should give us accurate representation of the ownership because of lost assignment check. |
| `offset_management/commit_offset_sync_when_no_offsets_spec.rb` | When we do not have any offsets committed, `store_offsets` and `store_offsets\!` should work as expected. `store_offsets\!` can be useful in cases when we do want to delegate the offset management to Karafka but still want to make sure that upon consumption start we own the assignment. In such cases `store_offsets\!` give us good knowledge of it. |
| `offset_management/from_latest_via_initial_offset_spec.rb` | Karafka should be able to start consuming from the latest offset whe set via initial_offset |
| `offset_management/from_latest_via_kafka_setting_spec.rb` | Karafka should be able to start consuming from the latest offset |
| `offset_management/messages_and_metadata_details_spec.rb` | Karafka messages data should be as defined here |
| `offset_management/offset_edge_cases_spec.rb` | Karafka should handle offset management edge cases properly |
| `offset_management/rebalance_commit_spec.rb` | Karafka flushes the offsets once in a while. When auto offset committing is disabled, we anyhow should commit the offset in rebalance |
| `offset_management/with_different_initial_offsets_many_groups_spec.rb` | Karafka should be able to consume many topics with different initial offset strategies and should handle that correctly for all the topics in many groups |
| `offset_management/with_different_initial_offsets_one_group_spec.rb` | Karafka should be able to consume many topics with different initial offset strategies and should handle that correctly for all the topics |
| `offset_management/with_marking_older_message_spec.rb` | When older message then the one we already marked is marked as processed, this should be ignored. We should only mark stuff moving forward. This aligns with librdkafka behaviour for offsets (see note): https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#at-least-once-processing |
| `offset_management/with_sync_marking_spec.rb` | Karafka should not raise any errors when we use mark_as_consumed\! |
| `pausing/manual_pause_on_a_revoked_partition_spec.rb` | When we pause a lost partition, it should not have any impact on us getting the data after we get it back (if we get it back) |
| `pausing/overwrite_one_pause_with_another_spec.rb` | Even if we are in a long active pause, a short pause should overwrite the long one |
| `pausing/pause_and_continue_without_offset_spec.rb` | We should be able to pause without any arguments and then pause will pause on the consecutive offset |
| `pausing/pause_early_and_skip_via_marking_spec.rb` | We should be able to pause the partition and still mark messages as consumed When another process would pick up the work, it should start from the last consumption marked |
| `pausing/pause_early_and_skip_via_seek_spec.rb` | We should be able to pause the partition and then seek to another offset After partition is un-paused, it should skip the messages we want to jump over |
| `pausing/pause_early_and_skip_when_resume_seek_spec.rb` | We should be able to pause the partition and then seek to another offset before we un-pause After partition is un-paused, it should skip the messages we want to jump over |
| `pausing/pause_prior_to_automatic_offset_and_mark_spec.rb` | When user pauses and marks by himself, we should not deal with this and let him do this. |
| `pausing/pause_prior_to_automatic_offset_spec.rb` | When user initiates a pause, the automatic offset management should not mark the last batch message as consumed automatically to prevent a scenario, where upon a rebalance during the pause the partition is gave away and processing starts now from the last offset prior to pause but from the automatic offset. |
| `pausing/pause_super_short_spec.rb` | When we pause for a time shorter than processing time, we should un-pause prior to the next poll We should not exceed two polls. First (before which we resume) may not get the data because after resuming we may not have enough time to start polling the given partition again. |
| `pausing/pausing_with_message_skip_spec.rb` | We should be able to pause the partition in a way that would allow us to skip a message. This is not something you want to do in production, nonetheless use-case like this should be supported. |
| `pausing/prolonging_manual_pause_spec.rb` | We should be able to prolong a manual pause that we did pause already and the times should add up |
| `pausing/seeking_offset_after_pausing_spec.rb` | After we pause and continue processing, we should be able to seek to previous offset and after un-pausing, we should be able to start from where we wanted (not from where we paused) |
| `pausing/using_manual_pause_with_backoff_defaults_spec.rb` | Important: this may not work as expected for a manual pause case. Please read the below message. When we pause a partition without providing the timeout, it should use the timeout defined in the retry settings. The case here is, that when processing is successful we reset the pause counter, so the retry is never acknowledged (like it is for cases with errors). This means that manual pausing will never use the automatic exponential backoff. |
| `pausing/using_manual_pause_with_non_backoff_defaults_spec.rb` | When we pause a partition without providing the timeout, it should use the timeout defined in the retry settings |
| `pausing/with_manual_pause_and_other_partitions_working_spec.rb` | When one partition is paused by us manually, we should be consuming others without interruption. Here we check, that the paused partition data is consumed as last (since we pause long enough) and that the other partition's data was consumed first. |
| `pausing/with_manual_pause_in_between_messages_spec.rb` | We should be able to manually pause for any time we want, bypassing the default pause time if we declare the pause time ourselves. |
| `pausing/with_manual_pause_same_message_loop_spec.rb` | We should be able pause on a message we've already seen from the batch and should always start from it after resuming. This will mean, we just process same message over and over again. |
| `pausing/with_pause_and_resume_after_reclaim_spec.rb` | We can pause and after revocation and re-claim we should continue processing like pause did not happen. We should not keep the pause after a rebalance. Messages that were consumed by another process should not be re-consumed. |
| `pausing/with_pause_and_resume_spec.rb` | We should be able to do things like pausing and resume in the same consumer flow. This should not break the ordering |
| `pro/cli/declaratives/health_multi_broker_sequential/all_healthy_multi_broker_spec.rb` | Integration test for topics health command when all topics are healthy |
| `pro/cli/declaratives/health_multi_broker_sequential/critical_rf_one_spec.rb` | Integration test for topics health command detecting RF=1 (no redundancy) |
| `pro/cli/declaratives/health_multi_broker_sequential/critical_zero_fault_tolerance_multi_broker_spec.rb` | Integration test for topics health command detecting RF <= min.insync (zero fault tolerance) |
| `pro/cli/declaratives/health_multi_broker_sequential/mixed_issues_multi_broker_spec.rb` | Integration test for topics health command with mixed critical and warning issues |
| `pro/cli/declaratives/health_multi_broker_sequential/single_broker_staging_spec.rb` | Integration test for topics health command in single-broker cluster (common staging setup) This test expects to detect RF=1 as a critical issue, which is typical in single-broker setups |
| `pro/cli/declaratives/health_multi_broker_sequential/skips_internal_topics_multi_broker_spec.rb` | Integration test verifying that health command skips internal Kafka topics (starting with __) |
| `pro/cli/declaratives/health_multi_broker_sequential/warning_low_durability_multi_broker_spec.rb` | Integration test for topics health command detecting min.insync=1 (low durability warning) |
| `pro/cli/declaratives/health_multi_broker_sequential/with_detailed_exitcode_healthy_multi_broker_spec.rb` | Integration test for topics health command with --detailed-exitcode when all topics are healthy |
| `pro/cli/declaratives/health_multi_broker_sequential/with_detailed_exitcode_issues_spec.rb` | Integration test for topics health command with --detailed-exitcode when issues are found |
| `pro/cli/declaratives/ignore_patterns_in_declaratives_spec.rb` | karafka declarative topics management should ignore pattern topics because they are dynamic subscriptions for which we do not know the target name until the regexp match |
| `pro/cli/declaratives/strict_setup_requirement_spec.rb` | When we have strict_declarative_topics set to true, we should ensure all non-pattern definitions of topics have their declarative references |
| `pro/cli/parallel_segments/collapse/with_conclusive_offsets_spec.rb` | Karafka parallel segments collapse should work when there is a topic that was consumed with parallel segments and their offset assignment is even |
| `pro/cli/parallel_segments/collapse/with_force_and_inconclusive_offsets_spec.rb` | Karafka parallel segments collapse should work with --force when offsets are inconclusive by picking the lowest offset for each partition |
| `pro/cli/parallel_segments/collapse/with_inconclusive_offsets_spec.rb` | Karafka parallel segments collapse should fail when parallel segments have inconclusive offsets |
| `pro/cli/parallel_segments/collapse/with_multiple_segments_spec.rb` | Karafka parallel segments collapse should work when there are multiple (10) segments |
| `pro/cli/parallel_segments/collapse/with_multiple_topics_spec.rb` | Karafka parallel segments collapse should work when there are multiple topics |
| `pro/cli/parallel_segments/collapse/with_no_consumer_groups_spec.rb` | Karafka parallel segments collapse should inform user when no consumer groups with parallel segments exist |
| `pro/cli/parallel_segments/collapse/with_nonexistent_consumer_group_spec.rb` | Karafka parallel segments collapse should fail when specified consumer group doesn't exist |
| `pro/cli/parallel_segments/collapse/with_one_segment_spec.rb` | Karafka parallel segments collapse should inform when only one parallel segment is found |
| `pro/cli/parallel_segments/collapse/with_specific_consumer_groups_spec.rb` | Karafka parallel segments collapse should work with specific consumer groups selected |
| `pro/cli/parallel_segments/collapse/without_segments_spec.rb` | Karafka parallel segments collapse should work when there are no segments to collapse |
| `pro/cli/parallel_segments/distribute/with_already_set_offsets_spec.rb` | Karafka parallel segments distribute should fail when offsets are already set |
| `pro/cli/parallel_segments/distribute/with_empty_topics_spec.rb` | Karafka parallel segments distribute should work with empty topics |
| `pro/cli/parallel_segments/distribute/with_force_and_existing_offsets_spec.rb` | Karafka parallel segments distribute should work with --force even when offsets are already set |
| `pro/cli/parallel_segments/distribute/with_multiple_segments_spec.rb` | Karafka parallel segments distribute should work with multiple segments |
| `pro/cli/parallel_segments/distribute/with_multiple_topics_spec.rb` | Karafka parallel segments distribute should work with multiple topics |
| `pro/cli/parallel_segments/distribute/with_no_consumer_groups_spec.rb` | Karafka parallel segments distribute should inform user when no consumer groups with parallel segments exist |
| `pro/cli/parallel_segments/distribute/with_nonexistent_consumer_group_spec.rb` | Karafka parallel segments distribute should fail when specified consumer group doesn't exist |
| `pro/cli/parallel_segments/distribute/with_one_segment_spec.rb` | Karafka parallel segments distribute should inform when only one parallel segment is found |
| `pro/cli/parallel_segments/distribute/with_specific_consumer_groups_spec.rb` | Karafka parallel segments distribute should work with specific consumer groups selected |
| `pro/cli/parallel_segments/distribute/without_segments_spec.rb` | Karafka parallel segments distribute should work when there are no segments to distribute |
| `pro/cli/parallel_segments/invalid_action_spec.rb` | Karafka parallel segments should fail with ArgumentError when an invalid action is provided |
| `pro/cli/parallel_segments/reset_combined_operation_spec.rb` | Karafka parallel segments reset should perform both collapse and distribute operations |
| `pro/cli/swarm/with_cg_exclusion_spec.rb` | Karafka in swarm should properly handle consumer group exclusion flag and skip processing the specified consumer groups |
| `pro/cli/swarm/with_cg_inclusion_spec.rb` | Karafka in swarm should properly handle consumer group inclusion flag and only process the specified consumer groups |
| `pro/cli/swarm/with_nothing_matching_spec.rb` | Karafka in swarm should raise well nested validated errors with the swarm context |
| `pro/cli/swarm/with_sg_exclusion_spec.rb` | Karafka in swarm should raise well nested validated errors with the swarm context This spec will cause it to crash if any validation that would fail is happening in the nodes post-fork |
| `pro/cli/swarm/with_sg_inclusion_spec.rb` | Karafka in swarm should properly handle subscription group inclusion flag and only process the specified subscription groups |
| `pro/cli/swarm/with_topic_exclusion_spec.rb` | Karafka in swarm should properly handle topic exclusion flag and skip processing the specified topics |
| `pro/cli/swarm/with_topic_inclusion_spec.rb` | Karafka in swarm should properly handle topic inclusion flag and only process the specified topics |
| `pro/consumption/adaptive_iterator/with_adaptive_estimated_processing_limited_spec.rb` | With adaptive margin, if the max cost of message would cause reaching max poll, we should seek back |
| `pro/consumption/adaptive_iterator/with_fast_exit_on_stop_spec.rb` | We should stop fast and not process all in batch and offset position should be preserved |
| `pro/consumption/adaptive_iterator/without_adaptive_stable_processing_limited_spec.rb` | When processing messages with iterator enabled and reaching max.poll.interval, we should never get any errors and processing should be consecutive |
| `pro/consumption/adaptive_iterator/without_anything_running_spec.rb` | When processing messages with iterator enabled but no features enabled, it should work |
| `pro/consumption/at_most_once_on_error_spec.rb` | When marking as consumed before the error, message should be skipped as it should be considered consumed |
| `pro/consumption/cleaner/automatic_cleaning_on_iterator_spec.rb` | We should be able to use automatic cleaning to get rid of the payload We should fail when trying to deserialize a cleaned message |
| `pro/consumption/cleaner/external_prepend_compatibility_spec.rb` | We should be able to use cleaning functionality even when external libraries prepend modules to Messages#each (like DataDog tracing) This tests the fix for: https://github.com/DataDog/dd-trace-rb/issues/4867 |
| `pro/consumption/cleaner/fail_on_dlq_dispatch_of_cleaned_spec.rb` | Attempt to dispatch to DLQ a cleaned message should always fail |
| `pro/consumption/cleaner/manual_cleaning_on_each_spec.rb` | We should be able to iterate over messages and run cleaning both with and without prior deserialization in a manual mode We should fail when trying to deserialize a cleaned message |
| `pro/consumption/cleaner/metadata_cleaning_on_iterator_spec.rb` | We should be able to use automatic cleaning to get rid of the key and headers We should fail when trying to deserialize a cleaned details |
| `pro/consumption/cleaner/without_metadata_spec.rb` | We should be able to clean only payload when needed |
| `pro/consumption/direct_assignments/all_partitions_spec.rb` | We should be able to assign to ourselves direct ownership of all partitions |
| `pro/consumption/direct_assignments/beyond_poll_interval_spec.rb` | When using direct assignments, we should be able to ignore max poll interval. |
| `pro/consumption/direct_assignments/from_earliest_spec.rb` | We should be able to assign to ourselves direct ownership of partitions we are interested in |
| `pro/consumption/direct_assignments/marking_as_consumed_spec.rb` | We should be able to mark as consumed |
| `pro/consumption/direct_assignments/multi_topic_in_one_cg_spec.rb` | We should be able to assign to ourselves multiple topics in one CG |
| `pro/consumption/direct_assignments/multi_topic_stream_merging_spec.rb` | When using direct assignments, we should be able to maintain a window and merge data from several streams Note, that this example is a simplification and does not take into consideration lags caused by sudden data spikes in one of the topics. It aims to illustrate the basics of stream merging. What we do here is simple: we publish users creation events and we publish their actions events Events cannot happen before user is created but since we have two topics, their delivery times may vary. We merge those two topics and dispatch the target data into "users_events" target topic This example is simplified. Offset management is not correct for production-grade flow. |
| `pro/consumption/direct_assignments/multi_topic_transactional_spec.rb` | We should be able to open a multi-topic transaction as long as we know which consumer is the source of underlying messages so we can track their offsets This works with an assumption that we have one producer @note There is code for consumer marking commented out because we want to ensure, that we mark as consumed only in the transaction and that it works as expected but in a production-grade system you also want to always synchronize internal state of consumers instances. |
| `pro/consumption/direct_assignments/starting_from_where_it_left_spec.rb` | We should start from where the offset was with direct assignments |
| `pro/consumption/direct_assignments/transactions_support_spec.rb` | We should be able to assign what we want and mark offsets in a transaction |
| `pro/consumption/eofed/vps_with_eof_spec.rb` | When using virtual partitions with eof, each VP should receive its own `#eofed` execution. |
| `pro/consumption/fast_non_blocking_unlock_spec.rb` | Because when we set time based locks that can bypass default ticking, we should be able to jump over ticking faster that the ticking interval We do not not instrument wait unlocks on the queue but since listener events are published, we can use next poll start to notice that it unlocks faster than the tick interval but slower than post-job immediate unlock |
| `pro/consumption/from_earliest_direct_iterator_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) exactly the same way with pro as it does without |
| `pro/consumption/from_earliest_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) exactly the same way with pro as it does without |
| `pro/consumption/inline_insights/from_earliest_required_spec.rb` | When insights are required, we should not proceed without them Because initial metrics fetch is slightly unpredictable (different with KRaft) and can be impacted by the cpu load, we simulate lack by patching the tracker so first first 10 seconds of running the process it returns no data |
| `pro/consumption/inline_insights/from_earliest_spec.rb` | We should be able to get the insights and use them via the API when they are defined In Pro despite extra option, should behave same as in OSS when no forced required |
| `pro/consumption/inline_insights/multi_topic_partition_spec.rb` | Inline Insights should ship correct partition and topic details and not mix them In Pro despite extra option, should behave same as in OSS when no forced required |
| `pro/consumption/inline_insights/post_involuntary_revocation_spec.rb` | When given partition is lost involuntary, we still should have last info In Pro despite extra option, should behave same as in OSS when no forced required |
| `pro/consumption/inline_insights/post_revocation_insights_lrj_spec.rb` | When given partition is revoked for LRJ, we should still have its last available statistics In Pro despite extra option, should behave same as in OSS when no forced required |
| `pro/consumption/inline_insights/without_being_enabled_spec.rb` | Karafka should not have the insights methods when insights are not enabled In Pro despite extra option, should behave same as in OSS when no forced required |
| `pro/consumption/multiplexing/dynamic_downscale_spec.rb` | When we start from too many connections, we should effectively go down and establish baseline |
| `pro/consumption/multiplexing/dynamic_pristine/upscale_to_big_setup_spec.rb` | This is a special spec that ensures librdkafka stability It is related to multiplexing that's why it is here. This spec covers this case: https://github.com/confluentinc/librdkafka/issues/4783 This spec runs 1.5 minute but it is expected to ensure stability of the multiplexing execution If the issue persists, this process will fail. This spec needs to run alone because otherwise the `top` command result can be biased by other specs running on the same machine alongside |
| `pro/consumption/multiplexing/dynamic_with_max_start_spec.rb` | When we start from max connections but there is is no space to grow, we should keep it and not downscale |
| `pro/consumption/multiplexing/dynamic_with_mid_start_spec.rb` | When we start from few connections but there is space to grow, we should |
| `pro/consumption/multiplexing/dynamic_with_min_start_spec.rb` | When we start from one connection but there is space to grow, we should |
| `pro/consumption/multiplexing/static_with_many_topics_and_partitions_spec.rb` | We should be able to have static group membership with cooperative on many topics and partitions with multiplexing starting with 1 without any issues. |
| `pro/consumption/multiplexing/with_patterns_spec.rb` | We should be able to multiplex a pattern and things should operate as expected |
| `pro/consumption/multiplexing/with_same_topic_in_multiple_subscription_groups_spec.rb` | When using separate subscription groups, each should have it's own underlying client and should operate independently for data fetching and consumption |
| `pro/consumption/offset_metadata/basic_automatic_flow_overwritten_last_marking_spec.rb` | When offset metadata is stored and we mark each, later marking as consumed on the same location should not end up with us loosing metadata |
| `pro/consumption/offset_metadata/basic_automatic_flow_overwritten_spec.rb` | When offset metadata is stored but a custom forced value is used, the forced on should be used. |
| `pro/consumption/offset_metadata/basic_automatic_flow_spec.rb` | Karafka should be able to take the offset from the `store_metadata_offset` invocation and use it |
| `pro/consumption/offset_metadata/basic_flow_spec.rb` | Karafka should be able to get offset metadata when non-existing, save it and force refresh with defaults and cache on @note We use mark_as_consumed\! to instantly flush |
| `pro/consumption/offset_metadata/dlq/recovery_with_metadata_spec.rb` | When stored metadata exists, it should be used in the DLQ flow. |
| `pro/consumption/offset_metadata/ftr/usage_from_filter_spec.rb` | Karafka should allow for usage of the offset metadata from filters as well. Since filters can accept routing topic and partition reference from the builder, they can store them and use the fetcher to obtain needed offset metadata. This can allow them to use the pre-rebalance context awareness. |
| `pro/consumption/offset_metadata/iterative_marking_cached_flow_spec.rb` | Offset should move forward with metadata but because we cache result, it should not update in memory as there is no rebalance. |
| `pro/consumption/offset_metadata/iterative_marking_no_cache_flow_spec.rb` | Offset should move forward with metadata and should be deserialized with a custom deserializer. Because we disabled the cache completely, we should get linear auto-refreshed data |
| `pro/consumption/offset_metadata/rebalanced_cache_refresh_spec.rb` | When rebalance occurs, even if we had cache, it should be invalidated |
| `pro/consumption/offset_metadata/vp/current_matching_spec.rb` | When we use current matching strategy on the given offset that can be materialized, it should use the most recently used offset metadata even if it was assigned to a different offset |
| `pro/consumption/offset_metadata/vp/exact_automatic_offset_with_marked_already_spec.rb` | When we mark as consumed everything, automatic marking should not overwrite the metadata |
| `pro/consumption/offset_metadata/vp/exact_matching_spec.rb` | When we use exact matching strategy on the given offset that can be materialized, it should be used even if later we use a different higher that cannot be materialized |
| `pro/consumption/offset_metadata/vp/exact_with_higher_matching_on_spec.rb` | When we use exact matching strategy for virtual partitions, metadata should match the most recent consecutive offset and not the latest marking offset kept in-memory In case materialized offset is behind the one that we marked due to the VPs distribution, "exact" value will be assigned from the highest offset of a virtual partition |
| `pro/consumption/parallel_segments/basic_distribution_flow_spec.rb` | Messages should be distributed to consumers based on their group_id assignment using the partitioner and reducer |
| `pro/consumption/parallel_segments/error_recovery_flow_spec.rb` | After errors, consumers should be able to recover and continue processing in their assigned groups without receiving messages from other groups |
| `pro/consumption/parallel_segments/group_assignment_with_multiple_partitions_spec.rb` | Messages from multiple partitions should be assigned to the same group when they match the group criteria |
| `pro/consumption/parallel_segments/mom/basic_distribution_flow_spec.rb` | With manual offset management, messages should be distributed correctly but no automatic marking should occur |
| `pro/consumption/parallel_segments/mom/with_full_filtering_spec.rb` | With manual offset management and all messages filtered in one segment, no offsets should be marked in this segment |
| `pro/consumption/parallel_segments/mom/without_offset_marking_spec.rb` | With manual offset management, no offsets should be automatically marked even when all messages are filtered |
| `pro/consumption/parallel_segments/with_active_job/job_distribution_spec.rb` | ActiveJob jobs should be distributed to consumers based on their segment group assignment |
| `pro/consumption/parallel_segments/with_all_filtered_offset_marking_spec.rb` | When all messages are filtered out, offsets should be marked as consumed to prevent lag |
| `pro/consumption/parallel_segments/with_dlq/error_dispatching_flow_spec.rb` | Error dispatching to DLQ should respect the parallel segment group assignment using segment-specific DLQ topics |
| `pro/consumption/parallel_segments/with_dlq/filtered_message_handling_spec.rb` | Filtered messages from parallel segments should not be sent to DLQ as they are part of normal operation |
| `pro/consumption/parallel_segments/with_dynamic_partitioner_spec.rb` | Dynamic partitioners should be able to change distribution rules mid-flow and consumers should adapt |
| `pro/consumption/parallel_segments/with_error_in_partitioner_spec.rb` | Errors in partitioner should be properly handled and should not crash the consumer |
| `pro/consumption/parallel_segments/with_error_in_reducer_spec.rb` | Errors in reducer should be properly handled and should not crash the consumer |
| `pro/consumption/parallel_segments/with_full_batch_filtering_auto_marking_spec.rb` | With automatic offset management, offsets should be marked even when all messages are filtered |
| `pro/consumption/parallel_segments/with_long_running_jobs/processing_continuity_spec.rb` | Long-running jobs should maintain their parallel segment group assignment throughout execution |
| `pro/consumption/parallel_segments/with_long_running_jobs/rebalance_continuity_spec.rb` | During rebalancing, long-running jobs should maintain their parallel segment assignments |
| `pro/consumption/parallel_segments/with_long_running_jobs/revoke_continuity_spec.rb` | After partition revocation, reacquired partitions should resume with consistent parallel segment assignment |
| `pro/consumption/parallel_segments/with_many_groups_distribution_spec.rb` | With many segment groups, distribution should remain balanced and consistent with the reducer formula |
| `pro/consumption/parallel_segments/with_many_topics_in_the_consumer_group_spec.rb` | With many topics in the consumer group, they all should work correctly and work should be distributed as expected. |
| `pro/consumption/parallel_segments/with_nil_key_handling_spec.rb` | Messages with nil keys should be handled gracefully and assigned to a consistent group |
| `pro/consumption/parallel_segments/with_pattern_matched_topics_spec.rb` | Parallel segments should work correctly with dynamically matched topics from patterns |
| `pro/consumption/parallel_segments/with_rebalancing_spec.rb` | During rebalancing, parallel segment assignment should remain consistent for the same messages |
| `pro/consumption/parallel_segments/with_throttling_spec.rb` | Throttling should be applied per parallel segment group rather than globally |
| `pro/consumption/parallel_segments/with_transactions/offset_marking_continuity_spec.rb` | Offset marking within transactions should maintain continuity in parallel segments |
| `pro/consumption/parallel_segments/with_transactions/with_aborted_transaction_spec.rb` | When a transaction is aborted, the parallel segments should handle this properly without marking offsets |
| `pro/consumption/parallel_segments/with_uneven_distribution_spec.rb` | Even with uneven distribution, each consumer should only process messages for its assigned group |
| `pro/consumption/parallel_segments/with_virtual_partitions/combined_distribution_flow_spec.rb` | When using both features, messages should first be distributed by parallel segments then by virtual partitions |
| `pro/consumption/parallel_segments/with_virtual_partitions/nested_partitioning_spec.rb` | Virtual partitions should correctly subdivide workloads within parallel segment groups |
| `pro/consumption/patterns/complex_regexp_match_spec.rb` | Karafka should match over complex regexp as long as they comply with the following format: https://github.com/ccxvii/minilibs/blob/master/regexp.c |
| `pro/consumption/patterns/negative_lookup_spec.rb` | We should be able to use posix negative lookup regexps to match all except certain topics |
| `pro/consumption/patterns/post_detected_with_default_deserializers_spec.rb` | Karafka should use default deserializers when defaults exist and topic is detected |
| `pro/consumption/patterns/postfix_match_spec.rb` | Karafka should match over postfix regexp |
| `pro/consumption/patterns/prefix_match_spec.rb` | Karafka should match over postfix regexp |
| `pro/consumption/patterns/unexpected_patterns_loop_spec.rb` | When consumer uses patterns and same pattern matches the DLQ, messages may be self-consumed creating endless loop. Not something you want. |
| `pro/consumption/patterns/with_matching_appearing_topic_spec.rb` | Karafka should be able to match non-existing topic when it appears later on. |
| `pro/consumption/patterns/with_matching_existing_topic_spec.rb` | Karafka should be able to match existing topic with pattern when starting processing and given topic already exists |
| `pro/consumption/patterns/with_multiple_existing_topics_spec.rb` | Karafka should be able to match existing topics when regexp specifies all of them No rebalances should occur in between detections |
| `pro/consumption/patterns/with_same_named_pattern_in_multiple_groups_spec.rb` | Karafka should work as expected when having same matching used in multiple CGs. |
| `pro/consumption/periodic_jobs/during_error_spec.rb` | By default ticking should not happen if we are in a recovery flow |
| `pro/consumption/periodic_jobs/during_error_with_recovery_spec.rb` | If we explicitely want to tick during recovery, it should be possible |
| `pro/consumption/periodic_jobs/filtering/on_constant_idle_spec.rb` | When we idle all the time on incoming data, we should never tick It is end user FTR makes this decision to skip |
| `pro/consumption/periodic_jobs/filtering/on_idle_and_nothing_spec.rb` | When we idle all and move on but no more messages for a while, tick should kick in |
| `pro/consumption/periodic_jobs/initialized_prior_to_ticking_spec.rb` | We should run the initialized hook prior to ticking |
| `pro/consumption/periodic_jobs/long_running_jobs/no_parallel_ticking_when_not_during_pause_spec.rb` | When running on LRJ, ticking should not happen alongside long processing when it is not turned on (default is off). Note that LRJ consume can start when tick is running. Only the other way around is not allowed |
| `pro/consumption/periodic_jobs/long_running_jobs/non_blocking_ticking_spec.rb` | When running on LRJ, ticking should not block and not cause max.poll.interval.ms to kick in |
| `pro/consumption/periodic_jobs/long_running_jobs/parallel_ticking_when_during_pause_spec.rb` | When running on LRJ, ticking should not happen alongside long processing if the long running job is running at the moment |
| `pro/consumption/periodic_jobs/long_running_jobs/ticking_overlaping_consuming_spec.rb` | When running LRJ and ticking, ticking is not synchronized with LRJ (unless synchronized via mutex). This means, that it should be possible to have a long living ticking that started when nothing was happening but meanwhile things started. |
| `pro/consumption/periodic_jobs/marking_from_ticking_spec.rb` | We should be able to mark as consumed from ticking if we want to |
| `pro/consumption/periodic_jobs/non_blocking_locking_with_periodic_unlock_spec.rb` | We should be able to lock in a non-blocking fashion our subscription group (or any other) and resume it from a periodic job on a different group Note: since we do not run periodics on the same go as a job, we cannot unlock from the same group. Time based locks are a different case covered in a different spec. |
| `pro/consumption/periodic_jobs/not_during_error_with_recovery_spec.rb` | Ticking should work before and after the recovery but not during in the default setup. |
| `pro/consumption/periodic_jobs/on_a_non_existing_topic_spec.rb` | If we try to tick on a non-existing topic, we should not. |
| `pro/consumption/periodic_jobs/partially_periodic_spec.rb` | We should be able to have part of topics with periodics in the same SG |
| `pro/consumption/periodic_jobs/pause_from_tick_spec.rb` | We should be able to pause from ticking |
| `pro/consumption/periodic_jobs/pause_resume_flow_from_tick_spec.rb` | We should be able to control flow from ticking |
| `pro/consumption/periodic_jobs/resume_from_tick_spec.rb` | We should be able to resume from ticking |
| `pro/consumption/periodic_jobs/seek_from_tick_spec.rb` | We should be able to seek from tick |
| `pro/consumption/periodic_jobs/virtual_partitions/with_appearing_virtual_partitions_spec.rb` | When we tick on one VP and more appear, they should start ticking as well |
| `pro/consumption/periodic_jobs/virtual_partitions/with_existing_virtual_partitions_spec.rb` | When we tick on existing VPs, they all should tick |
| `pro/consumption/periodic_jobs/virtual_partitions/with_not_used_virtual_partitions_spec.rb` | When we tick on VPs that were not created, we should only create one as we do not know how many we should pre-allocate |
| `pro/consumption/periodic_jobs/with_constant_short_poll_spec.rb` | When we get constant stream of data on other partition, our unused partition should anyhow tick only once every tick interval |
| `pro/consumption/periodic_jobs/with_constant_stream_of_messages_spec.rb` | When we get constant stream of data, we should not tick |
| `pro/consumption/periodic_jobs/with_custom_frequencies_spec.rb` | We should be able to define custom frequencies and it should work. It should use defaults where nothing defined |
| `pro/consumption/periodic_jobs/with_eof_status_spec.rb` | We should be able to maintain eof status when ticking |
| `pro/consumption/periodic_jobs/with_lost_partition_spec.rb` | When periodic ticking is on but we have lost a partition, we should stop ticking on that partition but we should continue on the one that we still have |
| `pro/consumption/periodic_jobs/with_other_consumer_spec.rb` | When periodic ticking is on but other consumers have some partitions, we should not tick on them as they are not assigned to us |
| `pro/consumption/periodic_jobs/with_saturated_execution_spec.rb` | When consume takes longer than the periodic interval, tick should still execute after consume finishes and should not be skipped |
| `pro/consumption/periodic_jobs/with_used_switched_spec.rb` | When consumer is being used at least once, we should be able to see it when ticking |
| `pro/consumption/periodic_jobs/without_any_data_ever_many_partitons_spec.rb` | When periodic gets an assignment it should tick in intervals despite never having any data It should work as expected also for many partitions |
| `pro/consumption/periodic_jobs/without_any_data_ever_spec.rb` | When periodic gets an assignment it should tick in intervals despite never having any data It should never indicate that it was used when it was not |
| `pro/consumption/periodic_jobs/without_ticking_on_pause_spec.rb` | When periodic jobs are configured not to tick when partition is paused we should not tick then Keep in mind, LRJ always pauses so you won't have ticking on it if this is set like this |
| `pro/consumption/piping/with_key/transactional_with_marking_spec.rb` | We should be able to pipe data to a different topic and it should be received Operating in transactional mode with marking as consumed inside transaction. |
| `pro/consumption/piping/with_key/with_alteration_to_headers_spec.rb` | We should be able to define extra method that we can use to alter headers |
| `pro/consumption/piping/with_key/without_any_changes_spec.rb` | We should be able to pipe data to a different topic and it should be received |
| `pro/consumption/piping/with_key/without_any_changes_transactional_spec.rb` | We should be able to pipe data to a different topic and it should be received Operating in transactional mode |
| `pro/consumption/piping/without_key/with_alteration_to_headers_spec.rb` | We should be able to define extra method that we can use to alter headers |
| `pro/consumption/piping/without_key/without_any_changes_spec.rb` | We should be able to pipe data to a different topic and it should be received |
| `pro/consumption/piping/without_key/without_any_changes_transactional_spec.rb` | We should be able to pipe data to a different topic and it should be received Operating in transactional mode |
| `pro/consumption/scheduling/assignments_aware_scheduling_spec.rb` | Example scheduler that evenly distributes work coming from multiple topics. It allocates workers evenly so each topic has resources. When rebalance occurs, the distribution ratio will change and will be updated |
| `pro/consumption/scheduling/full_ftr_with_idle_in_scheduler_spec.rb` | When using a custom scheduler we should by no means use consume scheduling API for idle jobs |
| `pro/consumption/scheduling/with_multiple_threads_but_finished_state_based_spec.rb` | We should be able to build a scheduler that can base further job scheduling based on the other jobs execution flow. While this scheduler is not explicitely single-threaded, it waits on previous job to finish before enqueuing more. This "makes it" single threaded. |
| `pro/consumption/scheduling/with_single_thread_materialization_scheduler_spec.rb` | When using a scheduler that would allow only a single thread, despite having more, we should never use them in any way |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom/with_non_recoverable_job_error_spec.rb` | Karafka should allow to run long AJ jobs with MOM, VPs and LRJ because we collapse upon errors. |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom/with_recoverable_slow_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom_vp/with_non_recoverable_job_error_spec.rb` | Karafka should allow to run long AJ jobs with MOM, VPs and LRJ because we collapse upon errors. |
| `pro/consumption/strategies/aj/dlq_ftr_lrj_mom_vp/with_recoverable_slow_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/dlq_ftr_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/dlq_ftr_mom/regular_processing_without_errors_spec.rb` | Karafka should be able to just process all the jobs Throttling should slow things down |
| `pro/consumption/strategies/aj/dlq_ftr_mom/with_non_recoverable_job_error_spec.rb` | AJ is only partially mom, because we actually mark AJ jobs as consumed after each. For this strategy it means, that applicability of DLQ will be regular. |
| `pro/consumption/strategies/aj/dlq_ftr_mom/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ Throttling should not impact order, etc |
| `pro/consumption/strategies/aj/dlq_ftr_mom_vp/regular_processing_without_errors_spec.rb` | Karafka should be able to just process all the jobs Throttling should slow things down |
| `pro/consumption/strategies/aj/dlq_ftr_mom_vp/with_non_recoverable_job_error_spec.rb` | AJ is only partially mom, because we actually mark AJ jobs as consumed after each. For this strategy it means, that applicability of DLQ will be regular after the VP collapse. |
| `pro/consumption/strategies/aj/dlq_ftr_mom_vp/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ Throttling should not impact order, etc |
| `pro/consumption/strategies/aj/dlq_lrj_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/dlq_lrj_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_lrj_mom/with_non_recoverable_slow_jobs_error_spec.rb` | Karafka should run the same strategy for AJ DLQ LRJ MOM as for DLQ LRJ MOM. For AJ based workloads it means marking after each. This means, we will end up in a loop. This resembles a non MOM standard flow for LRJ DLQ (management of work is on AJ base) because from the end user perspective the offset management is not manual - it is delegated to the framework and realized via the ActiveJob consumer itself. |
| `pro/consumption/strategies/aj/dlq_lrj_mom/with_recoverable_slow_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/dlq_lrj_mom/without_dispatch_to_dlq_spec.rb` | Karafka should run the same strategy for AJ DLQ LRJ MOM as for DLQ LRJ MOM. Karafka should NOT dispatch the message to the DLQ topic because it's set to false. |
| `pro/consumption/strategies/aj/dlq_lrj_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_lrj_mom_vp/with_non_recoverable_job_error_spec.rb` | Karafka should allow to run long AJ jobs with MOM, VPs and LRJ because we collapse upon errors. |
| `pro/consumption/strategies/aj/dlq_lrj_mom_vp/with_recoverable_slow_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/dlq_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/dlq_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_mom/tagging_with_job_class_spec.rb` | Karafka Pro should automatically assign the tag of job class that is executed |
| `pro/consumption/strategies/aj/dlq_mom/with_error_type_based_dispatcher_spec.rb` | Complex error strategies should work with ActiveJob |
| `pro/consumption/strategies/aj/dlq_mom/with_non_recoverable_job_error_spec.rb` | Karafka should run the same strategy for AJ DLQ MOM as for DLQ MOM. For AJ based workloads it means marking after each. This means, we will end up in a loop. This resembles a non MOM standard flow for DLQ (management of work is on AJ base) because from the end user perspective the offset management is not manual - it is delegated to the framework and realized via the ActiveJob consumer itself. |
| `pro/consumption/strategies/aj/dlq_mom/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/dlq_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another |
| `pro/consumption/strategies/aj/dlq_mom_vp/with_non_recoverable_job_error_spec.rb` | Karafka should run the same strategy for AJ DLQ MOM VP as for AJ DLQ MOM as we collapse the state and can move in a linear manner. |
| `pro/consumption/strategies/aj/dlq_mom_vp/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move on without publishing anything to the DLQ |
| `pro/consumption/strategies/aj/ftr_lrj_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/ftr_lrj_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another Throttling may throttle but should not impact order or anything else |
| `pro/consumption/strategies/aj/ftr_lrj_mom/with_non_recoverable_slow_jobs_error_spec.rb` | We should end up in a loop and not move forward. Throttling should not interrupt this |
| `pro/consumption/strategies/aj/ftr_lrj_mom/with_recoverable_slow_jobs_error_spec.rb` | Karafka should recover from this error and move on. Throttling should not impact order or anything else. |
| `pro/consumption/strategies/aj/ftr_lrj_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another and throttling should not create any problems aside from throttling |
| `pro/consumption/strategies/aj/ftr_lrj_mom_vp/with_non_recoverable_job_error_spec.rb` | Karafka should be stuck because we end up in a loop processing and failing |
| `pro/consumption/strategies/aj/ftr_lrj_mom_vp/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move forward slowing things down throttling |
| `pro/consumption/strategies/aj/ftr_mom/offset_marking_on_long_running_stop_spec.rb` | Karafka upon long running jobs shutdown in this scenario, should early stop but should not mark the non-processed messages as consumed. |
| `pro/consumption/strategies/aj/ftr_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another and throttling should not create any problems aside from throttling |
| `pro/consumption/strategies/aj/ftr_mom/with_non_recoverable_job_error_spec.rb` | Karafka should be stuck because we end up in a loop processing and failing |
| `pro/consumption/strategies/aj/ftr_mom/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move forward slowing things down throttling |
| `pro/consumption/strategies/aj/ftr_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the jobs one after another and throttling should not create any problems aside from throttling |
| `pro/consumption/strategies/aj/ftr_mom_vp/with_non_recoverable_job_error_spec.rb` | Karafka should be stuck because we end up in a loop processing and failing |
| `pro/consumption/strategies/aj/ftr_mom_vp/with_recoverable_jobs_error_spec.rb` | Karafka should recover from this error and move forward slowing things down throttling |
| `pro/consumption/strategies/aj/mom/parallel_current_attributes_coop_spec.rb` | Karafka should correctly assign and manage the current attributes in multiple threads |
| `pro/consumption/strategies/aj/mom/via_patterns_spec.rb` | Karafka should allow to define ActiveJob patterns via `#active_job_pattern` and those should be picked up and operable even when topics are created after the subscription starts |
| `pro/consumption/strategies/aj/mom/with_envelope_spec.rb` | Karafka should be able to process ActiveJob jobs wrapped in a custom envelope This demonstrates a common pattern where all Kafka messages must follow a specific format with additional metadata wrapping the actual job payload Custom deserializer that wraps jobs in an envelope with metadata This extends the default Karafka ActiveJob deserializer |
| `pro/consumption/strategies/aj/mom/with_multiple_custom_partitioners_spec.rb` | Karafka should use different partitioners and karafka options for jobs and not mutate in between |
| `pro/consumption/strategies/aj/mom/with_variant_producers_spec.rb` | We should be able to define producer variants and use them to dispatch different jobs based on our preferences |
| `pro/consumption/strategies/aj/mom_vp/execution_on_failing_saturated_spec.rb` | Karafka when running VPs with AJ and being saturated, should run further jobs if the first job in the queue failed because we use virtual offset management for handling this scenario. |
| `pro/consumption/strategies/dlq/complex_error_classification_spec.rb` | DLQ should handle complex error classification scenarios including error chaining, nested errors, timeout-based errors, and custom error hierarchies. |
| `pro/consumption/strategies/dlq/default/details_dispatch_transfer_spec.rb` | When DLQ transfer occurs, payload and many other things should be transferred to the DLQ topic. |
| `pro/consumption/strategies/dlq/default/details_dispatch_transfer_with_enhancement_spec.rb` | When DLQ transfer occurs, we should be able to build our own payload and headers via `#enhance_dlq_message` |
| `pro/consumption/strategies/dlq/default/dispatch_instrumentation_spec.rb` | When DLQ delegation happens, Karafka should emit appropriate event. |
| `pro/consumption/strategies/dlq/default/error_tracking/assignment_matching_immediate_dispatch_spec.rb` | Same trace id should be present when instrumenting errors and during the DLQ dispatch |
| `pro/consumption/strategies/dlq/default/error_tracking/assignment_matching_with_retries_spec.rb` | Same trace id should be present when instrumenting errors and during the DLQ dispatch Since we retry 2 times, only proper dispatch traces should be present in the final set |
| `pro/consumption/strategies/dlq/default/independent_constant_recovering_spec.rb` | When independent flag is on and the error is rolling, it should never go to the DLQ |
| `pro/consumption/strategies/dlq/default/manual_dispatch_to_dlq_spec.rb` | When having the DLQ defined, we should be able to manually dispatch things to the DLQ and continue processing whenever we want. We can use this API to manually move stuff to DLQ without raising any errors upon detecting a corrupted message. |
| `pro/consumption/strategies/dlq/default/multi_partition_source_target_flow_spec.rb` | When handling failing messages from a many partitions and there are many errors, DLQ will provide strong ordering warranties inside DLQ. |
| `pro/consumption/strategies/dlq/default/multi_partition_target_flow_spec.rb` | When handling failing messages from a single partition and there are many errors, enhanced DLQ will provide strong ordering warranties inside DLQ. |
| `pro/consumption/strategies/dlq/default/multi_topic_one_target_multi_partitions_flow_spec.rb` | When using single DLQ to handle errors from multiple topics, the dispatched message key should be preserved and consistent piping should be used. |
| `pro/consumption/strategies/dlq/default/original_offset_tracking_spec.rb` | When we move messages to the DLQ, their offset should be preserved in headers |
| `pro/consumption/strategies/dlq/default/pause_prior_to_automatic_offset_and_mark_spec.rb` | When user pauses and marks by himself, we should not deal with this and let him do this. |
| `pro/consumption/strategies/dlq/default/post_dispatch_errors_tracker_state_spec.rb` | After a successful dispatch to DLQ, next first attempt should not have errors in the tracker |
| `pro/consumption/strategies/dlq/default/raw_headers_dispatch_spec.rb` | DLQ dispatch should use raw headers and not deserialized headers similar to how payload is dispatched |
| `pro/consumption/strategies/dlq/default/with_custom_strategy_using_consumer_group_context_spec.rb` | We should be able to use error tracker topic understanding to route errors from the same topic to two topics based on the consumer group |
| `pro/consumption/strategies/dlq/default/with_enhanced_dispatch_data_spec.rb` | We should be able to dispatch to DLQ with usage of the enhanced errors tracking details |
| `pro/consumption/strategies/dlq/default/with_error_handling_pipeline_spec.rb` | We should be able to use DLQ in a pipeline where we handle each stage separately. |
| `pro/consumption/strategies/dlq/default/with_error_type_based_dispatcher_spec.rb` | When using a complex dispatch strategy based on error type, it should operate as expected. Please keep in mind, this spec does not care about the fact, that during recovery different errors occur. It uses notion of last error only. |
| `pro/consumption/strategies/dlq/default/with_error_type_based_topic_spec.rb` | When using a complex dispatch strategy based on error type, it should operate as expected. Please keep in mind, this spec does not care about the fact, that during recovery different errors occur. It uses notion of last error only. We should be able to dispatch to customized target topics bypassing the default if needed. |
| `pro/consumption/strategies/dlq/default/with_external_each_patch_spec.rb` | This spec verifies that Pro DLQ strategy works correctly when external libraries monkey-patch the Messages#each method (e.g., for tracing/instrumentation purposes). The Pro DLQ strategy internally uses messages.raw.find to locate skippable messages, which bypasses the patched each method by accessing the underlying array directly. @see https://github.com/karafka/karafka/issues/2939 |
| `pro/consumption/strategies/dlq/default/with_non_recoverable_error_with_one_retry_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after one retry and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/default/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/default/with_non_recoverable_error_without_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message and we don't want to do any retries, we should skip without retrying of processing. |
| `pro/consumption/strategies/dlq/default/with_non_recoverable_first_message_spec.rb` | When dead letter queue is used and we first message out of all is broken, things should behave like for any other broken message |
| `pro/consumption/strategies/dlq/default/with_non_recoverable_last_message_spec.rb` | When dead letter queue is used and we last message out of all is broken, things should behave like for any other broken message and we should pick up when more messages are present |
| `pro/consumption/strategies/dlq/default/with_recoverable_error_on_retry_spec.rb` | When dead letter queue is used and we encounter recoverable message, we should not skip and not move it to the dead letter topic. Just retry. |
| `pro/consumption/strategies/dlq/default/with_rolling_error_spec.rb` | Without the independent flag (default) Karafka will accumulate attempts on a batch in a rolling fashion when recoverable errors appear over and over again on a set of messages. |
| `pro/consumption/strategies/dlq/default/with_sync_dispatch_spec.rb` | When using DLQ, it should work when dispatch in sync |
| `pro/consumption/strategies/dlq/default/with_sync_marking_spec.rb` | When using DLQ, it should work when marking as consumed sync |
| `pro/consumption/strategies/dlq/default/without_any_errors_spec.rb` | When dead letter queue is used and we don't encounter any errors, all should be regular. |
| `pro/consumption/strategies/dlq/default/without_dispatch_with_retries_spec.rb` | When using the DLQ with topic set to false, Karafka should not dispatch the message further but should apply the DLQ skipping logic anyhow. |
| `pro/consumption/strategies/dlq/default/without_dispatch_without_retries_spec.rb` | When using the DLQ with topic set to false, Karafka should not dispatch the message further but should apply the DLQ skipping logic anyhow. |
| `pro/consumption/strategies/dlq/default/without_intermediate_marking_spec.rb` | When we do not perform any markings as consumed we should nonetheless skip one after another starting from the beginning of the batch on failed batches. This should allow us to move one by one slowly and should mark on successful batches. |
| `pro/consumption/strategies/dlq/default/without_post_dispatch_marking_spec.rb` | With marking disabled, the rolling of error should not cause offset storage on errors |
| `pro/consumption/strategies/dlq/ftr/pipeline_with_delayed_dlq_spec.rb` | With delayed jobs with should be able to build a pipeline where we delay re-processing of messages when their processing fails and they are moved to DLQ. |
| `pro/consumption/strategies/dlq/ftr/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr/with_manual_pause_spec.rb` | Karafka should favour a manual pause over throttling and take appropriate action when the pause has expired. So if we've reached throttling threshold and paused, after the manual pause is over, we should pause and not process We also should not have any duplicates and processing should be smooth |
| `pro/consumption/strategies/dlq/ftr/with_non_recoverable_error_with_one_retry_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after one retry and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr/with_non_recoverable_error_without_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message and we don't want to do any retries, we should skip without retrying of processing. |
| `pro/consumption/strategies/dlq/ftr/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one. |
| `pro/consumption/strategies/dlq/ftr/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. The message should not go to the DLQ because the error has recovered. |
| `pro/consumption/strategies/dlq/ftr/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. DLQ should not interact with this in any way. |
| `pro/consumption/strategies/dlq/ftr/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached. DLQ should have nothing to do with this. |
| `pro/consumption/strategies/dlq/ftr_lrj/fast_with_non_recoverable_errors_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj |
| `pro/consumption/strategies/dlq/ftr_lrj/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/ftr_lrj/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr_lrj/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr_lrj/with_long_manual_pause_spec.rb` | Karafka should not resume when manual pause is in use for DLQ LRJ |
| `pro/consumption/strategies/dlq/ftr_lrj/with_manual_pause_spec.rb` | Karafka should favour a manual pause over throttling and take appropriate action when the pause has expired. So if we've reached throttling threshold and paused, after the manual pause is over, we should pause and not process We also should not have any duplicates and processing should be smooth |
| `pro/consumption/strategies/dlq/ftr_lrj/with_non_recoverable_error_with_one_retry_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after one retry and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr_lrj/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr_lrj/with_non_recoverable_error_without_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message and we don't want to do any retries, we should skip without retrying of processing. |
| `pro/consumption/strategies/dlq/ftr_lrj/with_non_recoverable_slow_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer and after few incidents it should move data to the DLQ and just continue |
| `pro/consumption/strategies/dlq/ftr_lrj/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one. |
| `pro/consumption/strategies/dlq/ftr_lrj/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. The message should not go to the DLQ because the error has recovered. |
| `pro/consumption/strategies/dlq/ftr_lrj/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. DLQ should not interact with this in any way. |
| `pro/consumption/strategies/dlq/ftr_lrj/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached. DLQ should have nothing to do with this. |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/non_recoverable_moving_forward_batch_marking_spec.rb` | Upon non-recoverable errors, Karafka should move forward retrying from last one that was committed with duplicates in between comitted and crashed |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/non_recoverable_moving_forward_marking_spec.rb` | When using DLQ with FTR, LRJ, and MOM together, non-recoverable errors should skip failed messages and move forward, ensuring no duplicates when manual offset management is used |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/non_recoverable_moving_forward_no_marking_spec.rb` | Upon non-recoverable errors, Karafka should move forward skipping given message even if no marking happens. When operating on batches and no marking happens, we skip first message from the batch on which the error happened. |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/occasional_marking_spec.rb` | When using manual offset management and not marking often, we should have a smooth processing flow without extra messages or anything. |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/with_manual_pause_on_early_spec.rb` | When pausing not on a last message, we should un-pause from it and not from the next incoming. |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/with_non_recoverable_slow_no_marking_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj with mom but because of no marking, we should move forward, however upon picking up work, we should start from zero This can be risky upon rebalance but we leave it to the advanced users to manage. of no marking, we should move forward, however upon picking up work, we should move offset based on the crashes one by one from the first consumed or from zero |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/with_post_dispatch_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets until DLQ as long as DLQ has an explicit post-error marking (which is not the default) |
| `pro/consumption/strategies/dlq/ftr_lrj_mom/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/fast_with_non_recoverable_errors_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_long_manual_pause_spec.rb` | Karafka should not resume when manual pause is in use for DLQ LRJ VP Pausing may be not predictable because of VPs |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_manual_pause_spec.rb` | Karafka should favour a manual pause over throttling and take appropriate action when the pause has expired. So if we've reached throttling threshold and paused, after the manual pause is over, we should pause and not process We also should not have any duplicates and processing should be smooth |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_non_recoverable_error_with_one_retry_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after one retry and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_non_recoverable_slow_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer and after few incidents it should move data to the DLQ and just continue |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one. |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. The message should not go to the DLQ because the error has recovered. |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. DLQ should not interact with this in any way. |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached. DLQ should have nothing to do with this. |
| `pro/consumption/strategies/dlq/ftr_lrj_vp/without_intermediate_marking_spec.rb` | Karafka should mark correctly the final offset of collective group upon finish |
| `pro/consumption/strategies/dlq/ftr_mom/multi_partition_source_target_flow_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/multi_partition_target_flow_spec.rb` | Same as pure DLQ version until rebalance Needs to go to same partition |
| `pro/consumption/strategies/dlq/ftr_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/ftr_mom/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr_mom/with_error_handling_pipeline_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr_mom/with_manual_pause_spec.rb` | Karafka should favour a manual pause over throttling and take appropriate action when the pause has expired. So if we've reached throttling threshold and paused, after the manual pause is over, we should pause and not process We also should not have any duplicates and processing should be smooth |
| `pro/consumption/strategies/dlq/ftr_mom/with_non_recoverable_error_and_marking_spec.rb` | When non-recoverable error happens and we want explicit marking, it should mark |
| `pro/consumption/strategies/dlq/ftr_mom/with_non_recoverable_error_with_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/with_non_recoverable_error_without_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/with_non_recoverable_first_message_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/with_non_recoverable_last_message_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/with_recoverable_error_on_retry_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/without_any_errors_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/ftr_mom/without_intermediate_marking_spec.rb` | When we do not mark and user does not mark, we will end up with an infinite loop. This is expected and user should deal with this on his own. |
| `pro/consumption/strategies/dlq/ftr_mom_vp/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr_mom_vp/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr_mom_vp/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic Throttling should throttle but not break this flow. |
| `pro/consumption/strategies/dlq/ftr_mom_vp/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one. |
| `pro/consumption/strategies/dlq/ftr_mom_vp/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. The message should not go to the DLQ because the error has recovered. |
| `pro/consumption/strategies/dlq/ftr_mom_vp/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. DLQ should not interact with this in any way. |
| `pro/consumption/strategies/dlq/ftr_mom_vp/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached. DLQ should have nothing to do with this. |
| `pro/consumption/strategies/dlq/ftr_vp/marking_without_intermediate_marking_spec.rb` | Karafka should mark correctly the final offset of collective group upon finish |
| `pro/consumption/strategies/dlq/ftr_vp/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/dlq/ftr_vp/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/dlq/ftr_vp/with_non_recoverable_error_with_retries_spec.rb` | When dead letter queue is used and we encounter non-recoverable message, we should skip it after retries and move the broken message to a separate topic Throttling should throttle but not break this flow. |
| `pro/consumption/strategies/dlq/ftr_vp/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one. |
| `pro/consumption/strategies/dlq/ftr_vp/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. The message should not go to the DLQ because the error has recovered. |
| `pro/consumption/strategies/dlq/ftr_vp/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. DLQ should not interact with this in any way. |
| `pro/consumption/strategies/dlq/ftr_vp/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached. DLQ should have nothing to do with this. |
| `pro/consumption/strategies/dlq/lrj/fast_with_non_recoverable_errors_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj |
| `pro/consumption/strategies/dlq/lrj/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/lrj/with_long_manual_pause_spec.rb` | Karafka should not resume when manual pause is in use for DLQ LRJ |
| `pro/consumption/strategies/dlq/lrj/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement. Configured DLQ should have nothing to do with this |
| `pro/consumption/strategies/dlq/lrj/with_non_recoverable_slow_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer and after few incidents it should move data to the DLQ and just continue |
| `pro/consumption/strategies/dlq/lrj_mom/non_recoverable_moving_forward_batch_marking_spec.rb` | Upon non-recoverable errors, Karafka should move forward retrying from last one that was committed with duplicates in between comitted and crashed |
| `pro/consumption/strategies/dlq/lrj_mom/non_recoverable_moving_forward_marking_spec.rb` | Upon non-recoverable errors, Karafka should move forward skipping given message even if no marking happens for each batch. Since we mark the message prior to its processing (stupid but valid) we go to DLQ with the next one, hence the skip |
| `pro/consumption/strategies/dlq/lrj_mom/non_recoverable_moving_forward_no_marking_spec.rb` | Upon non-recoverable errors, Karafka should move forward skipping given message even if no marking happens. When operating on batches and no marking happens, we skip first message from the batch on which the error happened. |
| `pro/consumption/strategies/dlq/lrj_mom/non_recoverable_with_dispatch_marking_spec.rb` | Upon non-recoverable errors and the DLQ dispatch with marking, Karafka should mark |
| `pro/consumption/strategies/dlq/lrj_mom/occasional_marking_spec.rb` | When using manual offset management and not marking often, we should have a smooth processing flow without extra messages or anything. |
| `pro/consumption/strategies/dlq/lrj_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/lrj_mom/with_manual_pause_on_early_spec.rb` | When pausing not on a last message, we should un-pause from it and not from the next incoming. |
| `pro/consumption/strategies/dlq/lrj_mom/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement |
| `pro/consumption/strategies/dlq/lrj_mom/with_non_recoverable_slow_no_marking_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj with mom but because of no marking, we should move forward, however upon picking up work, we should start from zero This can be risky upon rebalance but we leave it to the advanced users to manage. of no marking, we should move forward, however upon picking up work, we should move offset based on the crashes one by one from the first consumed or from zero |
| `pro/consumption/strategies/dlq/lrj_mom/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. |
| `pro/consumption/strategies/dlq/lrj_mom_vp/last_marking_without_intermediate_on_vp_spec.rb` | When marking in VP mode, when we mark only last message, the offset should not be comitted because we did not mark previous messages. This should apply only to the current batch though We make sure here that DLQ does not interact when no errors. |
| `pro/consumption/strategies/dlq/lrj_mom_vp/with_non_recoverable_slow_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj with VP the same way as any normal consumer and after few incidents it should move data to the DLQ and just continue We collapse VP and retry in a collapsed mode, thus reducing this strategy to DlqLrjMom |
| `pro/consumption/strategies/dlq/lrj_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/lrj_vp/with_non_recoverable_slow_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj with VP the same way as any normal consumer and after few incidents it should move data to the DLQ and just continue We collapse VP and retry in a collapsed mode, thus reducing this strategy to DlqLrj |
| `pro/consumption/strategies/dlq/lrj_vp/without_intermediate_marking_spec.rb` | Karafka should mark correctly the final offset of collective group upon finish |
| `pro/consumption/strategies/dlq/mom/at_most_once_skipping_on_error_poll_one_spec.rb` | This example is a bit counter intuitive. Since we mark as consumed before we consume and raise error, Karafka will skip this message and move on. On another it will again encounter error and will dispatch to DLQ the next message because Karafka does not know, that marking happened prior to an error. This is however expected. If you want to alter this behaviour, you need to write your own custom strategy to compensate for this by moving the seek offset back for a DLQ dispatch case. |
| `pro/consumption/strategies/dlq/mom/at_most_once_skipping_on_error_spec.rb` | This example is a bit counter intuitive. Since we mark as consumed before we consume and raise error, Karafka will skip this message and move on. On another it will again encounter error and will dispatch to DLQ the next message because Karafka does not know, that marking happened prior to an error. This is however expected. If you want to alter this behaviour, you need to write your own custom strategy to compensate for this by moving the seek offset back for a DLQ dispatch case. |
| `pro/consumption/strategies/dlq/mom/multi_partition_source_target_flow_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/multi_partition_target_flow_spec.rb` | Same as pure DLQ version until rebalance Needs to go to same partition |
| `pro/consumption/strategies/dlq/mom/non_recoverable_with_dispatch_marking_spec.rb` | Upon non-recoverable errors and the DLQ dispatch with marking, Karafka should mark |
| `pro/consumption/strategies/dlq/mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/mom/with_error_handling_pipeline_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/with_non_recoverable_error_with_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/with_non_recoverable_error_without_retries_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/with_non_recoverable_first_message_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/with_non_recoverable_last_message_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/with_recoverable_error_on_retry_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/without_any_errors_spec.rb` | Same as pure DLQ version until rebalance |
| `pro/consumption/strategies/dlq/mom/without_intermediate_marking_spec.rb` | When we do not mark and user does not mark, we will end up with an infinite loop. This is expected and user should deal with this on his own. |
| `pro/consumption/strategies/dlq/mom_vp/collapse_and_continuity_spec.rb` | Karafka when with VP upon error should collapse the whole collective batch and should continue processing in the collapsed mode after a back-off until all the "infected" messages are done. After that, VPs should be resumed. In regards to DLQ, unless errors are persistent, DLQ should not kick in, so in this example DLQ should not kick in |
| `pro/consumption/strategies/dlq/mom_vp/dlq_skip_on_virtual_marking_with_last_failing_spec.rb` | When running MoM + VP and marking on each, when we crash on last message out of all, previous should be handled and we should restart and then dispatch only this single one. |
| `pro/consumption/strategies/dlq/mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/mom_vp/skip_after_collapse_many_spec.rb` | DLQ in the VP mode should collapse and skip when error occurs again in a collapsed mode also when there are many errors in the same collective batch. In a scenario like this, we should collapse and skip one after another. |
| `pro/consumption/strategies/dlq/mom_vp/skip_after_collapse_spec.rb` | DLQ in the VP mode should collapse and skip when error occurs again in a collapsed mode After that, we should move to processing in a non-collapsed mode again |
| `pro/consumption/strategies/dlq/vp/collapse_and_continuity_spec.rb` | Karafka when with VP upon error should collapse the whole collective batch and should continue processing in the collapsed mode after a back-off until all the "infected" messages are done. After that, VPs should be resumed. In regards to DLQ, unless errors are persistent, DLQ should not kick in, so in this example DLQ should not kick in |
| `pro/consumption/strategies/dlq/vp/marking_without_intermediate_marking_spec.rb` | Karafka should mark correctly the final offset of collective group upon finish |
| `pro/consumption/strategies/dlq/vp/patterned_custom_strategy_based_flow_spec.rb` | Karafka should retry over and over again same message if the error is not one of recognized |
| `pro/consumption/strategies/dlq/vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/dlq/vp/skip_after_collapse_many_spec.rb` | DLQ in the VP mode should collapse and skip when error occurs again in a collapsed mode also when there are many errors in the same collective batch. In a scenario like this, we should collapse and skip one after another. |
| `pro/consumption/strategies/dlq/vp/skip_after_collapse_spec.rb` | DLQ in the VP mode should collapse and skip when error occurs again in a collapsed mode After that, we should move to processing in a non-collapsed mode again |
| `pro/consumption/strategies/ftr/all_messages_filtered_with_marking_spec.rb` | When all messages are filtered out and the filter implements `mark_as_consumed?` returning true, Karafka should mark the offset at the cursor position (the last filtered message) so that processing can continue from the correct position. This test verifies that: 1. All messages are successfully filtered out 2. The offset is correctly stored at the cursor (the last message that was filtered) 3. Consumer lag does not grow indefinitely |
| `pro/consumption/strategies/ftr/chained_filters_with_post_throttle_spec.rb` | We should be able to chain filters and to achieve expected processing flow In this scenario we will filter all odd offsets and we will make sure, we process data with a delay and with throttling used to make sure we do not process more than 5 messages per second It is important to notice, that the order of filters is important. If we throttle before we filter, we will actually always process late because throttling will occur on odd and even. |
| `pro/consumption/strategies/ftr/chained_filters_with_pre_throttle_spec.rb` | We should be able to chain filters and to achieve expected processing flow In this scenario we will filter all odd offsets and we will make sure, we process data with a delay and with throttling used to make sure we do not process more than 5 messages per second We will throttle first and this will have impact on how many elements we will get into the consumer |
| `pro/consumption/strategies/ftr/complex_predicate_filtering_spec.rb` | Filtering should handle complex predicates including JSON parsing, regex matching, timestamp-based filtering, and composite conditions without performance degradation. |
| `pro/consumption/strategies/ftr/custom_per_topic_spec.rb` | Karafka should allow for usage of custom throttlers per topic |
| `pro/consumption/strategies/ftr/custom_to_delay_spec.rb` | Karafka should allow us to use throttling engine to implement delayed jobs |
| `pro/consumption/strategies/ftr/dlg/constant_flow_spec.rb` | When we are just getting new data, we should delay to match time expectations |
| `pro/consumption/strategies/ftr/dlg/with_only_old_enough_messages_spec.rb` | When Karafka delays processing and we have only old messages, there should be no pausing or seeking and we should just process |
| `pro/consumption/strategies/ftr/endless_skip_no_seek_spec.rb` | When we create a filter that just skips all the messages and does not return the cursor message, we should never seek and just go on with incoming messages |
| `pro/consumption/strategies/ftr/event_filter_all_filtered_spec.rb` | This spec tests a real-world event filtering scenario where we only want to process specific event types (e.g., "order_created") and filter out all others. When all messages in a batch are filtered out, the filter should: 1. Mark @applied as true (filtering was applied) 2. Set @all_filtered to true 3. Store the cursor at the last filtered message 4. Return true from mark_as_consumed? so offset advances This prevents consumer lag from growing when no relevant events are present. |
| `pro/consumption/strategies/ftr/event_filter_alternating_batches_spec.rb` | This spec tests a real-world event filtering scenario where batches alternate between having all messages filtered and having messages to process. This tests: 1. Correct offset advancement when alternating between filtered and processed batches 2. mark_as_consumed? behavior changes between all-filtered and mixed batches 3. Consumer is only called for batches with messages to process 4. Proper offset management across batch boundaries This scenario mimics real-world traffic where event types come in bursts. |
| `pro/consumption/strategies/ftr/event_filter_mixed_spec.rb` | This spec tests a real-world event filtering scenario where some messages match the target event type and others don't. When batches contain mixed messages: 1. @applied should be true when filtering occurs 2. @all_filtered should be false (some messages pass through) 3. mark_as_consumed? should return false (not all filtered) 4. Consumer must explicitly mark offset for processed messages This ensures proper offset management when processing mixed batches. |
| `pro/consumption/strategies/ftr/event_filter_none_filtered_spec.rb` | This spec tests a real-world event filtering scenario where all messages match the target event type (none are filtered out). In this case: 1. @applied should be false (no filtering was applied) 2. @all_filtered should be false (messages were not filtered) 3. mark_as_consumed? should return false (messages need processing) 4. Consumer must explicitly mark offset for processed messages This ensures the filter behaves correctly when it's a no-op. |
| `pro/consumption/strategies/ftr/exg/with_always_fresh_messages_spec.rb` | When Karafka has expiring enabled but all the messages are fresh, nothing should be expired |
| `pro/consumption/strategies/ftr/exg/with_never_fresh_messages_spec.rb` | When Karafka has expiring enabled and expiring is super short, we should never get any messages |
| `pro/consumption/strategies/ftr/exg/with_some_expiring_messages_spec.rb` | When Karafka encounters messages that are too old, it should skip them We simulate this by having short ttl and delaying processing to build up a lag |
| `pro/consumption/strategies/ftr/feature_toggle_persistent_pausing_spec.rb` | Demonstrates persistent topic pausing using the Filtering API with feature toggles. This pattern allows pause state to survive restarts and rebalances, unlike Web UI pausing. |
| `pro/consumption/strategies/ftr/increasing_lag_on_full_filtering_spec.rb` | When we have a filter that rejects all (or most of) the data all the time, since we do not mark as consumed, the offset will not be stored. This means, that lag will grow unless we explicitly request marking post-filtering. |
| `pro/consumption/strategies/ftr/not_increasing_lag_on_full_filtering_with_marking_spec.rb` | When we have a filter that rejects all (or most of) the data all the time, since we do mark as consumed, the offset will be stored. This means, that lag will not grow. |
| `pro/consumption/strategies/ftr/partial_messages_filtered_with_marking_spec.rb` | When some messages are filtered out and others are processed, with the filter implementing `mark_as_consumed?` returning true, Karafka should mark the offset after the last processed message when the consumer explicitly marks it. This test verifies that: 1. Filtered messages are skipped 2. Non-filtered messages are processed 3. The offset is correctly stored when consumer marks as consumed 4. The last processed message is correctly marked |
| `pro/consumption/strategies/ftr/partial_offset_storage_race_condition_spec.rb` | Test for partial offset storage race condition in filtering API We mark the last removed message on a cursor but then mark ahead and check what is going to be marked to ensure proper offset management This tests the scenario where we have a cursor pointing to a removed message while also marking ahead of the cursor position |
| `pro/consumption/strategies/ftr/persistent_post_rebalance_pausing_spec.rb` | When using the filtering API, we can use a persistent storage to transfer the pause over the rebalance to the same or other processes. Here we do not use cooperative.sticky to trigger a revocation during a pause and we continue the pause until its end after getting the assignment back. Fake DB layer We do not have to care about topics and partitions because for spec like this we use one partition |
| `pro/consumption/strategies/ftr/persistent_transactional_offset_spec.rb` | When using the Filtering API, we can use persistent storage to transfer last offset that we successfully operated on in case of rebalances, even if different process receives the partition on a rebalance. This allows us to ensure, that things are processed exactly once and in transactions. Note, though that this requires to have SQL timeouts, etc tuned well. Simulates a persistent DB backed offset that can be fetched in any process Keep in mind, this example is a simplification because it forces `#seek` for each assignment just to simplify the code. |
| `pro/consumption/strategies/ftr/starting_from_latest_always_spec.rb` | A spec that illustrates usage of a filter that will ensure that when we start Karafka, we always start from the latest message despite any previous assignments It should move to latest and then move forward |
| `pro/consumption/strategies/ftr/starting_from_latest_transactional_spec.rb` | A spec that illustrates usage of a filter that will ensure that when we start Karafka, we always start from the latest offset even if it is a transactional one. We can start from the high-watermark - 1 on transactional but it will just wait for more data |
| `pro/consumption/strategies/ftr/static_throttler_spec.rb` | We should be able to use a factory, that always returns the same throttler even after rebalances, so throttling still applies after rebalance |
| `pro/consumption/strategies/ftr/time_limit_check_spec.rb` | Karafka should throttle and wait and should not consume more in a given time window despite data being available |
| `pro/consumption/strategies/ftr/used_status_for_fully_delayed_consumer_spec.rb` | When we fully delay consumption and just run idle job, shutdown idle status should reflect that |
| `pro/consumption/strategies/ftr/vp/from_earliest_throttled_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) using multiple threads based on the used virtual partitioner. We should use more than one thread for processing of all the messages and we should throttle the performance to make sure we do not do it too fast |
| `pro/consumption/strategies/ftr/vp/throttled_collapse_and_continuity_spec.rb` | Karafka when with VP upon error should collapse the whole collective batch and should continue processing in the collapsed mode after a back-off until all the "infected" messages are done. After that, VPs should be resumed. Throttling should operate as expected. |
| `pro/consumption/strategies/ftr/with_delayed_shutdown_metadata_lags_spec.rb` | We should be able to reference both `#processing_lag` and `consumption_lag` even when we had delay and no data would be consumed prior. |
| `pro/consumption/strategies/ftr/with_hitting_limits_spec.rb` | Karafka should throttle and wait for the expected time period before continuing the processing |
| `pro/consumption/strategies/ftr/with_manual_pause_spec.rb` | Karafka should favour a manual pause over throttling and take appropriate action when the pause has expired. So if we've reached throttling threshold and paused, after the manual pause is over, we should pause and not process We also should not have any duplicates and processing should be smooth |
| `pro/consumption/strategies/ftr/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the filters. |
| `pro/consumption/strategies/ftr/with_one_partition_throttled_spec.rb` | Karafka should throttle only the partition that hit limits and not the other one |
| `pro/consumption/strategies/ftr/with_pattern_matched_later_topic_spec.rb` | Karafka should throttle and wait also on topics that are detected via patterns later on |
| `pro/consumption/strategies/ftr/with_recoverable_error_spec.rb` | Errors should be handled normally. There should be a backoff and retry and recovery should start from the message on which we broke. Throttling should have nothing to do with this. |
| `pro/consumption/strategies/ftr/with_throttled_error_spec.rb` | When we reach throttling limit and error, we should process again from the errored place |
| `pro/consumption/strategies/ftr/with_throttled_expired_error_spec.rb` | When we reach throttling limit and error, we should process again from the errored place If throttling went beyond and we should continue, this should not change anything |
| `pro/consumption/strategies/ftr/with_work_exceeding_throttle_spec.rb` | When doing work that is exceeding timeouts, we should not throttle. Instead we need to seek to the first throttled message and just move on. |
| `pro/consumption/strategies/ftr/without_hitting_limits_spec.rb` | Karafka should be able to just consume when throttling limits are not reached |
| `pro/consumption/strategies/lrj/default/attempts_tracking_with_errors_spec.rb` | When running jobs with recoverable errors, we should have the attempts count increased |
| `pro/consumption/strategies/lrj/default/attempts_tracking_with_non_recoverable_errors_spec.rb` | When running jobs with non-recoverable errors, we should have the attempts count increased |
| `pro/consumption/strategies/lrj/default/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt |
| `pro/consumption/strategies/lrj/default/concurrent_operations_execution_spec.rb` | When we make long polls the same time consumers operate, those operations should be executable in parallel. This spec is to ensure, that no `librdkafka` or `rdkafka-ruby` locks interfere with the expected concurrency boundaries of Karafka It did happen, that due to locking model changes, certain things would heavily impact ability to operate concurrently. |
| `pro/consumption/strategies/lrj/default/fast_non_blocking_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj It should work ok also when used via `non_blocking` API. |
| `pro/consumption/strategies/lrj/default/fast_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj |
| `pro/consumption/strategies/lrj/default/kip_848_revocation_detection_spec.rb` | Test KIP-848 with Long Running Jobs to ensure that when a rebalance occurs during long-running consumption with the new protocol, the consumer is properly notified via both #revoked and #revoked? methods |
| `pro/consumption/strategies/lrj/default/multiple_batches_processing_spec.rb` | When processing multiple incoming batches, the order should be preserved |
| `pro/consumption/strategies/lrj/default/no_revocation_on_low_concurrency_spec.rb` | When running LRJ with low concurrency and many LRJ topics, we should not be kicked out of the consumer group after reaching the interval. Pausing should happen prior to processing and it should ensure that all new LRJ topics and partitions assigned are paused even when there are no available workers to do the work. |
| `pro/consumption/strategies/lrj/default/parallel_non_lrj_with_lrj_spec.rb` | When using LRJ within a consumer group with other non-LRJ, the LRJ should be running while other jobs are consumed and they should not wait (as long as enough workers) |
| `pro/consumption/strategies/lrj/default/processing_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good. It should continue processing after resume and should pick up next messages |
| `pro/consumption/strategies/lrj/default/processing_via_non_blocking_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good. It should continue processing after resume and should pick up next messages Marking as LRJ should also work via `non_blocking` alias. |
| `pro/consumption/strategies/lrj/default/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/lrj/default/saturated_before_revocation_spec.rb` | When we have a LRJ job and revocation happens, non revocation aware LRJ should not cause a timeout because the revocation job is also non-blocking. |
| `pro/consumption/strategies/lrj/default/shutdown_order_spec.rb` | When running lrj, Karafka should never run the shutdown operations while consumption is in progress |
| `pro/consumption/strategies/lrj/default/starved_revocation_on_rebalance_spec.rb` | When running lrj, on revocation Karafka should change the revocation state even when there are no available slots for processing |
| `pro/consumption/strategies/lrj/default/with_auto_offset_management_spec.rb` | When using automatic offset management, we should end up with offset committed after the last message and we should "be" there upon returning to processing |
| `pro/consumption/strategies/lrj/default/with_continuous_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if it happens often. It should not impact processing order. |
| `pro/consumption/strategies/lrj/default/with_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer |
| `pro/consumption/strategies/lrj/default/with_long_manual_pause_spec.rb` | Karafka should not resume when manual pause is in use for LRJ |
| `pro/consumption/strategies/lrj/default/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement. |
| `pro/consumption/strategies/lrj/default/with_never_ending_error_spec.rb` | When doing work with error, we should slowly increase the attempt count for LRJ same as for regular workloads, despite pausing. |
| `pro/consumption/strategies/lrj/default/with_pre_consumption_revocation_spec.rb` | When LRJ jobs are in the processing queue prior to being picked by the workers and those LRJ jobs get revoked, the job should not run. |
| `pro/consumption/strategies/lrj/default/with_revocation_synchronization_spec.rb` | When LRJ job uses a critical section, the revocation job should wait and not be able to run its critical section alongside. |
| `pro/consumption/strategies/lrj/ftr/attempts_tracking_with_errors_spec.rb` | When running jobs with recoverable errors, we should have the attempts count increased |
| `pro/consumption/strategies/lrj/ftr/attempts_tracking_with_non_recoverable_errors_spec.rb` | When running jobs with non-recoverable errors, we should have the attempts count increased |
| `pro/consumption/strategies/lrj/ftr/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt even if throttling occurs |
| `pro/consumption/strategies/lrj/ftr/constant_reporting_lag_spec.rb` | When using running with a delay and producing in a loop, we should always have a lag not bigger than the total of things that are in front of our marked offset If we run a non-blocking marking that happens less frequently than polling, this can go beyond what we currently process + what is ahead, because technically we are behind |
| `pro/consumption/strategies/lrj/ftr/fast_non_throttled_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj and they should behave the same way as once without throttling enabled |
| `pro/consumption/strategies/lrj/ftr/parallel_non_lrj_with_lrj_spec.rb` | When using LRJ within a consumer group with other non-LRJ, the LRJ should be running while other jobs are consumed and they should not wait (as long as enough workers) and throttling should not cause any problems |
| `pro/consumption/strategies/lrj/ftr/processing_with_expired_throttling_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good. It should continue processing after resume and should pick up next messages. Expired throttling should just seek correctly as expected and allow us to move on |
| `pro/consumption/strategies/lrj/ftr/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/lrj/ftr/shutdown_order_spec.rb` | When running lrj, Karafka should never run the shutdown operations while consumption is in progress |
| `pro/consumption/strategies/lrj/ftr/with_auto_offset_management_spec.rb` | When using automatic offset management, we should end up with offset committed after the last message and we should "be" there upon returning to processing. Throttling should have nothing to do with this. |
| `pro/consumption/strategies/lrj/ftr/with_continuous_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if it happens often. It should not impact processing order even if throttling occurs. |
| `pro/consumption/strategies/lrj/ftr/with_long_manual_pause_when_throttled_spec.rb` | Karafka should not resume when manual pause is in use for LRJ. Karafka should not throttle (unless in idle which would indicate pause lift) when manually paused After un-pause, Karafka may do a full throttle if the previous throttling time did not finish |
| `pro/consumption/strategies/lrj/ftr/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement. Filter that would require seek, should use the user requested offset over its own |
| `pro/consumption/strategies/lrj/ftr/with_never_ending_error_spec.rb` | When doing work with error, we should slowly increase the attempt count for LRJ same as for regular workloads, despite pausing. |
| `pro/consumption/strategies/lrj/ftr/with_throttle_and_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if throttled. Error flow should be the same as non-throttled. |
| `pro/consumption/strategies/lrj/ftr_mom/attempts_tracking_with_errors_spec.rb` | When running jobs with recoverable errors, we should have the attempts count increased. We should NOT manage any offsets unless used manually. |
| `pro/consumption/strategies/lrj/ftr_mom/attempts_tracking_with_non_recoverable_errors_spec.rb` | When running jobs with non-recoverable errors, we should have the attempts count increased We should not commit any offsets |
| `pro/consumption/strategies/lrj/ftr_mom/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt even if throttling occurs |
| `pro/consumption/strategies/lrj/ftr_mom/fast_non_throttled_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj and they should behave the same way as once without throttling enabled |
| `pro/consumption/strategies/lrj/ftr_mom/occasional_marking_spec.rb` | When using manual offset management and not marking often, we should have a smooth processing flow without extra messages or anything. |
| `pro/consumption/strategies/lrj/ftr_mom/parallel_non_lrj_with_lrj_spec.rb` | When using LRJ within a consumer group with other non-LRJ, the LRJ should be running while other jobs are consumed and they should not wait (as long as enough workers) and throttling should not cause any problems |
| `pro/consumption/strategies/lrj/ftr_mom/processing_with_expired_throttling_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good. It should continue processing after resume and should pick up next messages. Expired throttling should just seek correctly as expected and allow us to move on |
| `pro/consumption/strategies/lrj/ftr_mom/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/lrj/ftr_mom/shutdown_order_spec.rb` | When running lrj, Karafka should never run the shutdown operations while consumption is in progress |
| `pro/consumption/strategies/lrj/ftr_mom/with_continuous_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if it happens often. We should retry from 0 because we do not commit offsets here at all We should not change the offset |
| `pro/consumption/strategies/lrj/ftr_mom/with_long_manual_pause_when_throttled_spec.rb` | Karafka should not resume when manual pause is in use for LRJ. Karafka should not throttle (unless in idle which would indicate pause lift) when manually paused After un-pause, Karafka may do a full throttle if the previous throttling time did not finish Offsets should not be changed |
| `pro/consumption/strategies/lrj/ftr_mom/with_manual_pause_on_early_spec.rb` | When pausing not on a last message, we should un-pause from it and not from the next incoming. |
| `pro/consumption/strategies/lrj/ftr_mom/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement. Filter that would require seek, should use the user requested offset over its own |
| `pro/consumption/strategies/lrj/ftr_mom/with_never_ending_error_spec.rb` | When doing work with error, we should slowly increase the attempt count for LRJ same as for regular workloads, despite pausing. |
| `pro/consumption/strategies/lrj/ftr_mom/with_throttle_and_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if throttled. Error flow should be the same as non-throttled. |
| `pro/consumption/strategies/lrj/ftr_mom/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. |
| `pro/consumption/strategies/lrj/ftr_mom_vp/attempts_tracking_with_errors_spec.rb` | When running jobs with recoverable errors, we should have the attempts count increased. We should NOT manage any offsets unless used manually. |
| `pro/consumption/strategies/lrj/ftr_mom_vp/attempts_tracking_with_non_recoverable_errors_spec.rb` | When running jobs with non-recoverable errors, we should have the attempts count increased We should not commit any offsets |
| `pro/consumption/strategies/lrj/ftr_mom_vp/attempts_tracking_without_errors_spec.rb` | When running jobs without problems, there should always be only one attempt even if throttling occurs |
| `pro/consumption/strategies/lrj/ftr_mom_vp/fast_non_throttled_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj and they should behave the same way as once without throttling enabled |
| `pro/consumption/strategies/lrj/ftr_mom_vp/occasional_marking_spec.rb` | When using manual offset management and not marking often, we should have a smooth processing flow without extra messages or anything. |
| `pro/consumption/strategies/lrj/ftr_mom_vp/parallel_non_lrj_with_lrj_spec.rb` | When using LRJ within a consumer group with other non-LRJ, the LRJ should be running while other jobs are consumed and they should not wait (as long as enough workers) and throttling should not cause any problems |
| `pro/consumption/strategies/lrj/ftr_mom_vp/processing_with_expired_throttling_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good. It should continue processing after resume and should pick up next messages. Expired throttling should just seek correctly as expected and allow us to move on |
| `pro/consumption/strategies/lrj/ftr_mom_vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/lrj/ftr_mom_vp/shutdown_order_spec.rb` | When running lrj, Karafka should never run the shutdown operations while consumption is in progress |
| `pro/consumption/strategies/lrj/ftr_mom_vp/with_continuous_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if it happens often. We should retry from 0 because we do not commit offsets here at all We should not change the offset |
| `pro/consumption/strategies/lrj/ftr_mom_vp/with_long_manual_pause_when_throttled_spec.rb` | Karafka should not resume when manual pause is in use for LRJ. Karafka should not throttle (unless in idle which would indicate pause lift) when manually paused After un-pause, Karafka may do a full throttle if the previous throttling time did not finish Offsets should not be changed |
| `pro/consumption/strategies/lrj/ftr_mom_vp/with_never_ending_error_spec.rb` | When doing work with error, we should slowly increase the attempt count for LRJ same as for regular workloads, despite pausing. |
| `pro/consumption/strategies/lrj/ftr_mom_vp/with_throttle_and_error_spec.rb` | Karafka should be able to recover from non-critical error when using lrj the same way as any normal consumer even if throttled. Error flow should be the same as non-throttled. |
| `pro/consumption/strategies/lrj/ftr_mom_vp/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. |
| `pro/consumption/strategies/lrj/immediate_completion_spec.rb` | LRJ should handle jobs that complete immediately without any actual long-running work, ensuring proper resource management and offset handling. |
| `pro/consumption/strategies/lrj/mom/occasional_marking_spec.rb` | When using manual offset management and not marking often, we should have a smooth processing flow without extra messages or anything. |
| `pro/consumption/strategies/lrj/mom/with_manual_pause_on_early_spec.rb` | When pausing not on a last message, we should un-pause from it and not from the next incoming. |
| `pro/consumption/strategies/lrj/mom/with_manual_seek_spec.rb` | Manual seek per user request should super-seed the automatic LRJ movement. |
| `pro/consumption/strategies/lrj/mom/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. |
| `pro/consumption/strategies/lrj/mom_vp/without_marking_spec.rb` | When using manual offset management and not marking anything at all, we should not change offsets but we should keep moving forward in the processing. VPs should not interact with this flow |
| `pro/consumption/strategies/lrj/vp/lrj_with_vp_from_earliest_spec.rb` | Virtual partitions and long-running jobs should not have any problems working together |
| `pro/consumption/strategies/lrj/vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/lrj/vp/with_in_the_middle_revoked_spec.rb` | When we start processing VP work with LRJ and some of the virtual partitions get revoked, they should not run if they were in the jobs queue. |
| `pro/consumption/strategies/mom/ftr/continuity_after_throttle_expires_spec.rb` | When we throttle with MoM enabled and we process longer than the throttle, it should not have any impact on the processing order. It should also not mark offsets in any way. |
| `pro/consumption/strategies/mom/ftr/with_manual_offset_and_error_spec.rb` | When using manual offset management with throttling, an error should move us back to where we had the last offset, but throttle backoff should continue linear. |
| `pro/consumption/strategies/mom/ftr_vp/continuity_after_throttle_expires_spec.rb` | When we throttle with MoM enabled and we process longer than the throttle, it should not have any impact on the processing order. It should also not mark offsets in any way. |
| `pro/consumption/strategies/mom/ftr_vp/marking_ahead_with_no_previous_in_batch_start_spec.rb` | When marking ahead, where there is no current offset to materialize on first batch, no offset should be marked and we should start from zero again Throttling should not impact it in any way |
| `pro/consumption/strategies/mom/ftr_vp/with_manual_offset_and_error_spec.rb` | When using manual offset management with throttling, an error should move us back to where we had the last offset, but throttle backoff should continue linear. |
| `pro/consumption/strategies/mom/ftr_vp/with_never_committed_errors_on_a_second_batch_spec.rb` | If we never mark offsets, upon errors we should start from beginning Throttling should not impact this in any way |
| `pro/consumption/strategies/mom/ftr_vp/with_reversed_offsets_marking_and_error_spec.rb` | If we mark virtual offsets that cannot be materialized to a state, we should start from beginning on errors. Throttling should not impact it in any way |
| `pro/consumption/strategies/mom/vp/marking_ahead_with_no_previous_in_batch_start_spec.rb` | When marking ahead, where there is no current offset to materialize on first batch, no offset should be marked and we should start from zero again |
| `pro/consumption/strategies/mom/vp/with_never_committed_errors_on_a_second_batch_spec.rb` | If we never mark offsets, upon errors we should start from beginning |
| `pro/consumption/strategies/mom/vp/with_partial_marking_spec.rb` | When we mark offsets in the middle of each, we should never end up with last marked |
| `pro/consumption/strategies/mom/vp/with_reversed_offsets_marking_and_error_spec.rb` | If we mark virtual offsets that cannot be materialized to a state, we should start from beginning on errors |
| `pro/consumption/strategies/mom/with_manual_offset_and_error_spec.rb` | Using manual offset management under rebalance and error happening, we should start from the last place that we were, even when there were many batches down the road and no checkpointing |
| `pro/consumption/strategies/vp/balanced_distribution_spec.rb` | When using Virtual Partitions with balanced distribution, we should ensure that: 1. Messages with the same key are processed together 2. Larger groups are processed first 3. Messages within each group maintain their order 4. Work is distributed across available workers Please note, that cross-batch VP assignment is not guaranteed with this distribution strategy. |
| `pro/consumption/strategies/vp/cancelled_vps_due_to_bad_partitioning_spec.rb` | This PR illustrates how partitioning based on constant payload content basically cancels the potential of VPs by creating only single virtual partition. |
| `pro/consumption/strategies/vp/collapsing/collapse_and_continuity_spec.rb` | Karafka when with VP upon error should collapse the whole collective batch and should continue processing in the collapsed mode after a back-off until all the "infected" messages are done. After that, VPs should be resumed. |
| `pro/consumption/strategies/vp/collapsing/collapse_with_intermediate_marking_filtering_spec.rb` | After we collapse, we should skip messages we marked as consumed, except those that were not processed. |
| `pro/consumption/strategies/vp/collapsing/continuous_retry_in_collabse_with_filtering_spec.rb` | When we continue to have an error on the same event in the collapse mode, others should be filtered for as long as the collapse lasts. |
| `pro/consumption/strategies/vp/collapsing/manual_collapse_and_continuity_spec.rb` | Karafka when with VPs should support manual collapsing. After we explicitely collapse VPs, the collapse should continue for as long as we wanted |
| `pro/consumption/strategies/vp/collapsing/manual_collapse_state_collisions_spec.rb` | When we collapse VPs manually, the `#collapse?` should NOT change for the currently running operation and only should collapse later |
| `pro/consumption/strategies/vp/collapsing/short_manual_collapse_spec.rb` | When Karafka collapses for a short time we should regain ability to process in VPs |
| `pro/consumption/strategies/vp/collapsing/with_pause_on_collapsed_spec.rb` | While VPs do not support pausing in the regular flow, we can pause while running VP when collapsed. This can be used to provide a manual back-off if we would want. |
| `pro/consumption/strategies/vp/collective_state_awareness_break_spec.rb` | We should be able to use `#failing?` to detect, that part of our work has already failed and that our current set of VPs will collapse. This can be used to stop processing when we know, it is going to be re-processed again |
| `pro/consumption/strategies/vp/consistent_distribution_spec.rb` | Karafka should always assign same consumer instance to the same virtual partitioner result. In case data from few virtual partitions is merged into one chunk, the partition should always stay the same (consistent). |
| `pro/consumption/strategies/vp/custom_expensive_message_scheduling_spec.rb` | This integration test demonstrates how to override the default LJF (Longest Job First) scheduling algorithm to implement custom logic that identifies and prioritizes Virtual Partitions containing expensive messages. This ensures expensive messages are processed first, allowing them to run in parallel with other messages rather than blocking at the end. Real-world scenario: In production, some messages require extensive processing (1+ minute) while others complete quickly (<1 second). With Virtual Partitions and limited worker threads, default LJF scheduling might not optimally handle this, causing idle workers while the expensive message blocks progress. |
| `pro/consumption/strategies/vp/custom_partitioning_engine_spec.rb` | When using Pro, we should be able to redefine and change the whole partitioner and use our own custom one without any problems. This spec illustrated, that you can overwrite the core partitioner and use your own that distributes work differently for different topics with awareness of the received batch size. |
| `pro/consumption/strategies/vp/different_coordinator_different_partitions_spec.rb` | Karafka should not use the same coordinator for jobs from different partitions |
| `pro/consumption/strategies/vp/distributing_work_randomly_spec.rb` | Karafka should support possibility of distributing work randomly when using virtual partitions Note that even when using random distribution, messages from different partitions will never mix within a batch. |
| `pro/consumption/strategies/vp/distributing_work_using_message_keys_spec.rb` | Karafka should support possibility of using message keys to distribute work We have two partitions but virtual partitioner should allow us to distribute this work across four threads concurrently. Note that you can get different combinations of messages for different batches fetched. The fact that the first time messages with key `a` were together with `c`, does not mean, that it will always be the same. The distribution combination is unique for the batch. One thing you can be sure, is that if you have messages with key `c`, they will always go to one of the virtual consumers. Virtual consumer instance is **not** warrantied. |
| `pro/consumption/strategies/vp/empty_and_nil_keys_spec.rb` | Virtual partitions should handle edge cases with empty keys, nil keys, and keys that result in invalid partitioning scenarios without crashing the consumer. |
| `pro/consumption/strategies/vp/errors_tracking/collapsed_consecutive_errors_accumulation_spec.rb` | When using virtual partitions and tracking errors, they under consecutive collapse should grow in terms of size |
| `pro/consumption/strategies/vp/errors_tracking/consecutive_errors_spec.rb` | When using virtual partitions and tracking errors, under collapse they should be present collectively from many partitions |
| `pro/consumption/strategies/vp/from_earliest_patterned_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) using multiple threads based on the used virtual partitioner. We should use more than one thread for processing of all the messages. This should also work as expected for pattern based topics. |
| `pro/consumption/strategies/vp/from_earliest_spec.rb` | Karafka should be able to easily consume all the messages from earliest (default) using multiple threads based on the used virtual partitioner. We should use more than one thread for processing of all the messages |
| `pro/consumption/strategies/vp/groups_aggregation_halt_on_recovery_spec.rb` | When using virtual partitions with errors during consumption, the virtual offset manager should limit group accumulation and halt further grouping during recovery to prevent memory issues |
| `pro/consumption/strategies/vp/many_batches_same_key_spec.rb` | When using virtual partitions and having a partitioner that always provides the same key, we should always use one thread despite having more available |
| `pro/consumption/strategies/vp/many_batches_spec.rb` | When using virtual partitions, we should easily consume data with the same instances on many batches and until there is a rebalance or critical error, the consumer instances should not change |
| `pro/consumption/strategies/vp/many_distributed_batches_end_order_spec.rb` | When consuming data with virtual partitions from many batches, the order of messages in between the single partition batches should be preserved. |
| `pro/consumption/strategies/vp/pausing_concurrently_spec.rb` | When we invoke pause from multiple VPs, the last one should win |
| `pro/consumption/strategies/vp/pre_work_marking_retry_continuity_spec.rb` | After an error, in case we marked before the processing, we should just skip the broken and move on as we should just filter out broken one. |
| `pro/consumption/strategies/vp/regular_processing_one_by_one_without_errors_spec.rb` | Karafka should be able to just process all the messages one after another |
| `pro/consumption/strategies/vp/round_robin_with_direct_reducer_spec.rb` | When using a round-robin partitioner, Karafka should assign messages correctly to utilize all VPs. We test two instances to make sure that they operate independently Since the default reducer does not work perfectly with all concurrency settings, we can use a custom reducer to match the virtual key with partitions 1:1. |
| `pro/consumption/strategies/vp/shared_coordinator_spec.rb` | Karafka should use the same coordinator for all the jobs in a group |
| `pro/consumption/strategies/vp/synchronized_with_marking_spec.rb` | Karafka should be able to run marking from a synchronization block and not crash despite using the same lock. This ensures, that user can run synchronized code that will also mark and that our internal synchronization is aligned with it. |
| `pro/consumption/strategies/vp/with_cycling_distribution_spec.rb` | Karafka should be able to evenly distribute work when using a cycling partitioner Since we do not have multiple partitions/topics in this example, there are no locks around it, but in complex cases there should be a more complex cycling engine |
| `pro/consumption/strategies/vp/with_error_in_partitioner_reduced_spec.rb` | When errors occur in partitioner, it should reduce to one partition whole batch |
| `pro/consumption/strategies/vp/with_error_in_partitioner_spec.rb` | When errors occur in partitioner, the partitioning should be reduced and the error should not leak out to the listener. Leaking could cause the total crash and critical recovery flow. |
| `pro/consumption/strategies/vp/with_error_in_the_middle_marking_spec.rb` | When using VPs and marking, we should end up with the last materialized offset sent to Kafka while the rest of things should be picked up until this offset |
| `pro/consumption/strategies/vp/with_error_on_another_batch_spec.rb` | When using virtual partitions, we should easily consume data with the same instances on many of the batches even if error occurred. |
| `pro/consumption/strategies/vp/with_error_spec.rb` | When Karafka consumes in the VP mode and error happens in any of the processing units we allow the rest to finish the work and we restart the processing from the first offset on a batch in a collapsed mode. This spec raises only one error once |
| `pro/consumption/strategies/vp/with_external_each_patch_spec.rb` | This spec verifies that VP strategy works correctly when external libraries monkey-patch the Messages#each method (e.g., for tracing/instrumentation purposes). The VP strategy internally uses messages.raw.map(&:offset) which bypasses the patched each method by accessing the underlying array directly. @see https://github.com/karafka/karafka/issues/2939 |
| `pro/consumption/strategies/vp/with_headers_usage_spec.rb` | Ruby had some header related bugs in 3.0.2. This PR checks that all works as expected |
| `pro/consumption/strategies/vp/with_limited_concurrency_spec.rb` | When using Virtual Partitions with limited max partitions, VP should not occupy all the threads but it should use at most what was allowed. This allows for having some worker threads that are always available for other work. |
| `pro/consumption/strategies/vp/with_more_vps_than_workers_spec.rb` | When using Virtual Partitions, we can distribute work in a way that allows us to gain granular control over what goes to a single virtual partition. We can create virtual partition based on any of the resource details |
| `pro/consumption/strategies/vp/with_parallel_errors_spec.rb` | When Karafka consumes in the VP mode and many errors happen in many of the processing units, we we should continue and we should restart the processing from the first offset on a batch in a collapsed mode until the collective offset is passed |
| `pro/consumption/strategies/vp/with_soft_marking_partial_error_spec.rb` | When using `mark_as_consumed` in virtual partitions, we should do virtual marking with correct state location materialization on errors. This prevents us from excessive re-processing because as much data as possible is marked as consumed |
| `pro/consumption/strategies/vp/without_intermediate_marking_spec.rb` | Karafka should mark correctly the final offset of collective group upon finish |
| `pro/consumption/time_based_non_blocking_locking_spec.rb` | We should be able to lock in a non-blocking fashion and then lock should expire based on time |
| `pro/consumption/transactions/after_transaction_error_spec.rb` | If transaction finishes and the error is after it, it should not impact the offset nor the location where we retry |
| `pro/consumption/transactions/async_produce_partial_failure_recovery_spec.rb` | This test verifies async production failure handling within transactions: 1. Produces 15 messages (offsets 0-14) to a topic 2. Each message produces async to its own unique target topic 3. mark_as_consumed is called after all async productions 4. On first attempt processing offset 1, inject a failure We verify that if the transaction completes without error, all async productions have been successfully acknowledged, so the callback cannot indicate failure later. This tests 15 total messages across 15 different topics. Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/async_produce_with_periodic_ticks_spec.rb` | This test verifies that transactions with async productions work correctly when using periodic ticks. Ensures that both consume and tick methods can safely use transactions with async productions. Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/async_production_failure_rollback_spec.rb` | This test verifies that when one of multiple produce_async operations fails within a transaction, the entire transaction rolls back including: 1. No messages are committed to any target topics 2. The offset is not marked as consumed 3. Messages can be reprocessed Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/concurrent_async_produce_stress_spec.rb` | Stress test for concurrent async productions within transactions. This test simulates high-throughput scenarios with: 1. Large message batches 2. Multiple async productions per message 3. Multiple concurrent consumers processing in parallel This verifies that under concurrent load: - All async productions complete successfully - No messages are lost - Offsets are correctly tracked - Transaction atomicity is maintained Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/connection_pool_pristine/flow_spec.rb` | Karafka should correctly handle transactional producers from a ConnectionPool with multiplexing, ensuring transactional integrity across concurrent consumers |
| `pro/consumption/transactions/dlq/with_crash_during_transational_dispatch_spec.rb` | In case transactional offset dispatch on post-error happens, Karafka should retry processing again and again. |
| `pro/consumption/transactions/dlq/with_transactional_dispatch_spec.rb` | We should be able to use transactional producer for DLQ dispatches |
| `pro/consumption/transactions/early_exit_spec.rb` | When attempting to exit a transaction block early using break, Karafka should raise EarlyTransactionExitNotAllowedError to prevent incomplete transactions |
| `pro/consumption/transactions/empty_transaction_spec.rb` | Running a transaction that does not publish data and does not store offsets should be ok |
| `pro/consumption/transactions/handler_wait_after_transaction_spec.rb` | This test verifies the behavior when explicitly waiting on handlers after a transaction finished. Once the transaction block exits without error, the transaction is committed and handlers should indicate success, even if checked afterwards. This addresses the question: "Does this mean the transaction commits even if produce_async failed later on?" The answer should be: if the transaction block completed, all async operations have been acknowledged. Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/large_batch_async_atomicity_spec.rb` | This test verifies atomicity guarantees for large batch processing with many async productions. Ensures that even with hundreds of async produce operations, either all succeed or all fail together with the offset marking. Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/lost_assignment_post_marking_late_produce_spec.rb` | We should be able to mark as consumed when we own the assignment and produce messages but if at the finalization moment we lost the assignment, we should fail the transaction with the assignment lost error |
| `pro/consumption/transactions/lost_assignment_with_offset_storage_spec.rb` | We should NOT be able to mark as consumed within a transaction on a lost partition because the transaction is expected to fail. |
| `pro/consumption/transactions/lost_assignment_without_offset_storage_spec.rb` | We should be able to use producer-only transactions even after we have lost the assignment |
| `pro/consumption/transactions/many_levels_error_spec.rb` | Nested transactions should not be allowed. |
| `pro/consumption/transactions/many_markings_spec.rb` | Karafka should track offsets in transaction but only mark last on success |
| `pro/consumption/transactions/marking_on_current_assignment_spec.rb` | When we mark with a transactional producer outside of the transactional block and we own the assignment, marking should return true |
| `pro/consumption/transactions/marking_on_lost_assignment_spec.rb` | When we mark as consumed outside of the transactional block but by using the transactional producer, in a case where we were not able to finalize the transaction it should not raise an error but instead should return false like the non-transactional one. This means we no longer have ownership of this partition. |
| `pro/consumption/transactions/mixed_sync_async_produce_spec.rb` | This test verifies behavior when mixing synchronous and asynchronous productions within the same transaction. Both types should be properly handled and committed atomically together with offset marking. Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/multi_topic_batch_async_produce_spec.rb` | This test simulates a realistic batch processing scenario where messages from one topic are transformed and dispatched to multiple target topics asynchronously within a transaction. Each message produces to multiple topics, creating a fan-out pattern. This verifies: 1. Batch async productions complete atomically 2. All target topics receive all expected messages 3. Offset marking waits for all async handlers across all topics 4. Transaction semantics are preserved in complex multi-topic scenarios Note: This spec works correctly regardless of how Kafka batches messages for delivery. |
| `pro/consumption/transactions/multiple_async_produce_with_mom_spec.rb` | When using transactions with multiple produce_async calls and manual offset management, we need to ensure that: 1. All async productions complete successfully before the transaction commits 2. The offset marking waits for all handlers 3. If any production fails, the entire transaction should roll back This test addresses the concern that mark_as_consumed might not wait for async handlers and could lead to skipped message processing if later productions fail. |
| `pro/consumption/transactions/nested_handler_error_marking_impact_spec.rb` | In case we mark as consumed after a nested transactional error, it should not reset anything and go as planned |
| `pro/consumption/transactions/nested_handler_error_marking_pre_impact_spec.rb` | In case we mark as consumed after a nested transactional error, it should not reset anything and go as planned |
| `pro/consumption/transactions/one_level_error_spec.rb` | If transaction fails, the offset marked internally should not propagate |
| `pro/consumption/transactions/one_level_standard_spec.rb` | Karafka should be able to use transactions with offset storage with metadata |
| `pro/consumption/transactions/post_revocation_cooperative_reclaim_spec.rb` | When a transactional consumer goes into a cooperative-sticky rebalance and gets the partitions back, it should not have duplicated data. |
| `pro/consumption/transactions/post_revocation_non_cooperative_reclaim_spec.rb` | When a transactional consumer goes into a non-cooperative-sticky rebalance and gets the partitions back, it should not have duplicated data. |
| `pro/consumption/transactions/vps/all_ok_current_spec.rb` | When working with transactions if all success and the strategy is current, last should be picked |
| `pro/consumption/transactions/vps/all_ok_exact_spec.rb` | When working with transactions if all success and the strategy is current, last should be picked |
| `pro/consumption/transactions/vps/collapsed_transaction_error_spec.rb` | When we operate in a collapsed mode, transactions should operate in a linear fashion |
| `pro/consumption/transactions/vps/collapsed_transaction_ok_spec.rb` | When we operate in a collapsed mode, transactions should operate in a linear fashion |
| `pro/consumption/transactions/vps/lost_assignment_with_offset_storage_spec.rb` | We should NOT be able to mark as consumed within a transaction on a lost partition because the transaction is expected to fail. |
| `pro/consumption/transactions/vps/lost_assignment_without_offset_storage_spec.rb` | We should be able to use producer-only transactions even after we have lost the assignment as offset management may not be transactional |
| `pro/consumption/transactions/vps/nested_handler_error_marking_impact_spec.rb` | In case we mark as consumed after a nested transactional error, it should not reset anything and go as planned |
| `pro/consumption/transactions/vps/nested_handler_error_marking_pre_impact_spec.rb` | In case we mark as consumed after a nested transactional error, it should not reset anything and go as planned |
| `pro/consumption/transactions/vps/nested_transaction_error_spec.rb` | In VPs we should also be safeguarded when trying to start transaction inside of a transaction |
| `pro/consumption/transactions/vps/one_ok_rest_error_spec.rb` | If transaction fails we should mark as consumed only to the consecutive offset that was reached |
| `pro/consumption/transactions/vps/post_transaction_error_spec.rb` | If there is error after the transaction, the offset should be recorded with current metadata |
| `pro/consumption/transactions/with_abort_spec.rb` | When a transaction is aborted, it should also fire the consuming event as abort similar to ActiveRecord transaction does not propagate and is handled internally, thus no error. |
| `pro/consumption/transactions/with_injected_producer_and_marking_spec.rb` | In case our main producer is not transactional or for any other reason, we should be able to inject a transactional one and use it if we want. |
| `pro/consumption/transactions/with_injected_producer_spec.rb` | In case our main producer is not transactional or for any other reason, we should be able to inject a transactional one and use it if we want. |
| `pro/consumption/transactions/with_lag_less_statistics_spec.rb` | When offsets are part of producer transactions, they will no longer appear in the consumer `statistics.emitted` events and should be compensated via Karafka instrumentation. |
| `pro/consumption/transactions/with_marking_tracking_via_instrumentation_spec.rb` | When using transactions, we should be able to track the stored offset by taking the seek offset |
| `pro/consumption/transactions/with_non_transactional_post_transaction_marking_attempt_spec.rb` | In case our main producer is not transactional and we mark in a transactional way, we should not be allowed to mark again in a non-transactional way Note, that failing marking will happen automatically because of lack of mom in the routing |
| `pro/consumption/transactions/with_wrapped_producer_spec.rb` | We should be able to replace the default producer with a transactional one We set the default producer to a broken location so in case our wrapping would not work, it will crash |
| `pro/consumption/with_granular_errors_tracker_tracking_spec.rb` | Error counting should happen per error class |
| `pro/consumption/with_never_ending_error_spec.rb` | When doing work with error, we should slowly increase the attempt count |
| `pro/consumption/with_pro_scheduler_from_earliest_spec.rb` | Karafka should use more than one thread to consume independent topics with the pro scheduler Here we just aim to ensure, that we schedule all the jobs and that things operate as expected |
| `pro/encryption/consumption/with_encrypted_correctly_spec.rb` | Karafka should work correctly when encryption is enabled |
| `pro/encryption/consumption/with_encrypted_with_old_public_key_spec.rb` | When using old public key to produce messages but this key is one of the supported, despite having new version, the message should be decrypted easily |
| `pro/encryption/consumption/with_key_mismatch_and_fingerprint_spec.rb` | When producing with version that is not supported in reading, it should raise an error |
| `pro/encryption/consumption/with_key_mismatch_spec.rb` | When using public key to publish and a key that is not matching on version, we should get OpenSSL error or (depending on time and libs versions) random data |
| `pro/encryption/consumption/with_only_public_key_for_producing_spec.rb` | Karafka should work correctly when we configure it only to have public key and for messages producing only. Decryption will not be possible. |
| `pro/encryption/consumption/with_unencrypted_message_spec.rb` | When despite using encryption we get an unencrypted message, all should work. This is expected as some producers may not yet be migrated, etc. |
| `pro/encryption/consumption/without_proper_private_key_version_spec.rb` | When producing with version that is not supported in reading, it should raise an error |
| `pro/encryption/setup/config_scope_not_in_use_spec.rb` | When encryption is not enabled, we should not inject or configure its components |
| `pro/encryption/setup/config_scope_presence_spec.rb` | When using correct public and private keys, there should be no issues. |
| `pro/encryption/setup/config_scope_validation_spec.rb` | When providing invalid config details for encryption, validation should kick in. |
| `pro/instrumentation/aj_per_job_consumption_events_spec.rb` | Karafka should instrument on particular active job jobs and should include details allowing for correlation of jobs with topic, messages, etc |
| `pro/instrumentation/first_idle_messages_metadata_spec.rb` | When the idle job kicks in before we had a chance to process any data, it should still have access to empty messages batch with proper offset positions (-1001) and no messages. It should also kick in proper instrumentation event that we can use prior to scheduling |
| `pro/instrumentation/vendors/appsignal/ticking_tracking_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications Majority of appsignal is tested in OSS so here we focus only on ticking that is a Pro feature |
| `pro/instrumentation/vendors/datadog/tick_error_logger_spec.rb` | Here we subscribe to our listener and make sure nothing breaks during the notifications We use a dummy client that will intercept calls that should go to DataDog |
| `pro/instrumentation/vendors/kubernetes/downscaled_multiplexing_support_spec.rb` | When Karafka downscales the connections as part of resources management, liveness should be as the downscaled threads should deregister themselves. |
| `pro/instrumentation/vendors/kubernetes/liveness_from_ticking_spec.rb` | When we only tick, it should be considered good as long as within time boundaries |
| `pro/iterator/all_early_stops_spec.rb` | When we stop processing early on all partitions, things should just stop. |
| `pro/iterator/custom_max_wait_time_spec.rb` | Karafka should use the custom poll time when defined |
| `pro/iterator/from_different_offsets_to_the_end_spec.rb` | When iterating over partitions and starting from different offsets, we should reach the end and data should be as expected. |
| `pro/iterator/future_time_lookup_spec.rb` | When we want to get something from the future and there is nothing, we should just stop |
| `pro/iterator/long_living_iterator_spec.rb` | Karafka should have a way to create long living iterators that wait for messages |
| `pro/iterator/long_living_time_based_iterator_spec.rb` | Karafka should be able to exit from iterator even if no more messages are being shipped |
| `pro/iterator/named_earliest_spec.rb` | We should be able to use earliest reference |
| `pro/iterator/named_latest_spec.rb` | We should be able to use latest reference but with eol on, it may end up with nothing |
| `pro/iterator/negative_multiple_topics_lookups_spec.rb` | We should be able to subscribe to multiple topics with custom per topic negative lookups and they should work on all partitions |
| `pro/iterator/negative_topic_multiple_partitions_spec.rb` | We should be able to use different negative offsets for different partitions of the same topic |
| `pro/iterator/stop_start_with_marking_spec.rb` | We should be able with proper settings |
| `pro/iterator/with_custom_group_id_spec.rb` | The iterator should use the group.id specified in the settings hash when provided. By default, the iterator uses its own admin group ID (not the routing-defined consumer group), but this can be overridden via the settings hash. This is useful when you want to: - Track iterator progress for resuming later - Use a specific consumer group for offset management - Share offset state between multiple iterator instances |
| `pro/iterator/with_granular_cleaning_spec.rb` | We should be able to clean messages while keeping the metadata |
| `pro/iterator/with_many_partitions_all_subscription_to_the_end_spec.rb` | When iterating over the topic and not providing partitions, it should stream all the messages one after another and should use JSON also for case where there are multiple partitions. |
| `pro/iterator/with_many_partitions_from_different_times_spec.rb` | When iterating over different topics/partitions with different times, each should start from the expected one. |
| `pro/iterator/with_many_partitions_from_same_time_spec.rb` | When iterating over the topic from the same time on all partitions, we should start from the expected on all and finish accordingly |
| `pro/iterator/with_many_partitions_from_the_same_offset_spec.rb` | When iterating over the topic from the same offset on all partitions, we should start from the expected on all and finish accordingly |
| `pro/iterator/with_many_partitions_one_early_stop_spec.rb` | When iterating over the topic, we should be able to stop iteration of any partition we want without impacting the others |
| `pro/iterator/with_many_partitions_one_middle_stop_spec.rb` | When iterating over the topic, we should be able to stop iteration of any partition we want without impacting the others even if we're in the middle of this partition. |
| `pro/iterator/with_negative_bigger_than_available_data_spec.rb` | When we request more data with negative lookup than there is in the partition, we should pick as much as there is and no more. |
| `pro/iterator/with_negative_lookup_for_last_messages_spec.rb` | When using negative lookups, we should get at most what we have requested |
| `pro/iterator/with_one_partition_to_the_end_spec.rb` | When iterating over the topic and not providing partitions, it should stream all the messages one after another and should use JSON. |
| `pro/iterator/with_pause_current_partition_usage_spec.rb` | When iterating over the topics, we should be able to use the pause current to stop only the current topic partition processing |
| `pro/licensing/configure_with_invalid_token_spec.rb` | Karafka should not start if the token is invalid |
| `pro/licensing/configure_with_valid_token_spec.rb` | Pro components should be loaded when we run in pro mode and a nice message should be printed |
| `pro/licensing/external_poro/loaded_via_custom_chain_spec.rb` | We should be able to use a license encrypted token as a setup source for Karafka and it should work as described in the integration docs |
| `pro/licensing/load_poro/configure_spec.rb` | Karafka in a PORO project should load without any problems |
| `pro/licensing/load_poro/early_components_visibility_spec.rb` | Karafka in a PORO project should load components after the require even prior to the setup, so we can use those when needed |
| `pro/licensing/post_checksum_verification_poro/verify_spec.rb` | Run the verification script post install to make sure it works as expected |
| `pro/pausing/with_per_topic_pause_setup_old_api_spec.rb` | When customizing the error pausing strategy using old API (setters), each topic should obey its own limitations This is a backwards compatibility test |
| `pro/pausing/with_per_topic_pause_setup_spec.rb` | When customizing the error pausing strategy, each topic should obey its own limitations |
| `pro/performance_tracking_spec.rb` | Karafka should track consumption rate metrics when pro This metrics tracker is then used internally for optimization purposes |
| `pro/rails/active_job/dispatching_async_job_spec.rb` | Karafka should be able to dispatch jobs using async pro adapter |
| `pro/rails/active_job/dispatching_sync_job_spec.rb` | Karafka should be able to dispatch jobs using sync pro adapter |
| `pro/rails/active_job/invalid_configuration_attempt_spec.rb` | When there is a misconfiguration of karafka options on ActiveJob job class, it should raise an error |
| `pro/rails/active_job/long_running_job_processing_spec.rb` | When a job is marked as lrj, it should keep running longer than max poll interval and all should be good |
| `pro/rails/active_job/long_running_jobs/fast_spec.rb` | Fast jobs should also not have any problems (though not recommended) when running as lrj |
| `pro/rails/active_job/long_running_jobs/with_error_spec.rb` | When we have an lrj job and it fails, it should use regular Karafka retry policies |
| `pro/rails/active_job/long_running_jobs/with_revocation_spec.rb` | When we have a batch of ActiveJobs jobs and we loose our partition assignment, jobs that did not start prior to the revocation should not start at all. To simulate this we will run long jobs from two partitions in parallel and we will "loose" one of them and detect this. |
| `pro/rails/active_job/long_running_jobs/with_shutdown_pickup_spec.rb` | Karafka with lrj ActiveJob when finishing in the middle of jobs on shutdown, should pick up where it stopped when started again We test it by starting a new consumer just to get the first message offset |
| `pro/rails/active_job/ordered_jobs_with_key_spec.rb` | When using the pro adapter, we should be able to use partitioner that will allow us to process ActiveJob jobs in their scheduled order using multiple partitions. We should be able to get proper results when using `:key`. |
| `pro/rails/active_job/ordered_jobs_with_partition_key_spec.rb` | When using the pro adapter, we should be able to use partitioner that will allow us to process ActiveJob jobs in their scheduled order using multiple partitions. We should be able to get proper results when using `:partition_key`. |
| `pro/rails/active_job/processing_few_on_shutdown_lrj_spec.rb` | Karafka with PRO should finish processing AJ jobs as fast as possible even if more were received in the batch. Since we are shutting down, those jobs will be picked up after Karafka is started again, so not worth waiting. The above should apply also to LRJ. |
| `pro/rails/active_job/processing_few_on_shutdown_spec.rb` | Karafka with PRO should finish processing AJ jobs as fast as possible even if more were received in the batch. Since we are shutting down, those jobs will be picked up after Karafka is started again, so not worth waiting |
| `pro/rails/active_job/virtual_partitions/from_earliest_spec.rb` | Parallel consumer should also work with ActiveJob, though it will be a bit nondeterministic unless we use headers data to balance work. |
| `pro/rails/active_job/virtual_partitions/with_error_on_next_job_spec.rb` | When we have an vp and not the first job fails, it should use regular Karafka retry policies for parallel jobs. It should mark previously done work as consumed accordingly to the VPs virtual offset management. |
| `pro/rails/active_job/virtual_partitions/with_error_spec.rb` | When we have an vp job and it fails, it should use regular Karafka retry policies for parallel jobs |
| `pro/rails/active_job/virtual_partitions/with_partitioning_using_key_spec.rb` | We should be able to mix partition delegation via `:key` with virtual partitions to achieve concurrent Active Job work execution. |
| `pro/rails/active_job/virtual_partitions/with_revocation_continuity_spec.rb` | When we have an vp AJ jobs in few batches, upon shutdown, not finished work should not be committed, but the previous offset should. Only fully finished AJ VP batches should be considered finished. |
| `pro/rails/active_job/virtual_partitions/with_revocation_spec.rb` | When we have a batch of ActiveJobs jobs and we loose our partition assignment, jobs that did not start prior to the revocation should not start at all. To simulate this we will have jobs on two partitions in parallel and we will "loose" one of them and detect this. We need to make consumption jobs long enough to jump with a rebalance in the middle. Since we internally mark as consumed on each job, we can be aware of revocation early enough. We need to run in an LRJ mode to make this happen, so rebalance does not block revocation. Note that for LRJ + VP, under shutdown we need to finish processing all because otherwise we might end up having extensive reprocessing |
| `pro/rails/active_job/with_envelope_spec.rb` | Karafka should be able to process ActiveJob jobs wrapped in a custom envelope This demonstrates a common pattern where all Kafka messages must follow a specific format with additional metadata wrapping the actual job payload Custom deserializer that wraps jobs in an envelope with metadata This extends the default Karafka ActiveJob deserializer |
| `pro/rails/rails72_pristine/just_a_dependency/railtie_setup_spec.rb` | Karafka+Pro should work with Rails 7 using the default setup Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails72_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails80_pristine/just_a_dependency/railtie_setup_spec.rb` | Karafka+Pro should work with Rails 8 using the default setup Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails80_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails81_pristine/just_a_dependency/railtie_setup_spec.rb` | Karafka+Pro should work with Rails 8.1 using the default setup Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails81_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails81_pristine/with-active_job_continuation/continuation_run_spec.rb` | Karafka Pro should work with Rails 8.1 ActiveJob Continuation feature This tests that jobs can be interrupted and resumed using the continuation API with Pro features Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails81_pristine/with-active_job_continuation/continuation_with_partitioning_spec.rb` | Karafka Pro should support combining ActiveJob Continuation with custom partitioning. This verifies that continuation jobs with the same partition key go to the same partition. The test validates that custom partitioning logic is preserved across job dispatches. Load all the Railtie stuff like when `rails server` |
| `pro/rails/rails81_pristine/with-active_job_continuation/continuation_with_scheduled_resume_spec.rb` | Karafka Pro should work with Rails 8.1 ActiveJob Continuation feature with scheduled resumes This tests that continuable jobs can use delayed resumes via Scheduled Messages Load all the Railtie stuff like when `rails server` |
| `pro/rebalancing/coordinator_replacement_after_rebalance_spec.rb` | Karafka should replace coordinator for consumer of a given topic partition after partition was taken away from us and assigned back |
| `pro/rebalancing/long_running_jobs/constant_rebalance_continuity_spec.rb` | When we consume data and several times we loose and regain partition, there should be continuity in what messages we pick up even if rebalances happens multiple times. This should apply to using LRJ as well. We may re-fetch certain messages but none should be skipped |
| `pro/rebalancing/long_running_jobs/continuity_after_error_and_rebalance_spec.rb` | When using the Long Running Job feature, in case partition is lost during the processing, after partition is reclaimed, process should pick it up and continue. It should not hang in the pause state forever. |
| `pro/rebalancing/long_running_jobs/coordinator_replacement_after_rebalance_spec.rb` | Karafka should replace coordinator for consumer of a given topic partition after partition was taken away from us and assigned back |
| `pro/rebalancing/long_running_jobs/rebalance_consumer_recreation_spec.rb` | When there is a rebalance and we get the partition back, we should start consuming with a new consumer instance. We should use one before and one after we got the partition back. |
| `pro/rebalancing/long_running_jobs/rebalance_continuity_spec.rb` | When a job is marked as lrj and there is a rebalance, we should be aware that our current instance had the partition revoked even if it is assigned back. The assignment back should again start from where it left |
| `pro/rebalancing/long_running_jobs/rebalance_continuity_without_marking_spec.rb` | When a job is marked as lrj and there is a rebalance, we should be aware that our current instance had the partition revoked even if it is assigned back. The assignment back should again start from where it left |
| `pro/rebalancing/long_running_jobs/revoke_continuity_spec.rb` | When Karafka looses a given partition but later gets it back, it should pick it up from the last offset committed without any problems |
| `pro/rebalancing/long_running_jobs/revoke_mark_as_consumed_enough_threads_spec.rb` | When Karafka tries to commit offset after the partition was revoked in a non-blocking way, it should return false for partition that was lost. It should also indicate when lrj is running that the revocation took place. It should indicate this before we mark as consumed as this state should be set on a consumer in the revocation job. This will work only when we have enough threads to be able to run the revocation jobs prior to finishing the processing. Otherwise when enqueued, will run after (for this we have another spec) |
| `pro/rebalancing/long_running_jobs/revoke_mark_as_consumed_not_enough_threads_spec.rb` | When Karafka tries to commit offset after the partition was revoked in a non-blocking way, it should return false for partition that was lost. It should also indicate when lrj is running that the revocation took place. It should not indicate this before we mark as consumed as this state could not be set on a consumer in the revocation job becuase it is pending in the queue. |
| `pro/rebalancing/long_running_jobs/revoke_mark_as_consumed_sync_enough_threads_spec.rb` | When Karafka tries to commit offset after the partition was revoked in a blocking way, it should return false for partition that was lost. It should also indicate when lrj is running that the revocation took place. It should indicate this before we mark as consumed as this state should be set on a consumer in the revocation job. This will work only when we have enough threads to be able to run the revocation jobs prior to finishing the processing. Otherwise when enqueued, will run after (for this we have another spec) |
| `pro/rebalancing/long_running_jobs/revoke_mark_as_consumed_sync_not_enough_threads_spec.rb` | When Karafka tries to commit offset after the partition was revoked in a blocking way, it should return false for partition that was lost. It should also indicate when lrj is running that the revocation took place. It should not indicate this before we mark as consumed as this state could not be set on a consumer in the revocation job becuase it is pending in the queue. |
| `pro/rebalancing/long_running_jobs/revoked_detection_spec.rb` | When a job is marked as lrj and a partition is lost, we should be able to get info about this by calling the `#revoked?` method. |
| `pro/rebalancing/on_partition_data_revoked_spec.rb` | Karafka should trigger a revoked action when a partition is being taken from us Initially we should own all the partitions and then after they are taken away, we should get back to two (as the last one will be owned by the second consumer). |
| `pro/rebalancing/revoke_continuity_spec.rb` | When Karafka Pro looses a given partition but later gets it back, it should pick it up from the last offset committed without any problems |
| `pro/rebalancing/saturated_execution_skip_spec.rb` | When we are over-saturated, the jobs that are in the queue should not run if the assignment was revoked even if the rebalance callback did not yet kick in. The `#revoked?` should be aware of unintended assignment. |
| `pro/rebalancing/transactions/revoke_continuity_spec.rb` | When Karafka looses a given partition but later gets it back, it should pick it up from the last offset committed without any problems |
| `pro/rebalancing/transactions/revoke_reclaim_continuity_cooperative_spec.rb` | When a consumer goes into a non-cooperative-sticky rebalance and gets the partitions back, it should not have duplicated data. |
| `pro/rebalancing/transactions/revoke_reclaim_continuity_spec.rb` | When a consumer goes into a non-cooperative-sticky rebalance and gets the partitions back, it should not have duplicated data. |
| `pro/rebalancing/virtual_partitions/coordinator_replacement_after_rebalance_spec.rb` | Karafka should replace coordinator for consumer of a given topic partition after partition was taken away from us and assigned back |
| `pro/rebalancing/virtual_partitions/rebalance_continuity_spec.rb` | When a job is marked as vp and there is a rebalance, we should be aware that our current instance had the partition revoked even if it is assigned back. The assignment back should again start from where it left |
| `pro/rebalancing/with_static_membership_reconnect_spec.rb` | Karafka process when stopped and started and configured with static membership should pick up the assigned work. It should not be reassigned to a different process. Karafka should maintain all the ordering and should not have duplicated. |
| `pro/recurring_tasks/defining_schedule_spec.rb` | We should be able to build up a schedule with recurring tasks that we can use |
| `pro/recurring_tasks/disable_enable_all_spec.rb` | We should be able to disable and enable back all the jobs in one go |
| `pro/recurring_tasks/disabled_state_replaying_spec.rb` | When we have an active schedule that got disabled, after a re-run we should be able to recover its inactive state |
| `pro/recurring_tasks/logging_data_published_on_errors_spec.rb` | When tasks are triggered, by default it should publish events to the logs topic even if those executions fail |
| `pro/recurring_tasks/logging_data_published_spec.rb` | When tasks are triggered, by default it should publish events to the logs topic This spec will hang if logs are not published |
| `pro/recurring_tasks/routes_creation_spec.rb` | When we decide to use recurring tasks, in specs it should automatically migrate and create appropriate topics |
| `pro/recurring_tasks/smarter_reconfiguration_spec.rb` | We should be able to reconfigure recurring tasks topics also via the direct config API This allows us to reconfigure things granularly. |
| `pro/recurring_tasks/state_publishing_frequency_spec.rb` | When we start cron and publish several events, the state should always be published after each command execution. This allows us to make assumptions about the topic data structure when buildings things like UI. |
| `pro/recurring_tasks/state_structure_spec.rb` | The state structure should comply with our expectations |
| `pro/recurring_tasks/trigger_all_spec.rb` | We should be able to trigger all tasks even if they are suppose to run once a year |
| `pro/recurring_tasks/version_downgrade_block_spec.rb` | When using versioned schedules, we should not be able to downgrade |
| `pro/recurring_tasks/version_upgrade_allowed_spec.rb` | When using versioned schedules, we should be able to upgrade There should be no exceptions or crashes and new schedule should work |
| `pro/recurring_tasks/with_changed_cron_between_versions_spec.rb` | When cron changes between versions, we should be using the new one. |
| `pro/recurring_tasks/with_completely_removed_schedule_spec.rb` | We should be able to remove all the tasks and it should not trigger them ever |
| `pro/recurring_tasks/with_simple_disabled_job_spec.rb` | Disabled job should not run even when manually triggered |
| `pro/recurring_tasks/with_simple_enabled_job_spec.rb` | When we decide to use recurring tasks and we have a job that never migrated from anything, it should work This spec may last up to 1 minute but we do want this as we want to test that it is invoked without explicit request to run |
| `pro/recurring_tasks/within_swarm_and_custom_producer_spec.rb` | Recurring tasks should work from a swarm with not used custom producer. If they don't this spec will run forever |
| `pro/recurring_tasks/within_swarm_spec.rb` | Recurring tasks should work from a swarm. If they don't this spec will run forever |
| `pro/recurring_tasks/without_any_schedules_spec.rb` | When we decide to use recurring tasks and we do not define a schedule, it should create one |
| `pro/recurring_tasks/without_fugit_pristine/fail_without_fugit_spec.rb` | Karafka+Pro should work with Rails 7 using the default setup Load all the Railtie stuff like when `rails server` |
| `pro/routing/direct_assignments/not_fulfilled_swarm_distribution_spec.rb` | When we decide to assign certain set of partitions and run in swarm, this set of partitions should match the nodes mapping We also should not be able to allocate partitions that are not assigned |
| `pro/routing/direct_assignments/with_mixed_group_spec.rb` | We should not be allowed to mix automatic and direct assignments |
| `pro/routing/direct_assignments/with_pattern_attempt_spec.rb` | We should not allow for usage of patterns with direct assignments |
| `pro/routing/dlq_with_vp_spec.rb` | Karafka should not allow for using VP with DLQ without retries |
| `pro/routing/limited_scope/with_inactive_and_disabled_spec.rb` | When combination of cli disabled topics and routing disabled topics meet, we should error. |
| `pro/routing/limited_scope/with_mix_but_inactive_without_consumer_spec.rb` | When we have inactive topics, they should be ok without consumer defined |
| `pro/routing/limited_scope/with_non_existing_consumer_groups_excluded_spec.rb` | When trying to exclude non existing consumer group, we should fail. |
| `pro/routing/limited_scope/with_non_existing_consumer_groups_spec.rb` | When trying to run non existing consumer group, we should fail. |
| `pro/routing/limited_scope/with_non_existing_subscription_groups_spec.rb` | When trying to run non existing subscription group, we should fail. |
| `pro/routing/limited_scope/with_non_existing_topics_spec.rb` | When trying to run non existing topics, we should fail. |
| `pro/routing/limited_scope/with_nothing_to_run_spec.rb` | When combination of consumer groups, subscription groups and topics we want to run is such, that they do not exist all together, we need to raise an error. |
| `pro/routing/limited_scope/with_only_inactive_topics_spec.rb` | When all our topics are disabled in routing, we should not allow Karafka to run |
| `pro/routing/multiplexing/ignore_whole_group_spec.rb` | When we decide to skip subscription group, it should skip all multiplexed sgs |
| `pro/routing/multiplexing/too_small_scale_delay_spec.rb` | We should not allow for extremely small scale delays |
| `pro/routing/multiplexing/validations_spec.rb` | Multiplexing details should be validated |
| `pro/routing/multiplexing/with_multiplexed_default_subscription_group_spec.rb` | Karafka should allow for multiplexing of the default anonymous subscription group |
| `pro/routing/multiplexing/with_multiplexed_subscription_group_spec.rb` | Karafka should allow for multiplexing subscription group |
| `pro/routing/non_pro_without_long_running_job_spec.rb` | When running non-pro, LRJ should not be available |
| `pro/routing/non_pro_without_virtual_partitioner_spec.rb` | When running non-pro, VP should not be available |
| `pro/routing/parallel_segments/naming_and_ids_spec.rb` | Proper segment names and ids should be generated |
| `pro/routing/parallel_segments/validations_flows_spec.rb` | Validations should work as expected and routing definitions should be ok |
| `pro/routing/patterns/anonymous_with_invalid_regexp_spec.rb` | When we have anonymous pattern with a non-regexp value, it should validate and fail |
| `pro/routing/patterns/excluded_by_given_name_spec.rb` | When having a named pattern that would be excluded by the CLI, it should not be used |
| `pro/routing/patterns/excluded_by_name_spec.rb` | When we define a pattern that gets assigned a matcher topic and this matcher topic is part of the topics we do not want to include (by assigned name) it should not use it |
| `pro/routing/patterns/included_by_automatic_name_spec.rb` | When we define a pattern that gets assigned a matcher topic and this matcher topic is part of the topics we want to include (by assigned name) it should be ok |
| `pro/routing/patterns/included_by_own_name_spec.rb` | When we define a pattern that gets assigned a matcher topic and this matcher topic is part of the topics we want to include (by given name) it should be ok |
| `pro/routing/patterns/kip_848_compatibility_spec.rb` | Karafka should work as expected with routing patterns using the KIP-848 consumer group protocol. This ensures that regex-based topic subscriptions work correctly with the consumer protocol, and that multiple consumer groups with same pattern can consume independently. |
| `pro/routing/patterns/named_with_invalid_regexp_spec.rb` | When we have a valid name but provide regexp that is not a regexp, we should fail |
| `pro/routing/patterns/named_with_same_in_many_consumer_groups_spec.rb` | It should be possible to use same named pattern twice in different consumer groups |
| `pro/routing/patterns/named_with_same_in_one_consumer_group_spec.rb` | It should not be possible to use same named pattern twice in same consumer group with different names. In case of same name, it will be considered equivalent of a multiplex |
| `pro/routing/patterns/named_with_same_regexp_in_one_consumer_group_spec.rb` | It should not be possible to use same named pattern with different name but same pattern in the same consumer group |
| `pro/routing/patterns/not_included_by_given_name_spec.rb` | When naming a pattern and then not including it, it should not be used |
| `pro/routing/patterns/not_included_by_name_spec.rb` | When we define a pattern that gets assigned a matcher topic and this matcher topic is not part of the topics we want to include (by assigned name), we should not include it. |
| `pro/routing/patterns/routing_definitions_spec.rb` | We should have ability to define patterns in routes for dynamic topics subscriptions It should assign virtual topics and patters to the appropriate consumer groups |
| `pro/routing/patterns/same_pattern_in_multiple_consumer_groups_spec.rb` | We should be able to define same pattern multiple times in the many consumer groups |
| `pro/routing/patterns/same_pattern_in_same_consumer_group_spec.rb` | We should not be able to define same pattern multiple times in the same consumer group |
| `pro/routing/patterns/with_non_existing_topics_excluded_spec.rb` | When we define a pattern that gets assigned a matcher topic and this matcher topic is not part of the exclusion, it should work |
| `pro/routing/patterns/without_pro_spec.rb` | When running non-pro, patterns should not be available |
| `pro/routing/pausing/with_bidirectional_compatibility_spec.rb` | Verify that both old and new pause configuration styles work and are bidirectionally compatible |
| `pro/routing/pausing/with_config_object_old_api_spec.rb` | Verify that the old pause configuration API (setters) still works correctly This is a backwards compatibility test |
| `pro/routing/pausing/with_config_object_spec.rb` | Verify that the new pausing config object approach works correctly |
| `pro/routing/pausing/with_custom_invalid_pausing_spec.rb` | When overwriting the default pausing strategy on a per topic basis with something invalid, validations should kick in and stop it |
| `pro/routing/pausing/with_custom_valid_old_api_spec.rb` | When altering the default pausing using old API (setters), it should not impact other topics This is a backwards compatibility test |
| `pro/routing/pausing/with_custom_valid_spec.rb` | When altering the default pausing, it should not impact other topics |
| `pro/routing/pro_with_long_running_job_spec.rb` | I should be able to define a topic consumption with long-running job indication It should not impact other jobs and the default should not be lrj |
| `pro/routing/pro_with_virtual_partitioner_spec.rb` | I should be able to define a topic consumption with virtual partitioner. It should not impact other jobs and the default should not have it. |
| `pro/routing/recurring_tasks/validations_flows_spec.rb` | When providing invalid config details for scheduled messages, validation should kick in. |
| `pro/routing/scheduled_messages/validations_flows_spec.rb` | When providing invalid config details for scheduled messages, validation should kick in. |
| `pro/routing/valid_with_features_usage_spec.rb` | Karafka should auto-load all the routing features |
| `pro/routing/with_different_backoff_settings_in_one_sg_spec.rb` | Karafka should not build separate SGs when altering pause settings per topic in a SG/CG |
| `pro/routing/with_same_topic_in_multiple_same_subscription_groups_spec.rb` | Karafka should not allow for same topic to be present in multiple subscription groups in the same consumer group with same subscription group name |
| `pro/routing/with_same_topic_in_multiple_sgs_diff_consumer_spec.rb` | Karafka should not allow for same topic to be present in multiple subscription groups in the same consumer group when trying to define different consumer classes. |
| `pro/routing/with_same_topic_in_multiple_subscription_groups_spec.rb` | Karafka should allow for same topic to be present in multiple subscription groups in the same consumer group as long as subscription groups have different names and same consumer class |
| `pro/scheduled_messages/active_job_future_job_spec.rb` | Karafka should be able to dispatch jobs via scheduled messages |
| `pro/scheduled_messages/active_job_past_job_spec.rb` | Karafka should schedule past messages directly, bypassing the scheduled messages completely |
| `pro/scheduled_messages/day_rotation_spec.rb` | When day ends, we should rotate it and move on Moving on should not trigger a second dispatch of already dispatched or cancelled events |
| `pro/scheduled_messages/dispatch_crash_recovery_spec.rb` | When dispatch of a given batch / message crashes, we should retry the same This gives us at-least once in case of non-transactional consumer and exactly-once in case of the transactional one |
| `pro/scheduled_messages/fast_close_past_schedules_spec.rb` | Messages scheduled close in past should work without any issues |
| `pro/scheduled_messages/future_scheduled_cancelling_spec.rb` | When we cancel future already loaded daily message, it should not go out |
| `pro/scheduled_messages/ordered_dispatches_spec.rb` | Messages going to one partition from one schedule topic partition should be always dispatched in order |
| `pro/scheduled_messages/past_timestamps_spec.rb` | Scheduled messages with past timestamps should be handled gracefully - either executed immediately or properly discarded based on configuration. |
| `pro/scheduled_messages/seeking_setup_flow_spec.rb` | When we define scheduled messages setup, it should have correct offset position configuration for offset reset |
| `pro/scheduled_messages/with_constant_non_eof_flow_spec.rb` | When there are constantly published messages, the state should be switched within acceptable time after the messages reach the current time. This should happen even when EOF is not triggered. |
| `pro/scheduled_messages/with_dispatched_except_one_for_dispatch_spec.rb` | When there are messages for dispatch but they were already dispatched (tombstone exists), we should not dispatch them again. We should only dispatch the once that were not |
| `pro/scheduled_messages/with_mostly_future_dispatches_spec.rb` | When there are future dispatches, they should not be dispatched unless the time is right Stats should be populated nicely though. One message (newest in the topic) should be dispatched though |
| `pro/scheduled_messages/within_swarm_spec.rb` | When running in swarm things should work as expected |
| `pro/scheduled_messages/without_any_messages_spec.rb` | When there are no messages at all, we should publish nothing except an empty state |
| `pro/shutdown/long_running_jobs/revocation_awareness_on_shutdown_spec.rb` | When running LRJ jobs upon shutdown, those jobs will keep running until finished or until reached max wait time. During this time, the rebalance changes should propagate and we should be able to make decisions also based on the revocation status. |
| `pro/shutdown/long_running_jobs/stopping_awareness_spec.rb` | When running a long running job, we should be able to detect that Karafka is stopping so we can early exit the job. Note, that for this to work correctly in regards to offsets, manual offset management need to be turned on. |
| `pro/shutdown/on_reached_quiet_spec.rb` | When we have reached quiet state, we should still be subscribed to what we had |
| `pro/shutdown/virtual_partitions/shutdown_on_all_consumers_spec.rb` | Karafka should run the shutdown on all the consumers that processed virtual partitions. |
| `pro/swarm/direct_granular_assignments_spec.rb` | We should be able to use direct assignments per node |
| `pro/swarm/direct_granular_ranges_spec.rb` | We should be able to use direct assignments per node with ranges |
| `pro/swarm/limited_per_node_spec.rb` | We should be able to instruct swarm to run consumption only on a certain node despite having more |
| `pro/swarm/low_multiplexing_on_many_topics_spec.rb` | Swarm should work with a single subscription group with many topics and many partitions. |
| `pro/swarm/with_liveness_downscaling_multiplex_spec.rb` | Liveness reporter should be totally ok with connections downscaling when using multiplexing and should not see it as a problem |
| `pro/swarm/with_liveness_exceeding_memory_spec.rb` | If we use liveness API to report issue, Karafka should restart the node |
| `pro/swarm/with_unreachable_nodes_spec.rb` | Karafka should fail when we define nodes that cannot be reached |
| `pro/swarm/with_unused_nodes_spec.rb` | Karafka should fail when we define nodes that do not have any assignments |
| `pro/testing/rspec/multiplexing_spec.rb` | We should be able to use testing with multiplexing without any exceptions |
| `pro/web/commanding/quiet_spec.rb` | Karafka should react to direct quiet from commanding |
| `pro/web/commanding/stop_usage_spec.rb` | Karafka should react to direct stop from commanding |
| `pro/web/commanding/trace_spec.rb` | Karafka should react to probing and should create trace result in the commands topic |
| `pro/web/commanding/wildcard_quiet_spec.rb` | Karafka should react to wildcard quiet from commanding |
| `pro/web/commanding/wildcard_stop_usage_spec.rb` | Karafka should react to stop in the wildcard mode |
| `pro/web/dlq_dispatching_deadlock_flow_spec.rb` | Karafka Web should handle high-concurrency DLQ message dispatching without deadlocking, ensuring tracking data is properly synchronized across many partitions |
| `pro/web/dlq_tracing_match_spec.rb` | Error references should match the DLQ traces |
| `pro/web/from_earliest_spec.rb` | Karafka should be able to consume and web tracking should not interfere |
| `pro/web/with_errors_spec.rb` | Karafka should be able to recover from non-critical errors and web tracking instrumentation should not break anything and should not crash |
| `pro/web/with_idle_and_fast_shutdown_spec.rb` | When the idle job kicks in before we had a chance to process any data, it should still have access to empty messages batch with proper offset positions (-1001) and no messages. Web tracking should pick it up and not fail |
| `pro/web/with_tick_and_eofed_errors_spec.rb` | When we use web-ui and error happens during eofed and tick when no offsets were marked as consumed, web UI should not fail. It used to fail because it was trying to extract offsets from not (yet) existing marking |
| `production/async_produce_queue_full_spec.rb` | This spec tests the behavior of produce_async when the producer's internal queue is full. Real-world scenario: Messages to an unknown/unreachable topic pile up in the queue (can't be delivered), eventually filling the queue. When full, subsequent produce_async calls raise immediately with WaterDrop::Errors::ProduceError. Key insight: This is different from unknown topic errors - queue full is a resource exhaustion issue that manifests as an immediate exception. |
| `production/async_produce_unknown_topic_spec.rb` | This spec tests the behavior of produce_async when publishing to a non-existent topic Real-world scenario: After commit hooks in Rails using produce_async to unknown topics Questions answered by this test: 1. Can produce_async raise immediately? SOMETIMES - depends on cached metadata - First attempt: May return handler (topic not yet in metadata cache) - Second attempt: May raise immediately (topic known to be invalid) 2. Do messages queued BEFORE the unknown topic continue? YES - they succeed 3. Do messages queued AFTER the unknown topic continue? YES - they succeed 4. When is the error discovered? Either immediately OR during handler.wait() Key insight: The behavior varies based on librdkafka's metadata cache state |
| `production/compression_different_no_conflict_spec.rb` | Kafka allows to use different default compression and per producer compression type It should be supported in Karafka |
| `production/dropped_message_error_handling_spec.rb` | WaterDrop should handle dropped messages gracefully when Kafka is unavailable This spec demonstrates how to track dropped messages using the labeling API and error.occurred notifications |
| `production/inflight_topic_removal_spec.rb` | Karafka should emit an inline error if topic that was used was suddenly removed In async, it should emit it via the error pipeline |
| `production/reject_no_key_message_when_compacted_spec.rb` | Kafka should reject a message on a compacted topic, when this message does not contain a key |
| `production/via_overwritten_producer_spec.rb` | User should be able to redefine the per consumer producer instance via direct `#producer` redefinition. It should be then fully usable |
| `production/waterdrop_connection_pool_pristine/connection_pool_spec.rb` | User should be able to use WaterDrop default connection pool from within consumers This allows for producer pooling and reuse across different consumers |
| `production/waterdrop_transactional_direct_pristine/transactional_pool_spec.rb` | User should be able to use WaterDrop connection pool with transactional direct api Each producer in the pool should have a unique transactional.id |
| `production/waterdrop_transactional_pool_pristine/transactional_pool_spec.rb` | User should be able to use WaterDrop connection pool with transactional producers Each producer in the pool should have a unique transactional.id |
| `production/with_key_and_partition_key_spec.rb` | When using both key and partition key to target partition, partition key should take precedence |
| `rails/rails72_pristine/just-a-dependency/rails_setup_spec.rb` | Karafka should work fine with Rails 7.2 even when it is just a transitive dependency and is not in active use. In case like this KARAFKA_BOOT_FILE needs to be set to "false" @see https://github.com/karafka/karafka/issues/813 Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/just-a-dependency/with_potential_name_collision_spec.rb` | Karafka should work fine when someone has root level components named based on the features @see https://github.com/karafka/karafka/issues/1144 Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/just-a-loaded-dependency/rails_setup_spec.rb` | Karafka should be default require Rails when `KARAFKA_REQUIRE_RAILS` is not set to `"false"` |
| `rails/rails72_pristine/just-a-not-loaded-dependency/rails_setup_spec.rb` | Karafka should work without Rails even when Rails is in the Gemfile as long as the KARAFKA_REQUIRE_RAILS is set to `"false"` |
| `rails/rails72_pristine/with-active-job-transactional/schedule_with_sqlite_spec.rb` | Karafka should work with Rails and AJ + #enqueue_after_transaction_commit |
| `rails/rails72_pristine/with-active_job/extended_routing_absence_spec.rb` | Karafka should injected extended ActiveJob routing when ActiveJob is available Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/with-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 7.2 using the default setup Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/with-active_job/without_double_logging_spec.rb` | Karafka with Rails should not log twice to stdout @see https://github.com/karafka/karafka/issues/1155 |
| `rails/rails72_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/with-active_job_and_current_attributes/resource_accesibility_and_isolation_spec.rb` | This spec validates, that while we run work in parallel in multiple threads, the context of `Current` never leaks in between them and that `Current` is always current to the local thread in which given consumer operates |
| `rails/rails72_pristine/with-active_job_and_perform_all_later/perform_all_later_run_spec.rb` | Karafka should work with Rails and AJ + #perform_all_later Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/with_kubernetes_listener_in_the_console/console_with_rails_spec.rb` | Karafka should work with Rails 7.2 and Karafka console should not crash Additionally when we use the Kubernetes Liveness Probing, it should not activate itself as Karafka does not start the liveness probing until it boots and it should NOT boot in the console |
| `rails/rails72_pristine/without-active_job/extended_routing_absence_spec.rb` | Karafka should not injected extended ActiveJob routing when ActiveJob is not available Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/without-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 7.2 that does not use ActiveJob using the default setup and should just ignore the ActiveJob components Load all the Railtie stuff like when `rails server` |
| `rails/rails72_pristine/without-active_job_rspec/console_with_rails_spec.rb` | Karafka should work with Rails 7.2 and Karafka console should not crash We should be able to create a small new Rails project and run the console and it should not crash |
| `rails/rails72_pristine/without-active_job_rspec/rspec_rails_setup_spec.rb` | Karafka should work with Rails 7.2 and rspec/rails when it is required and should not crash @see https://github.com/karafka/karafka/issues/803 Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/just-a-dependency/rails_setup_spec.rb` | Karafka should work fine with Rails 8 even when it is just a transitive dependency and is not in active use. In case like this KARAFKA_BOOT_FILE needs to be set to "false" @see https://github.com/karafka/karafka/issues/813 Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/just-a-dependency/with_potential_name_collision_spec.rb` | Karafka should work fine when someone has root level components named based on the features @see https://github.com/karafka/karafka/issues/1144 Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/just-a-loaded-dependency/rails_setup_spec.rb` | Karafka should be default require Rails when `KARAFKA_REQUIRE_RAILS` is not set to `"false"` |
| `rails/rails80_pristine/just-a-not-loaded-dependency/rails_setup_spec.rb` | Karafka should work without Rails even when Rails is in the Gemfile as long as the KARAFKA_REQUIRE_RAILS is set to `"false"` |
| `rails/rails80_pristine/with-active-job-transactional/schedule_with_sqlite_spec.rb` | Karafka should work with Rails and AJ + #enqueue_after_transaction_commit |
| `rails/rails80_pristine/with-active_job/extended_routing_absence_spec.rb` | Karafka should injected extended ActiveJob routing when ActiveJob is available Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/with-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 8 using the default setup Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/with-active_job/without_double_logging_spec.rb` | Karafka with Rails should not log twice to stdout @see https://github.com/karafka/karafka/issues/1155 |
| `rails/rails80_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/with-active_job_and_current_attributes/resource_accesibility_and_isolation_spec.rb` | This spec validates, that while we run work in parallel in multiple threads, the context of `Current` never leaks in between them and that `Current` is always current to the local thread in which given consumer operates |
| `rails/rails80_pristine/with-active_job_and_perform_all_later/perform_all_later_run_spec.rb` | Karafka should work with Rails and AJ + #perform_all_later Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/with_kubernetes_listener_in_the_console/console_with_rails_spec.rb` | Karafka should work with Rails 8 and Karafka console should not crash Additionally when we use the Kubernetes Liveness Probing, it should not activate itself as Karafka does not start the liveness probing until it boots and it should NOT boot in the console |
| `rails/rails80_pristine/with_setup_via_initializers/rails_console_spec.rb` | Karafka should work with Rails 8 and Karafka console should not crash when we put karafka setup in the initializers and skip the boot file |
| `rails/rails80_pristine/without-active_job/extended_routing_absence_spec.rb` | Karafka should not injected extended ActiveJob routing when ActiveJob is not available Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/without-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 8 that does not use ActiveJob using the default setup and should just ignore the ActiveJob components Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/without-active_job_rspec/console_with_rails_spec.rb` | Karafka should work with Rails 8 and Karafka console should not crash We should be able to create a small new Rails project and run the console and it should not crash |
| `rails/rails80_pristine/without-active_job_rspec/rspec_rails_setup_spec.rb` | Karafka should work with Rails 8 and rspec/rails when it is required and should not crash @see https://github.com/karafka/karafka/issues/803 Load all the Railtie stuff like when `rails server` |
| `rails/rails80_pristine/without_boot_file/rails_console_spec.rb` | Karafka should crash if we try to run CLI or Rails console when no boot file is defined and no initializer |
| `rails/rails81_pristine/just-a-dependency/rails_setup_spec.rb` | Karafka should work fine with Rails 8.1 even when it is just a transitive dependency and is not in active use. In case like this KARAFKA_BOOT_FILE needs to be set to "false" @see https://github.com/karafka/karafka/issues/813 Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/just-a-dependency/with_potential_name_collision_spec.rb` | Karafka should work fine when someone has root level components named based on the features @see https://github.com/karafka/karafka/issues/1144 Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/just-a-loaded-dependency/rails_setup_spec.rb` | Karafka should be default require Rails when `KARAFKA_REQUIRE_RAILS` is not set to `"false"` |
| `rails/rails81_pristine/just-a-not-loaded-dependency/rails_setup_spec.rb` | Karafka should work without Rails even when Rails is in the Gemfile as long as the KARAFKA_REQUIRE_RAILS is set to `"false"` |
| `rails/rails81_pristine/with-active-job-transactional/schedule_with_sqlite_spec.rb` | Karafka should work with Rails and AJ + #enqueue_after_transaction_commit |
| `rails/rails81_pristine/with-active_job/extended_routing_absence_spec.rb` | Karafka should injected extended ActiveJob routing when ActiveJob is available Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/with-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 8.1 using the default setup Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/with-active_job/without_double_logging_spec.rb` | Karafka with Rails should not log twice to stdout @see https://github.com/karafka/karafka/issues/1155 |
| `rails/rails81_pristine/with-active_job_and_current_attributes/active_job_run_spec.rb` | Karafka should work with Rails and AJ + Current Attributes Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/with-active_job_and_current_attributes/resource_accesibility_and_isolation_spec.rb` | This spec validates, that while we run work in parallel in multiple threads, the context of `Current` never leaks in between them and that `Current` is always current to the local thread in which given consumer operates |
| `rails/rails81_pristine/with-active_job_and_perform_all_later/perform_all_later_run_spec.rb` | Karafka should work with Rails and AJ + #perform_all_later Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/with-active_job_continuation/continuation_run_spec.rb` | Karafka should work with Rails 8.1 ActiveJob Continuation feature This tests that jobs can be interrupted and resumed using the continuation API Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/with_kubernetes_listener_in_the_console/console_with_rails_spec.rb` | Karafka should work with Rails 8.1 and Karafka console should not crash Additionally when we use the Kubernetes Liveness Probing, it should not activate itself as Karafka does not start the liveness probing until it boots and it should NOT boot in the console |
| `rails/rails81_pristine/with_setup_via_initializers/rails_console_spec.rb` | Karafka should work with Rails 8.1 and Karafka console should not crash when we put karafka setup in the initializers and skip the boot file |
| `rails/rails81_pristine/without-active_job/extended_routing_absence_spec.rb` | Karafka should not injected extended ActiveJob routing when ActiveJob is not available Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/without-active_job/railtie_setup_spec.rb` | Karafka should work with Rails 8.1 that does not use ActiveJob using the default setup and should just ignore the ActiveJob components Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/without-active_job_rspec/console_with_rails_spec.rb` | Karafka should work with Rails 8.1 and Karafka console should not crash We should be able to create a small new Rails project and run the console and it should not crash |
| `rails/rails81_pristine/without-active_job_rspec/rspec_rails_setup_spec.rb` | Karafka should work with Rails 8.1 and rspec/rails when it is required and should not crash @see https://github.com/karafka/karafka/issues/803 Load all the Railtie stuff like when `rails server` |
| `rails/rails81_pristine/without_boot_file/rails_console_spec.rb` | Karafka should crash if we try to run CLI or Rails console when no boot file is defined and no initializer |
| `rebalancing/consecutive_reclaim_flow_cooperative_spec.rb` | When consumer reclaims messages, it should not skip and should not have duplicated Marking should happen automatically Cooperative should not collide with this. |
| `rebalancing/consecutive_reclaim_flow_spec.rb` | When consumer reclaims messages, it should not skip and should not have duplicated Marking should happen automatically |
| `rebalancing/constant_rebalance_continuity_spec.rb` | When we consume data and several times we loose and regain partition, there should be continuity in what messages we pick up even if rebalances happens multiple times We may re-fetch certain messages but none should be skipped |
| `rebalancing/cooperative_sticky_full_spec.rb` | When marking as consumed during rebalance on cooperative-sticky, it will reject this marking and will work as expected (not revoking all). It was fixed in librdkafka 2.10 @note Prior to librdkafka 2.10 this is how it worked: Karafka will run a full rebalance in case we use cooperative-sticky but force commit when rebalance happens. This is how `librdkafka` works |
| `rebalancing/cooperative_sticky_immediate_stop_spec.rb` | Karafka should be able to stop even before the initial rebalance and should not crash @see https://github.com/confluentinc/librdkafka/issues/4312 We can get a forceful shutdown here when we decide to close before actually polling and triggering a rebalance We ignore it because here we check if the whole process does not crash due to librdkafka issue |
| `rebalancing/cooperative_sticky_spec.rb` | Karafka should support a cooperative-sticky rebalance strategy without any problems |
| `rebalancing/cooperative_sticky_unassigned_shutdown_spec.rb` | Karafka should shutdown cooperative-sticky after first rebalance, even if the rebalance did not grant any assignments to Karafka We have one partition that we assign to a different consumer and then start karafka that should not get the assignment. Despite that it should close gracefully because it did went through one (empty) rebalance. |
| `rebalancing/cooperative_sticky_without_assignment_spec.rb` | When work already happens somewhere else and cooperative-sticky kicks in, we should not get the assignment but an event should propagate from the rebalance nonetheless |
| `rebalancing/coordinator_replacement_after_rebalance_spec.rb` | Karafka should replace coordinator for consumer of a given topic partition after partition was taken away from us and assigned back |
| `rebalancing/exceeding_max_poll_interval_spec.rb` | When processing beyond the poll interval, we should be kicked out and we should loose the assigned partition. The revocation job should kick in with a proper revoked state. |
| `rebalancing/exceeding_max_poll_with_fast_commit_spec.rb` | When processing beyond the poll interval, with fast offset commit, we should pick up from where we left without duplicates |
| `rebalancing/exceeding_max_poll_with_late_commit_spec.rb` | When processing beyond the poll interval, with slower offset commit, we will restart processing and there should be duplicated messages. |
| `rebalancing/kip_848/configurable_remote_assignor_spec.rb` | Test that KIP-848 works with configurable remote assignor The new consumer protocol supports different server-side assignors like 'uniform' and 'range' |
| `rebalancing/kip_848/mixed_protocols_spec.rb` | Test that both old (eager/cooperative) and new (KIP-848) rebalance protocols can work simultaneously in different consumer groups within Karafka We'll set up multiple Karafka apps with different configurations to simulate different consumer groups with different protocols Consumer class for KIP-848 group |
| `rebalancing/kip_848/multiple_groups_spec.rb` | Test multiple consumer groups using the new KIP-848 consumer group protocol This ensures different consumer groups can consume the same topic independently |
| `rebalancing/kip_848/newly_joined_fencing_spec.rb` | Test rebalancing with KIP-848 by verifying that new consumer joining with same group instance id should be fenced If new consumer is not fenced, it will run forever and spec will timeout |
| `rebalancing/kip_848/rebalancing_spec.rb` | Test rebalancing with KIP-848 by verifying partition assignment changes when a second consumer joins the group |
| `rebalancing/kip_848/simple_spec.rb` | Test that KIP-848 consumer group protocol works with basic consumption |
| `rebalancing/kip_848/static_membership_spec.rb` | Test that static group membership works correctly with KIP-848 consumer protocol Static membership allows a consumer to maintain its partition assignment across restarts |
| `rebalancing/lifecycle_events_spec.rb` | When rebalance occurs, we should go through all the proper lifecycle events and only for the subscription group for which rebalance occurs. Details should be present and second group should be intact. |
| `rebalancing/lost_partition_revoked_execution_time_spec.rb` | When partition is lost but our job is still running, it should be allowed to finish work and should not run while code execution is still running. Revocation code should wait on the whole job to finish before running `#revoked`. @note This works differently in the Pro LRJ. |
| `rebalancing/message_order_with_sync_commit_spec.rb` | Messages should not be reprocessed out of order during a rebalance triggered by an unhealthy consumer with additional manual synchronous commits using #mark_as_consumed\!. |
| `rebalancing/message_order_without_sync_commit_spec.rb` | Messages can be reprocessed out of order during a rebalance triggered by an unhealthy consumer without additional manual synchronous commits. |
| `rebalancing/multiple_consumer_groups_stability_spec.rb` | When using multiple consumer groups and when one is rebalanced, it should not affect the one that was not a rebalance subject |
| `rebalancing/on_partition_data_revoked_spec.rb` | Karafka should trigger a revoked action when a partition is being taken from us Initially we should own all the partitions and then after they are taken away, we should get back to two (as the last one will be owned by the second consumer). |
| `rebalancing/revoke_continuity_spec.rb` | When Karafka looses a given partition but later gets it back, it should pick it up from the last offset committed without any problems |
| `rebalancing/revoke_reclaim_continuity_spec.rb` | When a consumer goes into a non-cooperative-sticky rebalance and gets the partitions back, it should not have duplicated data. |
| `rebalancing/stop_when_revoked_and_continue_spec.rb` | When given partition was revoked, we should be able to stop processing and it should not affect other consumers from consuming. On the other hand, when we get back what we have lost, we should be able to get going and we should not have to process what was processed by others. |
| `rebalancing/stored_reclaimed_continuity_cooperative_spec.rb` | When consumer looses the partition but later on gets it back, it should not have duplicates as long as marking always happened (which is the case in this scenario) There also should be no duplicates |
| `rebalancing/stored_reclaimed_continuity_spec.rb` | When consumer looses the partition but later on gets it back, it should not have duplicates as long as marking always happened (which is the case in this scenario) There also should be no duplicates |
| `rebalancing/voluntary_revocation_spec.rb` | When we are not kicked out forcefully but a legit rebalance occurs, we should not be marked as revoked when running tasks prior to reassignments. |
| `rebalancing/with_static_membership_reconnect_spec.rb` | Karafka process when stopped and started and configured with static membership should pick up the assigned work. It should not be reassigned to a different process. Karafka should maintain all the ordering and should not have duplicated. |
| `rebalancing/without_using_revoked_partition_data_spec.rb` | Karafka should not use data that was fetched partially for a partition that was lost in the same consumer instance that was revoked. There should be new instance created. New consumer instance should pick it up keeping continuity with the previous batch. When rebalance occurs and we're in the middle of data polling, data from a lost partition should be rejected as it is going to be picked up by a different process / instance. We simulate lost partition by starting a second consumer that will trigger a rebalance. |
| `routing/alternative_kafka_scope_defaults_population_spec.rb` | When we define alternative kafka cluster setup, it also should use same defaults |
| `routing/and_cli_validations_for_exclusions_spec.rb` | Karafka should fail when excluding non existing routing elements |
| `routing/and_cli_validations_for_inclusions_spec.rb` | Karafka should fail when defined routing is invalid Karafka should fail if we want to listen on a topic that was not defined |
| `routing/and_cli_validations_for_many_inclusions_spec.rb` | Karafka should correctly load resources separated by spaces or commas in the CLI usage |
| `routing/blockless_defaults_spec.rb` | We should be able to set up a shared setup for topics and not use any block configuration This can be used for example to define routing for a separate app that only manages the topics states as a migrator app |
| `routing/circular_dependencies_spec.rb` | Karafka should detect circular dependencies and duplicate definitions |
| `routing/cluster_info_based_discovery_spec.rb` | We should be able to build topics based on cluster info |
| `routing/conflicting_consumer_group_settings_spec.rb` | Karafka should detect conflicting configurations that cause namespace collisions |
| `routing/consumer_group_empty_topics_spec.rb` | Karafka should detect and reject consumer groups with no topics defined |
| `routing/consumer_group_reopening/comprehensive_reopening_spec.rb` | Comprehensive test covering multiple consumer group reopening scenarios: - Multiple named consumer groups - Implicit default group - Reopening multiple times - All in a single test to verify complex interactions |
| `routing/consumer_group_reopening/duplicate_topic_in_reopened_group_spec.rb` | Test what happens when the same topic name is defined twice when reopening a consumer group Expected: Karafka should raise InvalidConfigurationError because duplicate topic names within a consumer group are not allowed |
| `routing/consumer_group_reopening/explicit_reopening_spec.rb` | Karafka should allow reopening explicitly named consumer groups across multiple draw calls This test verifies that topics accumulate correctly when the same consumer group is defined multiple times |
| `routing/consumer_group_reopening/implicit_default_group_reopening_spec.rb` | Karafka should allow reopening the implicit default consumer group when using the simple topic style across multiple draw calls |
| `routing/consumer_group_reopening/mixed_explicit_implicit_reopening_spec.rb` | Test mixing explicit consumer group definitions with implicit default group usage This verifies that default group and named groups can coexist |
| `routing/consumer_group_reopening/multiple_redraws_stability_spec.rb` | This spec tests the behavior of subscription groups, member IDs, and positions when routes are redrawn multiple times in various scenarios: - Reopening consumer groups and adding topics - Adding new consumer groups - Empty draws (no-op) - Mixed scenarios with static group membership |
| `routing/consumer_group_reopening/multiple_subscription_groups_stability_spec.rb` | This spec tests the behavior when topics within the same consumer group have different settings and end up in different subscription groups. It verifies that positions remain stable across redraws even with multiple subscription groups per consumer group. |
| `routing/consumer_group_reopening/static_membership_stability_spec.rb` | When using static group membership with consumer group reopening, the subscription group positions and resulting group.instance.id values should remain stable This is critical for swarm mode where position affects the group.instance.id Ref: https://github.com/karafka/karafka/issues/2915 |
| `routing/consumer_group_reopening/subscription_groups_rebuild_spec.rb` | Karafka should rebuild subscription groups when consumer groups are reopened This ensures that topics added in subsequent draw calls are included in subscription groups Ref: https://github.com/karafka/karafka/issues/2915 |
| `routing/consumer_group_reopening/swarm_static_membership_spec.rb` | When using static group membership in swarm mode with consumer group reopening, the group.instance.id values should remain stable across multiple draw calls. This ensures no fencing issues occur when consumer groups are reopened. Ref: https://github.com/karafka/karafka/issues/2915 |
| `routing/consumer_group_reopening_pristine/multi_file_spec.rb` | This pristine spec simulates the real-world scenario from GitHub issue #2363 where routing configuration is split across multiple files, each defining topics in the same consumer group. Directory structure: - routes/users_routes.rb  - defines topics in a shared consumer group - routes/orders_routes.rb - reopens the same consumer group with more topics - consumers/              - consumer implementations Expected behavior: All topics should accumulate in the same consumer group, allowing for modular routing configuration. |
| `routing/different_cluster_consumer_groups_spec.rb` | Karafka should allow for defining separate consumer groups that will talk with separate clusters Distribution in subscription groups should match same cluster with same settings |
| `routing/dynamic_topic_creation_spec.rb` | Karafka should handle dynamic routing patterns and topic configuration edge cases |
| `routing/eofed/with_eof_disabled_and_kafka_eof_spec.rb` | Karafka should allow to use kafka scope eof without eofed enabled. |
| `routing/eofed/with_eof_in_different_scope_spec.rb` | Karafka should not allow to use eofed when kafka scope setup is with eof in different sub-scope |
| `routing/eofed/with_eof_in_same_scope_spec.rb` | Karafka should allow to use eofed when kafka scope setup is with eof in same sub-scope |
| `routing/eofed/with_eof_spec.rb` | Karafka should allow to use eofed when kafka scope setup is with eof |
| `routing/eofed/without_eof_spec.rb` | Karafka should not allow to use eofed when kafka scope setup is not with eof |
| `routing/invalid_regex_patterns_spec.rb` | Karafka should handle invalid topic names and patterns |
| `routing/limited_scope/with_inactive_and_disabled_spec.rb` | When combination of cli disabled topics and routing disabled topics meet, we should error. |
| `routing/limited_scope/with_mix_but_inactive_without_consumer_spec.rb` | When we have inactive topics, they should be ok without consumer defined |
| `routing/limited_scope/with_non_existing_consumer_groups_excluded_spec.rb` | When trying to exclude non existing consumer group, we should fail. |
| `routing/limited_scope/with_non_existing_consumer_groups_spec.rb` | When trying to run non existing consumer group, we should fail. |
| `routing/limited_scope/with_non_existing_subscription_groups_spec.rb` | When trying to run non existing subscription group, we should fail. |
| `routing/limited_scope/with_non_existing_topics_spec.rb` | When trying to run non existing topics, we should fail. |
| `routing/limited_scope/with_nothing_to_run_spec.rb` | When combination of consumer groups, subscription groups and topics we want to run is such, that they do not exist all together, we need to raise an error. |
| `routing/limited_scope/with_only_inactive_topics_spec.rb` | When all our topics are disabled in routing, we should not allow Karafka to run |
| `routing/overlapping_consumer_patterns_spec.rb` | Karafka should handle complex consumer patterns and prevent conflicts |
| `routing/pausing/with_global_config_backwards_compatibility_spec.rb` | Verify that both old and new global pause configuration APIs work and are bidirectionally compatible |
| `routing/pausing/with_global_config_new_api_spec.rb` | Verify that the new global pause configuration API works correctly |
| `routing/pausing/with_global_config_old_api_spec.rb` | Verify that the old global pause configuration API (setters) still works correctly This is a backwards compatibility test |
| `routing/runtime_routing_expansion_spec.rb` | When Karafka runs, we should be able to inject new routes and they should be picked It does NOT mean they will be used (not yet) but it should mean, that valid once are accepted and invalid are validated and an error is raised. This spec ensures, that dynamic routing expansions in runtime are subject to validation |
| `routing/special_character_topics_spec.rb` | Karafka should handle topics with special characters properly |
| `routing/stable_with_limited_consumer_groups_excluded_spec.rb` | When building subscription groups and then using a limited subset of consumer groups simulating the --exclude_consumer_groups flag should not impact the numbers in the group |
| `routing/stable_with_limited_consumer_groups_spec.rb` | When building subscription groups and then using a limited subset of consumer groups simulating the --consumer_groups flag should not impact the numbers in the group |
| `routing/stable_with_subscription_groups_spec.rb` | Building subscription groups multiple times should not change their static group membership ids as long as topics structure is unchanged. |
| `routing/topic_custom_attributes_spec.rb` | You should be able to define your own features to assign them to topics You should also be able to use it from the consumer |
| `routing/topic_pattern_matching_spec.rb` | Karafka should handle topic routing patterns and naming edge cases |
| `routing/valid_with_features_usage_spec.rb` | Karafka should auto-load all the routing features |
| `routing/with_active_false_overwritten_spec.rb` | When by default all topics are not active, we should be able to explicitely set it to active |
| `routing/with_active_true_overwritten_spec.rb` | When by default all topics are active, we should be able to explicitely set it to inactive |
| `routing/with_altered_routing_flow_spec.rb` | @see https://github.com/karafka/karafka/issues/2344 @see https://github.com/flipp-oss/deimos @see https://github.com/flipp-oss/deimos/blob/fc89c645/lib/deimos/ext/routing_defaults.rb This is to ensure that Deimos routing patches work as expected |
| `routing/with_consumer_group_reopening_spec.rb` | Karafka should allow reopening consumer groups across multiple draw calls This enables splitting routing configuration across multiple files |
| `routing/with_defaults_configured_spec.rb` | Karafka should use the defaults if they were configured but only if the appropriate config setup was not executed |
| `routing/with_incomplete_kafka_reconfiguration_spec.rb` | When reconfiguring kafka scope without providing bootstrap.servers, we should fail |
| `routing/with_kafka_scope_inherit_empty_hash_spec.rb` | Test that kafka scope settings with inherit: true properly merge defaults even when empty hash is provided as config. This addresses the edge case where calling kafka(**{}) with inherit: true should still preserve the defaults from Karafka::App.config.kafka instead of overriding them. |
| `routing/with_nameless_subscription_groups_spec.rb` | Karafka should allow for nameless subscription groups inside of consumer groups |
| `routing/with_namespace_collisions_spec.rb` | Karafka should now allow for topics that would have metrics namespace collisions It should not be allowed within same consumer group |
| `routing/with_namespace_collisions_without_strict_spec.rb` | Karafka should allow for topics that would have metrics namespace collisions if strict topic names validation is off |
| `routing/with_per_topic_consumer_persistence_spec.rb` | We should be able to disable consumer persistence for all the topics by default but at the same time we should be able to customize things in a per topic basis. This can be useful when injecting extensions/plugins that would use routing. For example our Web-UI despite dev not having persistence needs to have persistence in order to be able to materialize and compute the states correctly |
| `routing/with_same_topic_in_multiple_subscription_groups_spec.rb` | Karafka should not allow for multiple subscriptions to the same topic in the same consumer group even if subscription group names are different |
| `routing/with_subscription_groups_block_based_spec.rb` | Karafka should allow for nice subscription groups management style with nesting DSL |
| `routing/with_subscription_groups_chaotic_order_spec.rb` | Karafka should support chaotic order group definitions and should correctly assign the subscription groups |
| `routing/with_symbol_based_subscription_group_spec.rb` | Karafka should allow for nice subscription groups management style with symbols. |
| `routing/with_the_simple_routes_style_spec.rb` | Karafka should allow to use the '.routes' alias for routing builder and it should support a case where there is a single consumer group with multiple topics |
| `routing/with_topic_inherited_settings_spec.rb` | We should be able to define defaults and then use the inherit flag to get per topic Kafka scope reconfiguration |
| `seek/back_to_expired_spec.rb` | When moving back to expired it should seek to -1 which is latest (upcoming) This is expected as all expired mean there is nothing except high watermark (-1/latest) |
| `seek/back_with_reset_and_mark_again_spec.rb` | We should be able to seek back and reset the offset + mark as consumed previous offset and it should have been stored |
| `seek/back_without_reset_and_mark_again_spec.rb` | We should be able to seek back and without reset it should not store the seeked offset |
| `seek/backward_from_consumer_spec.rb` | This spec aims to test seeking process. We use seek to process first message out of all and then we move backwards till 0 |
| `seek/beyond_high_watermark_offset_spec.rb` | Seeking beyond high watermark should trigger reset back to earliest |
| `seek/in_a_loop_higher_partition_spec.rb` | Karafka should be able to consume same messages all the time in an endless loop This can be useful when doing development with a fixed data-set in kafka over which we don't have control (that is, cannot be changed) |
| `seek/in_a_loop_spec.rb` | Karafka should be able to consume same messages all the time in an endless loop This can be useful when doing development with a fixed data-set in kafka over which we don't have control (that is, cannot be changed) |
| `seek/seek_on_many_partitions_with_reset_spec.rb` | When we have a lot of partitions and we seek back with offset reset, it should correctly allow us to mark offsets per partition |
| `seek/to_a_given_offset_from_consumer_on_higher_partition_spec.rb` | Karafka should be able to easily consume from a given offset, instead of starting from 0. Karafka makes sure, we do not process in parallel incoming data from the same partition ahead, so we can easily rewind the offset and then the client queue will be cleared. What that means is that we do not have to worry beyond our current batch. It is worth keeping in mind, that in order for this to work, the current consumption process need to happen from the "previous" offset before seeking. Otherwise consumer instance won't be initialized |
| `seek/to_a_given_offset_from_consumer_spec.rb` | Karafka should be able to easily consume from a given offset, instead of starting from 0. Karafka makes sure, we do not process in parallel incoming data from the same partition ahead, so we can easily rewind the offset and then the client queue will be cleared. What that means is that we do not have to worry beyond our current batch. It is worth keeping in mind, that in order for this to work, the current consumption process need to happen from the "previous" offset before seeking. Otherwise consumer instance won't be initialized |
| `seek/to_earliest_spec.rb` | If we seek to the earliest, we should start from the beginning |
| `seek/to_high_watermark_offset_spec.rb` | Seeking to the high watermark offset should not reset the consumer offset position |
| `seek/to_latest_spec.rb` | If we seek to the latest, we should start from the high watermark |
| `seek/to_non_existing_future_on_higher_partition_spec.rb` | When we seek to a future where there are no offsets, we should seek to the first not yet produced message and get it |
| `seek/to_non_existing_future_spec.rb` | When we seek to a future where there are no offsets, we should seek to the first not yet produced message and get it |
| `seek/to_time_before_start_on_higher_partition_spec.rb` | If we seek to a long ago, before even topic was created, we should start from beginning on other partitions than 0 as well |
| `seek/to_time_before_start_spec.rb` | If we seek to a long ago, before even topic was created, we should start from beginning |
| `seek/with_cleaned_error_count_post_seek_spec.rb` | Karafka should reset the error count after seeking as it is a non-error flow |
| `seek/with_time_spec.rb` | Karafka should be able to easily consume from a given offset based on the time of the event and not the offset numeric value. |
| `setup/admin_invalid_configuration_spec.rb` | When we try to use invalid admin configuration, it should fail |
| `setup/attributes_distribution_spec.rb` | When defining in Karafka settings for both producer and consumer, there should be no warnings raised by librdkafka as no producer settings should go to consumer and the other way around. |
| `setup/config_validation_spec.rb` | Karafka should use rdkafka errors raised when invalid sub-settings after routing is defined We do it after the routing, so any overwrites are also handled |
| `setup/config_with_inherited_client_id_spec.rb` | Karafka should not use client_id for kafka.client.id if kafka.client.id is not provided, as it is injected when we define routes |
| `setup/config_with_overwritten_client_id_spec.rb` | Karafka should not use client_id for kafka.client.id if kafka.client.id is provided |
| `setup/debug_mode_spec.rb` | Karafka should switch all components int a debug mode when we use explicit debug\! |
| `setup/load_poro/configure_spec.rb` | Karafka in a PORO project should load without any problems when regular (not pro) |
| `setup/max_messages_format_as_string_spec.rb` | When max messages format is not as expected, we should fail |
| `setup/producer_block_configuration_spec.rb` | Karafka should support configuring the producer via a block during setup. This allows users to customize producer settings without manually creating a producer instance. |
| `setup/producer_block_execution_order_spec.rb` | Karafka should execute the producer configuration block AFTER all user settings are applied and AFTER the default producer is created. This ensures the block has access to the fully configured producer and can override any defaults. Karafka should also setup producer after ALL Karafka level configuration is available |
| `setup/producer_block_with_middleware_spec.rb` | Karafka should allow adding middleware to the producer via the configuration block |
| `setup/producer_without_block_spec.rb` | Karafka should work correctly when no producer configuration block is provided. The default producer should be created with standard settings. |
| `setup/re_initialization_of_cached_resources_spec.rb` | Karafka should update the cached references to the monitor, logger and producer once those are altered during the configuration |
| `setup/with_custom_worker_thread_priority_spec.rb` | When reconfiguring the worker thread priority, worker threads should have proper priority set When reconfiguring internal listener worker thread priority, it should also work |
| `setup/with_kip_848_protocol_incorrect_config_spec.rb` | Karafka should crash when new and old rebalance protocol settings are mixed even if independently they are correct. Note that those errors only happen upon subscription, not during configuration. |
| `setup/with_patched_object_spec.rb` | Karafka setup should not crash even when Object class has been patched with logger methods that raise errors |
| `shutdown/consumer_shutdown_during_processing_spec.rb` | Karafka should call shutdown hooks during server termination |
| `shutdown/extremely_fast_stop_spec.rb` | When we stop right after run, it should not hang even on extreme edge cases |
| `shutdown/fast_shutdown_on_long_polling_spec.rb` | Karafka should not wait full long polling cycle when shutdown is issued due to fast circuit breaker flow. No assertions are needed as it will just wait forever if not working correctly |
| `shutdown/on_hanging_jobs_and_a_shutdown_spec.rb` | When Karafka is being shutdown and the consumer is hanging, it should force a shutdown |
| `shutdown/on_hanging_listener_and_shutdown_spec.rb` | When Karafka is being shutdown and the consumer thread is hanging for too long, it should force a shutdown despite having active connections to Kafka |
| `shutdown/on_hanging_on_shutdown_job_and_a_shutdown_spec.rb` | When Karafka is being shutdown and the consumer is hanging, it should force a shutdown |
| `shutdown/on_quiet_with_work_finalization_spec.rb` | When Karafka is being moved to quiet mode, it is requested to first reach quieting and then stop on the quiet state. |
| `shutdown/on_reached_quiet_spec.rb` | When we have reached quiet state, we should still be subscribed to what we had |
| `shutdown/on_shutdown_when_messages_received_spec.rb` | When we received messages, shutdown should kick in for every consumer that did any work |
| `shutdown/on_shutdown_when_no_messages_received_spec.rb` | When we received no messages, shutdown should not happen |
| `shutdown/on_shutdown_when_partition_in_error_pause_spec.rb` | When partition was paused due to an error and this pause is still lasting, on shutdown the `#shutdown` method still should be invoked |
| `shutdown/shutdown_jobs_exceeding_max_poll_interval_spec.rb` | When Karafka is configured with short 'max.poll.interval.ms', shorter than a shutdown job, it should not matter. Shutdown jobs should not be terminated unless they exceed `shutdown_timeout` |
| `shutdown/shutdown_jobs_threads_processing_spec.rb` | Karafka shutdown jobs should run in workers threads, not form the fetcher thread |
| `shutdown/via_idle_timeout_listener_spec.rb` | We should be able to build a listener that will monitor how long karafka is running without receiving any new messages and that it will stop after the expected time by sending the stop signal |
| `signals/on_sigint_spec.rb` | When Karafka receives sigint, it should stop |
| `signals/on_sigquit_spec.rb` | When Karafka receives sigquit, it should stop |
| `signals/on_sigterm_spec.rb` | When Karafka receives sigterm, it should stop |
| `signals/on_sigtstp_spec.rb` | When Karafka receives sigstp it should finish all the work and keep polling not to trigger rebalances, however the shutdown hooks should happen and new work should not be picked up |
| `signals/on_sigttin_spec.rb` | When Karafka receives sigttin, it should print some backtrace details using logger when logger listener is enabled Here it is enabled by default |
| `specs_naming_spec.rb` | This spec iterates over all the files in the `spec/integrations` and makes sure, that all specs containing specs end with `_spec.rb` to make sure they are executed. It happened to me few times already, that I would forget about the postfix and the spec would not run. |
| `swarm/cli_validations_for_inclusions_spec.rb` | Karafka should fail if we specify a consumer group that was not defined also when working in the swarm mode. |
| `swarm/dead_node_cleanup_spec.rb` | After node dies (for any reason), it should be cleaned up and not left hanging as a zombie |
| `swarm/dead_node_recovery_spec.rb` | After node dies (for any reason), it should be restarted by the supervisor |
| `swarm/forceful_nodes_restart_graceful_supervisor_stop_spec.rb` | When supervisor restarts nodes that are hanging it should emit a status and when nodes are no longer hanging it should gracefully stop macOS ARM64 needs more generous timeouts due to slower process startup |
| `swarm/forceful_shutdown_of_hanging_spec.rb` | When supervisor stops work, hanging processes should be killed |
| `swarm/liveness_failure_restart_spec.rb` | If we use liveness API to report issue, Karafka should restart the node |
| `swarm/regular_flow_with_post_fork_producer_spec.rb` | We should be able to use the regular producer with the inherited config to operate and send messages |
| `swarm/start_stop_after_time_spec.rb` | We should be able to create a swarm where one of the processes gets assignment and consumes messages. After this we should be able to stop everything. |
| `swarm/static_memberships_without_a_conflict_spec.rb` | We should be able to create a swarm with static members that can go up and down and receive same assignments. They also should not conflict with each other meaning each of them should consume some data and there should be no fencing |
| `swarm/with_blocking_at_exit_spec.rb` | When supervisor stops work but do not stop because of blocking `at_exit` in them, supervisor should kill them |
| `swarm/with_different_producer_payload_sizes_spec.rb` | When creating a producer in forks, forks should inherit all producer settings and not only the once coming from th kafka scope The id should NOT be inherited not to create confusion. |
| `testing/minitest_pristine/test_flow_with_marking_spec.rb` | Karafka when used with minitest should work as expected when using a client reference Covers this case: https://github.com/karafka/karafka-testing/pull/198 |
| `testing/rspec/dynamic_sg_dispatch_spec.rb` | We should be able to use testing lib with rspec for topics that belong to one of few SGs from a single CG |
| `testing/rspec/simple_integration_spec.rb` | We should be able to use testing lib with rspec in a simple manner |
| `testing/rspec_pristine/test_flow_with_marking_spec.rb` | Karafka when used with minitest should work as expected when using a client reference Covers this case: https://github.com/karafka/karafka-testing/pull/198 |
| `web/end_to_end_install_flow_pristine/install_with_rails_spec.rb` | Karafka 2.4+ should work ok with 0.9.0+ |
| `web/from_earliest_drifted_spec.rb` | Karafka should be able to handle a case where the cluster and consumer times drifted Some metrics may not be 100% accurate and this should not happen often but may |
| `web/from_earliest_spec.rb` | Karafka should be able to consume and web tracking should not interfere |
| `web/incompatible_version_pristine_0.7/console_with_rails_spec.rb` | Karafka 2.4.0+ should not work with Web UI version 0.9.0 lower than 0.9.0 |
| `web/with_errors_spec.rb` | Karafka should be able to recover from non-critical errors and web tracking instrumentation should not break anything and should not crash |
| `web/with_request_without_rails_pristine/install_and_run_spec.rb` | Karafka 2.4+ should work ok with 0.9.0+ |
| `web/without_web_ui_topics_spec.rb` | Karafka should be able to consume and web tracking should not interfere even when web ui topics are not created |


---

*Last modified: 2026-02-18 13:43:33*