Librdkafka Errors

Handling Non-Permanent and Permanent Errors from librdkafka

Most errors from librdkafka are non-permanent and are gracefully managed by internal retries, reducing the immediate impact on your application. These errors are generally not critical and often involve temporary issues such as a timed_out error, which indicates a delay in response or operation but does not necessarily signify a permanent problem. Those types of errors, while reported, usually resolve themselves through subsequent retries without requiring direct intervention.

Karafka also incorporates several recovery mechanisms that enable it to automatically reconnect and continue operations whenever possible, further enhancing the robustness of your message-handling processes.

Errors Classification

Internal errors (RD_KAFKA_RESP_ERR__) in librdkafka originate within the client library. They are often labeled as "Local" issues. In contrast, the Kafka broker generates broker errors (RD_KAFKA_RESP_ERR_). They are typically referred to as "Broker" issues, indicating problems on the server side.

Understanding whether an error is internal or from a broker is crucial for troubleshooting and resolving issues effectively:

• Debugging: Knowing the error's source helps debug. If the error is internal, the issue must be resolved within the client environment (e.g., fixing configuration and updating the client library for bugs). If the error is from the broker, it might involve server configuration, network policies, or other aspects managed by the Kafka infrastructure.

• Scalability and Reliability: By properly handling these errors, developers and operators can ensure the creation of more robust and scalable client applications. For instance, internal errors might require code changes or updates, while broker errors could lead to changes in how the client interacts with the Kafka ecosystem.

• Operational Awareness: For operators and developers, the categorization of errors provides valuable insights into the health and status of both the Kafka client and the server, fostering informed operational decisions and monitoring strategies.

By categorizing errors this way, librdkafka provides a more straightforward interface for developers to handle, log, and react to various conditions affecting their applications' interaction with Apache Kafka.

Local Errors

Internal errors originate within the librdkafka client itself. They are not generated by interactions with the Kafka broker. Still, they are caused by issues within the client's processes and operations.

These errors are prefixed with two underscores (__) in their full error code names.

Code	Symbol	Name	Description
-199	:bad_msg	RD_KAFKA_RESP_ERR__BAD_MSG	Local: Bad message format
-198	:bad_compression	RD_KAFKA_RESP_ERR__BAD_COMPRESSION	Local: Invalid compressed data
-197	:destroy	RD_KAFKA_RESP_ERR__DESTROY	Local: Broker handle destroyed
-196	:fail	RD_KAFKA_RESP_ERR__FAIL	Local: Communication failure with broker
-195	:transport	RD_KAFKA_RESP_ERR__TRANSPORT	Local: Broker transport failure
-194	:crit_sys_resource	RD_KAFKA_RESP_ERR__CRIT_SYS_RESOURCE	Local: Critical system resource failure
-193	:resolve	RD_KAFKA_RESP_ERR__RESOLVE	Local: Host resolution failure
-192	:msg_timed_out	RD_KAFKA_RESP_ERR__MSG_TIMED_OUT	Local: Message timed out
-191	:partition_eof	RD_KAFKA_RESP_ERR__PARTITION_EOF	Broker: No more messages
-190	:unknown_partition	RD_KAFKA_RESP_ERR__UNKNOWN_PARTITION	Local: Unknown partition
-189	:fs	RD_KAFKA_RESP_ERR__FS	Local: File or filesystem error
-188	:unknown_topic	RD_KAFKA_RESP_ERR__UNKNOWN_TOPIC	Local: Unknown topic
-187	:all_brokers_down	RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN	Local: All broker connections are down
-186	:invalid_arg	RD_KAFKA_RESP_ERR__INVALID_ARG	Local: Invalid argument or configuration
-185	:timed_out	RD_KAFKA_RESP_ERR__TIMED_OUT	Local: Timed out
-184	:queue_full	RD_KAFKA_RESP_ERR__QUEUE_FULL	Local: Queue full
-183	:isr_insuff	RD_KAFKA_RESP_ERR__ISR_INSUFF	Local: ISR count insufficient
-182	:node_update	RD_KAFKA_RESP_ERR__NODE_UPDATE	Local: Broker node update
-181	:ssl	RD_KAFKA_RESP_ERR__SSL	Local: SSL error
-180	:wait_coord	RD_KAFKA_RESP_ERR__WAIT_COORD	Local: Waiting for coordinator
-179	:unknown_group	RD_KAFKA_RESP_ERR__UNKNOWN_GROUP	Local: Unknown group
-178	:in_progress	RD_KAFKA_RESP_ERR__IN_PROGRESS	Local: Operation in progress
-177	:prev_in_progress	RD_KAFKA_RESP_ERR__PREV_IN_PROGRESS	Local: Previous operation in progress
-176	:existing_subscription	RD_KAFKA_RESP_ERR__EXISTING_SUBSCRIPTION	Local: Existing subscription
-175	:assign_partitions	RD_KAFKA_RESP_ERR__ASSIGN_PARTITIONS	Local: Assign partitions
-174	:revoke_partitions	RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS	Local: Revoke partitions
-173	:conflict	RD_KAFKA_RESP_ERR__CONFLICT	Local: Conflicting use
-172	:state	RD_KAFKA_RESP_ERR__STATE	Local: Erroneous state
-171	:unknown_protocol	RD_KAFKA_RESP_ERR__UNKNOWN_PROTOCOL	Local: Unknown protocol
-170	:not_implemented	RD_KAFKA_RESP_ERR__NOT_IMPLEMENTED	Local: Not implemented
-169	:authentication	RD_KAFKA_RESP_ERR__AUTHENTICATION	Local: Authentication failure
-168	:no_offset	RD_KAFKA_RESP_ERR__NO_OFFSET	Local: No offset stored
-167	:outdated	RD_KAFKA_RESP_ERR__OUTDATED	Local: Outdated
-166	:timed_out_queue	RD_KAFKA_RESP_ERR__TIMED_OUT_QUEUE	Local: Timed out in queue
-165	:unsupported_feature	RD_KAFKA_RESP_ERR__UNSUPPORTED_FEATURE	Local: Required feature not supported by broker
-164	:wait_cache	RD_KAFKA_RESP_ERR__WAIT_CACHE	Local: Awaiting cache update
-163	:intr	RD_KAFKA_RESP_ERR__INTR	Local: Operation interrupted
-162	:key_serialization	RD_KAFKA_RESP_ERR__KEY_SERIALIZATION	Local: Key serialization error
-161	:value_serialization	RD_KAFKA_RESP_ERR__VALUE_SERIALIZATION	Local: Value serialization error
-160	:key_deserialization	RD_KAFKA_RESP_ERR__KEY_DESERIALIZATION	Local: Key deserialization error
-159	:value_deserialization	RD_KAFKA_RESP_ERR__VALUE_DESERIALIZATION	Local: Value deserialization error
-158	:partial	RD_KAFKA_RESP_ERR__PARTIAL	Local: Partial response
-157	:read_only	RD_KAFKA_RESP_ERR__READ_ONLY	Local: Read-only object
-156	:noent	RD_KAFKA_RESP_ERR__NOENT	Local: No such entry
-155	:underflow	RD_KAFKA_RESP_ERR__UNDERFLOW	Local: Read underflow
-154	:invalid_type	RD_KAFKA_RESP_ERR__INVALID_TYPE	Local: Invalid type
-153	:retry	RD_KAFKA_RESP_ERR__RETRY	Local: Retry operation
-152	:purge_queue	RD_KAFKA_RESP_ERR__PURGE_QUEUE	Local: Purged in queue
-151	:purge_inflight	RD_KAFKA_RESP_ERR__PURGE_INFLIGHT	Local: Purged in flight
-150	:fatal	RD_KAFKA_RESP_ERR__FATAL	Local: Fatal error
-149	:inconsistent	RD_KAFKA_RESP_ERR__INCONSISTENT	Local: Inconsistent state
-148	:gapless_guarantee	RD_KAFKA_RESP_ERR__GAPLESS_GUARANTEE	Local: Gap-less ordering would not be guaranteed if proceeding
-147	:max_poll_exceeded	RD_KAFKA_RESP_ERR__MAX_POLL_EXCEEDED	Local: Maximum application poll interval (max.poll.interval.ms) exceeded
-146	:unknown_broker	RD_KAFKA_RESP_ERR__UNKNOWN_BROKER	Local: Unknown broker
-145	:not_configured	RD_KAFKA_RESP_ERR__NOT_CONFIGURED	Local: Functionality not configured
-144	:fenced	RD_KAFKA_RESP_ERR__FENCED	Local: This instance has been fenced by a newer instance
-143	:application	RD_KAFKA_RESP_ERR__APPLICATION	Local: Application generated error
-142	:assignment_lost	RD_KAFKA_RESP_ERR__ASSIGNMENT_LOST	Local: Group partition assignment lost
-141	:noop	RD_KAFKA_RESP_ERR__NOOP	Local: No operation performed
-140	:auto_offset_reset	RD_KAFKA_RESP_ERR__AUTO_OFFSET_RESET	Local: No offset to automatically reset to
-139	:log_truncation	RD_KAFKA_RESP_ERR__LOG_TRUNCATION	Local: Partition log truncation detected

Broker Errors

Broker errors are those reported by the Kafka broker to the librdkafka client. These errors arise from the server side of Kafka operations and are relayed back to the client during normal communication.

These errors are prefixed with a single underscore (_) in their error code names.

Code	Symbol	Name	Description
1	:offset_out_of_range	RD_KAFKA_RESP_ERR_OFFSET_OUT_OF_RANGE	Broker: Offset out of range
2	:invalid_msg	RD_KAFKA_RESP_ERR_INVALID_MSG	Broker: Invalid message
3	:unknown_topic_or_part	RD_KAFKA_RESP_ERR_UNKNOWN_TOPIC_OR_PART	Broker: Unknown topic or partition
4	:invalid_msg_size	RD_KAFKA_RESP_ERR_INVALID_MSG_SIZE	Broker: Invalid message size
5	:leader_not_available	RD_KAFKA_RESP_ERR_LEADER_NOT_AVAILABLE	Broker: Leader not available
6	:not_leader_for_partition	RD_KAFKA_RESP_ERR_NOT_LEADER_FOR_PARTITION	Broker: Not leader for partition
7	:request_timed_out	RD_KAFKA_RESP_ERR_REQUEST_TIMED_OUT	Broker: Request timed out
8	:broker_not_available	RD_KAFKA_RESP_ERR_BROKER_NOT_AVAILABLE	Broker: Broker not available
9	:replica_not_available	RD_KAFKA_RESP_ERR_REPLICA_NOT_AVAILABLE	Broker: Replica not available
10	:msg_size_too_large	RD_KAFKA_RESP_ERR_MSG_SIZE_TOO_LARGE	Broker: Message size too large
11	:stale_ctrl_epoch	RD_KAFKA_RESP_ERR_STALE_CTRL_EPOCH	Broker: StaleControllerEpochCode
12	:offset_metadata_too_large	RD_KAFKA_RESP_ERR_OFFSET_METADATA_TOO_LARGE	Broker: Offset metadata string too large
13	:network_exception	RD_KAFKA_RESP_ERR_NETWORK_EXCEPTION	Broker: Broker disconnected before response received
14	:coordinator_load_in_progress	RD_KAFKA_RESP_ERR_COORDINATOR_LOAD_IN_PROGRESS	Broker: Coordinator load in progress
15	:coordinator_not_available	RD_KAFKA_RESP_ERR_COORDINATOR_NOT_AVAILABLE	Broker: Coordinator not available
16	:not_coordinator	RD_KAFKA_RESP_ERR_NOT_COORDINATOR	Broker: Not coordinator
17	:topic_exception	RD_KAFKA_RESP_ERR_TOPIC_EXCEPTION	Broker: Invalid topic
18	:record_list_too_large	RD_KAFKA_RESP_ERR_RECORD_LIST_TOO_LARGE	Broker: Message batch larger than configured server segment size
19	:not_enough_replicas	RD_KAFKA_RESP_ERR_NOT_ENOUGH_REPLICAS	Broker: Not enough in-sync replicas
20	:not_enough_replicas_after_append	RD_KAFKA_RESP_ERR_NOT_ENOUGH_REPLICAS_AFTER_APPEND	Broker: Message(s) written to insufficient number of in-sync replicas
21	:invalid_required_acks	RD_KAFKA_RESP_ERR_INVALID_REQUIRED_ACKS	Broker: Invalid required acks value
22	:illegal_generation	RD_KAFKA_RESP_ERR_ILLEGAL_GENERATION	Broker: Specified group generation id is not valid
23	:inconsistent_group_protocol	RD_KAFKA_RESP_ERR_INCONSISTENT_GROUP_PROTOCOL	Broker: Inconsistent group protocol
24	:invalid_group_id	RD_KAFKA_RESP_ERR_INVALID_GROUP_ID	Broker: Invalid group.id
25	:unknown_member_id	RD_KAFKA_RESP_ERR_UNKNOWN_MEMBER_ID	Broker: Unknown member
26	:invalid_session_timeout	RD_KAFKA_RESP_ERR_INVALID_SESSION_TIMEOUT	Broker: Invalid session timeout
27	:rebalance_in_progress	RD_KAFKA_RESP_ERR_REBALANCE_IN_PROGRESS	Broker: Group rebalance in progress
28	:invalid_commit_offset_size	RD_KAFKA_RESP_ERR_INVALID_COMMIT_OFFSET_SIZE	Broker: Commit offset data size is not valid
29	:topic_authorization_failed	RD_KAFKA_RESP_ERR_TOPIC_AUTHORIZATION_FAILED	Broker: Topic authorization failed
30	:group_authorization_failed	RD_KAFKA_RESP_ERR_GROUP_AUTHORIZATION_FAILED	Broker: Group authorization failed
31	:cluster_authorization_failed	RD_KAFKA_RESP_ERR_CLUSTER_AUTHORIZATION_FAILED	Broker: Cluster authorization failed
32	:invalid_timestamp	RD_KAFKA_RESP_ERR_INVALID_TIMESTAMP	Broker: Invalid timestamp
33	:unsupported_sasl_mechanism	RD_KAFKA_RESP_ERR_UNSUPPORTED_SASL_MECHANISM	Broker: Unsupported SASL mechanism
34	:illegal_sasl_state	RD_KAFKA_RESP_ERR_ILLEGAL_SASL_STATE	Broker: Request not valid in current SASL state
35	:unsupported_version	RD_KAFKA_RESP_ERR_UNSUPPORTED_VERSION	Broker: API version not supported
36	:topic_already_exists	RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS	Broker: Topic already exists
37	:invalid_partitions	RD_KAFKA_RESP_ERR_INVALID_PARTITIONS	Broker: Invalid number of partitions
38	:invalid_replication_factor	RD_KAFKA_RESP_ERR_INVALID_REPLICATION_FACTOR	Broker: Invalid replication factor
39	:invalid_replica_assignment	RD_KAFKA_RESP_ERR_INVALID_REPLICA_ASSIGNMENT	Broker: Invalid replica assignment
40	:invalid_config	RD_KAFKA_RESP_ERR_INVALID_CONFIG	Broker: Configuration is invalid
41	:not_controller	RD_KAFKA_RESP_ERR_NOT_CONTROLLER	Broker: Not controller for cluster
42	:invalid_request	RD_KAFKA_RESP_ERR_INVALID_REQUEST	Broker: Invalid request
43	:unsupported_for_message_format	RD_KAFKA_RESP_ERR_UNSUPPORTED_FOR_MESSAGE_FORMAT	Broker: Message format on broker does not support request
44	:policy_violation	RD_KAFKA_RESP_ERR_POLICY_VIOLATION	Broker: Policy violation
45	:out_of_order_sequence_number	RD_KAFKA_RESP_ERR_OUT_OF_ORDER_SEQUENCE_NUMBER	Broker: Broker received an out of order sequence number
46	:duplicate_sequence_number	RD_KAFKA_RESP_ERR_DUPLICATE_SEQUENCE_NUMBER	Broker: Broker received a duplicate sequence number
47	:invalid_producer_epoch	RD_KAFKA_RESP_ERR_INVALID_PRODUCER_EPOCH	Broker: Producer attempted an operation with an old epoch
48	:invalid_txn_state	RD_KAFKA_RESP_ERR_INVALID_TXN_STATE	Broker: Producer attempted a transactional operation in an invalid state
49	:invalid_producer_id_mapping	RD_KAFKA_RESP_ERR_INVALID_PRODUCER_ID_MAPPING	Broker: Producer attempted to use a producer id which is not currently assigned to its transactional id
50	:invalid_transaction_timeout	RD_KAFKA_RESP_ERR_INVALID_TRANSACTION_TIMEOUT	Broker: Transaction timeout is larger than the maximum value allowed by the broker's max.transaction.timeout.ms
51	:concurrent_transactions	RD_KAFKA_RESP_ERR_CONCURRENT_TRANSACTIONS	Broker: Producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing
52	:transaction_coordinator_fenced	RD_KAFKA_RESP_ERR_TRANSACTION_COORDINATOR_FENCED	Broker: Indicates that the transaction coordinator sending a WriteTxnMarker is no longer the current coordinator for a given producer
53	:transactional_id_authorization_failed	RD_KAFKA_RESP_ERR_TRANSACTIONAL_ID_AUTHORIZATION_FAILED	Broker: Transactional Id authorization failed
54	:security_disabled	RD_KAFKA_RESP_ERR_SECURITY_DISABLED	Broker: Security features are disabled
55	:operation_not_attempted	RD_KAFKA_RESP_ERR_OPERATION_NOT_ATTEMPTED	Broker: Operation not attempted
56	:kafka_storage_error	RD_KAFKA_RESP_ERR_KAFKA_STORAGE_ERROR	Broker: Disk error when trying to access log file on disk
57	:log_dir_not_found	RD_KAFKA_RESP_ERR_LOG_DIR_NOT_FOUND	Broker: The user-specified log directory is not found in the broker config
58	:sasl_authentication_failed	RD_KAFKA_RESP_ERR_SASL_AUTHENTICATION_FAILED	Broker: SASL Authentication failed
59	:unknown_producer_id	RD_KAFKA_RESP_ERR_UNKNOWN_PRODUCER_ID	Broker: Unknown Producer Id
60	:reassignment_in_progress	RD_KAFKA_RESP_ERR_REASSIGNMENT_IN_PROGRESS	Broker: Partition reassignment is in progress
61	:delegation_token_auth_disabled	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_AUTH_DISABLED	Broker: Delegation Token feature is not enabled
62	:delegation_token_not_found	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_NOT_FOUND	Broker: Delegation Token is not found on server
63	:delegation_token_owner_mismatch	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_OWNER_MISMATCH	Broker: Specified Principal is not valid Owner/Renewer
64	:delegation_token_request_not_allowed	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_REQUEST_NOT_ALLOWED	Broker: Delegation Token requests are not allowed on this connection
65	:delegation_token_authorization_failed	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_AUTHORIZATION_FAILED	Broker: Delegation Token authorization failed
66	:delegation_token_expired	RD_KAFKA_RESP_ERR_DELEGATION_TOKEN_EXPIRED	Broker: Delegation Token is expired
67	:invalid_principal_type	RD_KAFKA_RESP_ERR_INVALID_PRINCIPAL_TYPE	Broker: Supplied principalType is not supported
68	:non_empty_group	RD_KAFKA_RESP_ERR_NON_EMPTY_GROUP	Broker: The group is not empty
69	:group_id_not_found	RD_KAFKA_RESP_ERR_GROUP_ID_NOT_FOUND	Broker: The group id does not exist
70	:fetch_session_id_not_found	RD_KAFKA_RESP_ERR_FETCH_SESSION_ID_NOT_FOUND	Broker: The fetch session ID was not found
71	:invalid_fetch_session_epoch	RD_KAFKA_RESP_ERR_INVALID_FETCH_SESSION_EPOCH	Broker: The fetch session epoch is invalid
72	:listener_not_found	RD_KAFKA_RESP_ERR_LISTENER_NOT_FOUND	Broker: No matching listener
73	:topic_deletion_disabled	RD_KAFKA_RESP_ERR_TOPIC_DELETION_DISABLED	Broker: Topic deletion is disabled
74	:fenced_leader_epoch	RD_KAFKA_RESP_ERR_FENCED_LEADER_EPOCH	Broker: Leader epoch is older than broker epoch
75	:unknown_leader_epoch	RD_KAFKA_RESP_ERR_UNKNOWN_LEADER_EPOCH	Broker: Leader epoch is newer than broker epoch
76	:unsupported_compression_type	RD_KAFKA_RESP_ERR_UNSUPPORTED_COMPRESSION_TYPE	Broker: Unsupported compression type
77	:stale_broker_epoch	RD_KAFKA_RESP_ERR_STALE_BROKER_EPOCH	Broker: Broker epoch has changed
78	:offset_not_available	RD_KAFKA_RESP_ERR_OFFSET_NOT_AVAILABLE	Broker: Leader high watermark is not caught up
79	:member_id_required	RD_KAFKA_RESP_ERR_MEMBER_ID_REQUIRED	Broker: Group member needs a valid member ID
80	:preferred_leader_not_available	RD_KAFKA_RESP_ERR_PREFERRED_LEADER_NOT_AVAILABLE	Broker: Preferred leader was not available
81	:group_max_size_reached	RD_KAFKA_RESP_ERR_GROUP_MAX_SIZE_REACHED	Broker: Consumer group has reached maximum size
82	:fenced_instance_id	RD_KAFKA_RESP_ERR_FENCED_INSTANCE_ID	Broker: Static consumer fenced by other consumer with same group.instance.id
83	:eligible_leaders_not_available	RD_KAFKA_RESP_ERR_ELIGIBLE_LEADERS_NOT_AVAILABLE	Broker: Eligible partition leaders are not available
84	:election_not_needed	RD_KAFKA_RESP_ERR_ELECTION_NOT_NEEDED	Broker: Leader election not needed for topic partition
85	:no_reassignment_in_progress	RD_KAFKA_RESP_ERR_NO_REASSIGNMENT_IN_PROGRESS	Broker: No partition reassignment is in progress
86	:group_subscribed_to_topic	RD_KAFKA_RESP_ERR_GROUP_SUBSCRIBED_TO_TOPIC	Broker: Deleting offsets of a topic while the consumer group is subscribed to it
87	:invalid_record	RD_KAFKA_RESP_ERR_INVALID_RECORD	Broker: Broker failed to validate record
88	:unstable_offset_commit	RD_KAFKA_RESP_ERR_UNSTABLE_OFFSET_COMMIT	Broker: There are unstable offsets that need to be cleared
89	:throttling_quota_exceeded	RD_KAFKA_RESP_ERR_THROTTLING_QUOTA_EXCEEDED	Broker: Throttling quota has been exceeded
90	:producer_fenced	RD_KAFKA_RESP_ERR_PRODUCER_FENCED	Broker: There is a newer producer with the same transactionalId which fences the current one
91	:resource_not_found	RD_KAFKA_RESP_ERR_RESOURCE_NOT_FOUND	Broker: Request illegally referred to resource that does not exist
92	:duplicate_resource	RD_KAFKA_RESP_ERR_DUPLICATE_RESOURCE	Broker: Request illegally referred to the same resource twice
93	:unacceptable_credential	RD_KAFKA_RESP_ERR_UNACCEPTABLE_CREDENTIAL	Broker: Requested credential would not meet criteria for acceptability
94	:inconsistent_voter_set	RD_KAFKA_RESP_ERR_INCONSISTENT_VOTER_SET	Broker: Indicates that the either the sender or recipient of a voter-only request is not one of the expected voters
95	:invalid_update_version	RD_KAFKA_RESP_ERR_INVALID_UPDATE_VERSION	Broker: Invalid update version
96	:feature_update_failed	RD_KAFKA_RESP_ERR_FEATURE_UPDATE_FAILED	Broker: Unable to update finalized features due to server error
97	:principal_deserialization_failure	RD_KAFKA_RESP_ERR_PRINCIPAL_DESERIALIZATION_FAILURE	Broker: Request principal deserialization failed during forwarding