Queues for Kafka

`Kafka Queues: Now and in the Future` breaks down how consumer groups work today and what updates are coming in future Kafka versions. Future is the most interesting part actually.

So, Kafka is exceptionally effective for streaming large volumes of data through topics and partitions. Consumer groups manage this data by dividing the workload based on the number of partitions. While this method offers high performance, it also presents notable limitations:
- Consumers are exclusively assigned specific partitions.
- Parallel data processing is restricted by the number of partitions in a topic
- There's no option for individual acknowledgment or message redelivery
- Messages cannot be marked as rejected
- Message processing times can vary unevenly

So, KIP-932 has been introduced to enhance Kafka's queue capabilities. This proposal suggests a 'share group' with a `share-partition` abstractions.
Key changes:
- A share-partition can be assigned to any number of consumers, removing the scaling limitation by the number of partitions.
- Each message has a state: available for processing, acquired, acknowledged, or rejected.
- Consumers receive available messages from shared partitions, transitioning the message state to acquired
- Messages not acknowledged within a specific timeframe are returned to the available state.
- If the number of delivery attempts exceeds a defined threshold, a message is marked as rejected, halting further delivery attempts
- Consumed messages are not guaranteed to be ordered

Currently, the KIP is in the Accepted state and is planned for release in Kafka 4.0.

#news #architecture #kafka