In version 3.6.0 Kafka introduced early access to the Tiered Storage feature (KIP-405) that significantly improves operation experience and decreases cluster costs.
Existing problems with scalability and efficiency that this change is intended to solve:
📍Huge disk capacity that is required to keep data for long period of time (retention policy for days, weeks or even months).
📍Processing speed may be impacted by big amount of data kept in the cluster
📍Home-grown implementations to copy old data to external storages like HDFS
📍Expensive scaling approach. Kafka is scaled by adding new brokers that also require RAM and CPU, not possible to scale disks only
📍Copying a lot of data in case node failure as new node must copy all the data that was on the failed broker from other replicas
📍High recovery time. The time for recovery and rebalancing is proportional to the amount of data stored locally on a Kafka broker.
Suggested solution:
✏️ Use Tiered Storage pattern. Split data management on separate tires based on performance and access requirements, cost considerations. Most commonly used tiers:
-"Hot": Local storage to keeps the most critical and frequently accessed data
- "Warm": Remote lower-cost storage that keeps less critical or infrequently accessed data
-"Cold": Low-cost storage to keep periodic backup data
✏️ Kafka storage is split on local and remote storages. Local storage is the same as it's in Kafka now. Remote storage is a pluggable storage that can be HDFS, S3, Azure blob, etc.
✏️ Inactive segments are copied to the remote storage according to the configured retention policy
✏️ Remote and local storages have their own retention policies. So local storage can be very short like few hours.
✏️ Any data that exceeds the local retention threshold will not be removed until successfully uploaded to the remote storage
✏️ Clients can still get older data, it will be read from the remote storage
✏️ Feature is enabled by
remote.log.storage.system.enable on the cluster and remote.storage.enable on the topic✏️ New metrics are introduced to monitor integration performance with the remote storage
Current limitations:
* Compacted topics are not supported
* To disable remote storage, the topic must be recreated
To sum up, Tiered Storage allows scale storage independently from cluster size that reduces overall usage costs. It is still in an early access state (3.8.0 version) and is not recommended for production use. However, there is significant interest in it. AWS has announced S3 support in their MSK service, and Uber has reported successfully running the feature in production.
#news #architecture #technologies #kafka