Skip to Main Content

Difference with Tiered Storage

AutoMQ leverages Object storage as its principal storage service, and starting with version 3.6.0, Apache Kafka® has introduced KIP-405, a tiered storage solution that utilizes Object storage to offload historical data. This article explores the benefits and distinctions between AutoMQ and the tiered storage version of Apache Kafka®.

Architecture: EBS as Disk Versus EBS as Service​

Drawing from the framework outlined in KIP-405, the tiered storage model of Apache Kafka® employs a dual-tier storage strategy, utilizing both local disks and Object storage. Initially, message data is written to local disks and later asynchronously transferred to Object storage using a "turn cold" strategy. Considering the susceptibility of local disks to damage, each message, according to the replication strategy, must be duplicated across nodes to multiple disks via the ISR mechanism.

Currently, in the Public Cloud setting deploying the tiered storage version, the architecture of Apache Kafka® remains unchanged; it still utilizes EBS as an alternative for local disks, necessitating message replication across multiple EBS volumes. In essence, Apache Kafka® continues to treat EBS as a traditional block storage medium, akin to a physical disk in a local data center.

Public Cloud providers deliver EBS services with robust reliability and high availability guarantees. AutoMQ considers EBS a cloud storage service, capitalizing on the inherent reliability of EBS (featuring three replicas, offering five nines to nine nines of reliability) and its fault tolerance capabilities within and across availability zones. Consequently, AutoMQ can forego additional replications on EBS, thereby conserving significant storage, network, and computing resources.

Cost: EBS as Storage Versus EBS as Recovery WAL​

In Apache Kafka®'s tiered storage architecture, the primary EBS storage continues to act as the main storage, providing read and write services, with each Kafka partition required to maintain the last active Segment on the primary storage. This results in the following situation:

  • The required EBS space is variable and directly proportional to the number of cluster partitions.

  • Production environments necessitate reserving significant EBS space to reduce risks.

  • Reserving EBS is expensive, and the storage savings from tiered architecture are minimal.

Example:

Consider Apache Kafka's default setup, where each segment is 1GB. With 1000 active partitions, reserving 1TB of EBS is essential.

In AutoMQ's design, object storage serves as the main storage, with EBS used for fault recovery WAL, configured as a circular overwriting raw device. Each AutoMQ Broker node requires just a 2GB EBS volume, capable of temporarily holding around 500MB of data (these capacities are adjustable).

This approach ensures that AutoMQ's use of EBS space is predictable, maintaining very low storage costs, as a 2GB EBS volume costs merely 1 yuan per month.

Elasticity: Hour-level Reassignment Versus Second-level Reassignment​

Apache Kafka's layered storage design leads to a variable primary storage size and unpredictable data volumes left on EBS per partition, which affects the speed of operations like elastic scaling and fault reassignment, hindering rapid scalability.

In AutoMQ, buffer data up to 500MB can be uploaded to object storage in seconds, facilitating partition reassignments at an equivalent pace.

When examining a scenario with Confluent in a high-traffic cluster, the expansion process takes 43 hours without a multi-tier storage architecture, and still 1.4 hours with it.

Overview​

Compared to Apache Kafka's multi-tier storage solution, AutoMQ heralds a radical shift resulting from incremental improvements. By optimizing the architecture, AutoMQ adopts a "stateless" configuration that enables seamless scaling and swift partition reassignments. On the other hand, Apache Kafka's multi-tier storage solution remains a stateful optimization strategy, posing challenges for lightweight scaling and quick partition adjustments.