Skip to main content
AutoMQ uses object storage as its primary storage service, while Apache Kafka® introduced tiered storage with KIP-405 starting from version 3.6.0, leveraging object storage to offload historical data. The Overview▸ of AutoMQ consists of WAL (Write-Ahead Logging) storage and data main memory, whereas Kafka’s tiered storage includes primary EBS storage and secondary object storage. Developers often assume AutoMQ’s WAL storage is similar to Kafka’s primary EBS storage, but they fundamentally differ. This article will highlight the advantages and differences of AutoMQ in comparison to Apache Kafka’s tiered storage.

​
Architectural Differences

According to the design outlined in KIP-405, Apache Kafka’s tiered storage version adopts a two-tier storage approach, relying on both local disk and object storage. Message data is initially written to the local disk and then asynchronously uploaded to object storage based on a cooling-off strategy. Since local disks are susceptible to failure, each message needs to be replicated across multiple disks on different nodes via the ISR mechanism to ensure durability. Currently, when deploying the tiered storage version in a Public Cloud environment, Apache Kafka’s architecture remains unchanged, still using EBS as a replacement for local disks, requiring messages to be replicated across multiple EBS instances. In summary, Apache Kafka still treats EBS as a standard block storage device, with no fundamental difference from a physical hard drive in a local data center. AutoMQ employs object storage as its primary storage method, without the concept of storage tiers. However, to optimize storage efficiency, such as reducing latency for writing to object storage and improving write efficiency for large partitions, AutoMQ introduces a WAL storage mechanism. The architectural comparison is as follows: Storage architecture comparison: AutoMQ WAL storage and object storage vs Apache Kafka Tiered Storage with EBS primary and object storage secondary Since WAL storage can use EBS as storage media, developers might think it has similarities with Kafka’s primary storage. However, WAL storage fundamentally differs from Kafka in terms of design philosophy and implementation, including storage responsibilities, storage efficiency, storage space, storage media, durability, and multi-AZ disaster recovery design. Please refer to the table below for detailed differences:

​
Cost Advantage

In Apache Kafka’s tiered storage architecture, the first tier of EBS storage is still used as the primary storage for read and write operations. Each Kafka partition must retain at least the latest active segment on the first tier storage. This leads to the following phenomenon:
  • EBS space is uncertain and directly related to the number of partitions in the cluster.
  • Reserving a large EBS space in the production environment is necessary to reduce risks.
  • EBS reservation costs are high, and the cost reduction potential through tiered storage is limited.
Example:Taking the default configuration of Apache Kafka® as an example, with each segment size set to 1GB, if the number of active partitions is 1000, it still requires reserving 1TB of EBS.
In AutoMQ’s architecture, object storage is used as the primary storage. WAL storage provides a lightweight buffer for fault recovery, with multiple backend options available. For example, with EBS WAL, each AutoMQ Broker node only needs a 2GB EBS volume and can guarantee the temporary storage of approximately 500MB of data (the aforementioned space sizes are customizable). With S3 WAL (the default for AutoMQ Open Source), no additional block storage is needed at all. This design ensures that AutoMQ’s WAL space consumption is predictable. When using EBS WAL, the additional storage cost remains low because the required EBS volume is small. When using S3 WAL (the default for AutoMQ Open Source), no additional block storage cost is incurred at all.

​
Operations Advantage

Due to the non-fixed primary storage space in Apache Kafka’s multi-tiered storage architecture, the data left on EBS for each partition is also non-fixed. Therefore, during operations like elastic scaling and fault reassignment, the time required is also uncertain, making quick scaling unachievable. While AutoMQ’s buffer only contains up to 500MB of data that needs to be uploaded to object storage, the upload can be completed within seconds, thereby supporting second-level partition reassignment.
In the case of Confluent, an expansion operation on a high-traffic cluster takes 43 hours in a non-tiered storage architecture and still requires 1.4 hours in a tiered storage architecture.

​
Summary

Compared to Apache Kafka’s tiered storage solution, AutoMQ represents a qualitative leap driven by quantitative changes. Through architectural optimization, AutoMQ achieves a “stateless” state, allowing for arbitrary scaling and second-level partition reassignments. In contrast, Apache Kafka’s tiered storage architecture remains an optimized yet stateful solution, making it challenging to achieve lightweight scaling and partition reassignment.