Difference with Apache Kafka
AutoMQ is a new generation of Apache Kafka® distribution, reengineered with cloud-native principles. It boasts a cost advantage of up to 10 times and enhanced scalability, while maintaining full compatibility with the Apache Kafka protocol. This article focuses on the key differences and relationships between AutoMQ and Apache Kafka.
Differences from Apache Kafka
Architecture: Shared Nothing vs. Shared Storage
Apache Kafka utilizes local disk storage and establishes a highly reliable storage system through software-driven high availability replica replication logic (ISR mechanism), offering an "infinite" stream storage abstraction to the business logic layer. All Kafka data is stored on local disks based on a specific logic, typically referred to as the Shared Nothing architecture.
Conversely, AutoMQ employs a compute-storage separation strategy, shifting away from local disks and instead using shared object storage services for data storage. AutoMQ introduces an S3Stream storage repository (software library) to replace Apache Kafka’s local Log storage, seamlessly leveraging object storage to maintain Kafka data while preserving the upper-layer Apache Kafka functional semantics. This approach is known as the Shared Storage architecture.
The table below compares the two different architectures of Apache Kafka and AutoMQ:
| Apache Kafka Adopts Shared Nothing architecture | AutoMQ Adopts Shared Storage architecture |
|---|---|
| Data is on local disks, requiring cross-node replication of multiple copies | Data is in S3 shared storage (three-replica high reliability), eliminating the need for multiple replica replication |
| Data is isolated among nodes, with access bound to specific nodes | Data is shared across nodes, enabling cross-node access |
| Adding nodes or replacing faulty nodes requires reassigning shard data | Adding nodes or replacing faulty nodes allows for quick switching without the need for data reassignment |
Note:
Apache Kafka, starting with version 3.6, has introduced plans for tiered storage (which is not yet ready for production), enabling the offloading of historical data to object storage services. This architecture shares both similarities and differences with AutoMQ, which relies solely on object storage for its storage infrastructure. For an in-depth comparison, please refer to Difference with Tiered Storage▸.
Elasticity: Seconds-level Partition Reassignment vs. Hours-level Partition Reassignment
Partition reassignment is a common and necessary task in Kafka production environments, required in scenarios such as local node failure, cluster scaling, and managing local traffic spikes.
Apache Kafka employs a Shared Nothing architecture, wherein each partition's data is stored exclusively on a specific storage node. Should a partition reassignment occur, it requires the transfer of the complete partition's data to a new target node to restore service. This process can be extensive and unpredictable.
Example:
Taking a Kafka partition with a write throughput of 100MiB/s as an example, the resulting daily data volume is approximately 8.2TiB. If reassignment of this partition is necessary, all data must be relocated to other nodes. Even with a 1Gbps network bandwidth, this reassignment process could take several hours.
AutoMQ utilizes a compute-storage separation architecture, where the complete data of each partition is stored in S3 object storage. During partition reassignment, only a minimal amount of metadata needs to be synchronized to complete the switch. AutoMQ can ensure the switch is completed within seconds for partitions at any scale of write throughput.
AutoMQ supports second-level partition reassignment, providing AutoMQ with enhanced and more predictable flexibility in scenarios such as cluster scaling and fault recovery, compared to Apache Kafka.
Cost: 10 Times Cost Difference
Given the differences in technical architecture described above, AutoMQ and Apache Kafka also exhibit significant variations in their cost structures for computing and storage. AutoMQ does not require cross-node replication of multiple copies when writing messages, which substantially reduces cross-node replication traffic and stress. Additionally, AutoMQ employs S3 object storage as its storage medium, which is considerably less expensive than the EBS block storage mounted on each node in typical public cloud environments.
Specific comparison items are as follows:
| Cost Comparison | Apache Kafka | AutoMQ |
|---|---|---|
| Storage unit price |
|
|
| Inter-node Replication Traffic |
|
|
Remarks:
The storage unit prices mentioned are based on AWS S3 in the US East region for EBS GP3 instances and S3 Standard Storage. For more information, see this link.
The inter-node replication traffic cost is exemplified by AWS AZ inter-traffic transfer costs.
For a detailed cost comparison between AutoMQ and Apache Kafka, refer to Cost-Effective: AutoMQ vs. Apache Kafka▸.
Capacity: Reserved Vs Pay-as-you-go
Capacity planning is another challenge in large-scale Kafka deployment in production environments. Due to architectural differences and storage media variations between AutoMQ and Apache Kafka, there are differences in capacity planning:
| Apache Kafka Uses local disk, integrated storage-computation | AutoMQ Uses S3 object storage, separated storage-computation |
|---|---|
| Disk space needs to be reserved in advance. | Storage space does not need to be reserved; it is used as needed and paid for by the amount used. |
| Single-node storage is limited and has poor scalability. | S3 object storage space is virtually unlimited, providing good scalability. |
| 100% Compatible with Apache Kafka.
AutoMQ represents a next-generation Kafka distribution, redesigned for cost-efficiency and elasticity, while maintaining full compatibility with Apache Kafka. Applications built on Apache Kafka can be seamlessly transitioned to AutoMQ without any need for changes or adjustments.
In the storage layer, AutoMQ introduces S3Stream as a streaming storage solution, replacing Apache Kafka's local log storage. This layer maintains the same partition abstraction, allowing higher-level modules such as KRaft metadata management, Coordinator, ReplicaManager, and KafkaApis to leverage existing code. Thus, the upper storage layer effectively utilizes Apache Kafka’s established logic. AutoMQ ensures complete protocol and semantic compatibility with Apache Kafka, while staying updated with Apache’s latest enhancements and bug fixes.
For detailed information on AutoMQ's compatibility with Apache Kafka, please visit Compatibility with Apache Kafka▸.