Skip to Main Content

Overview

AutoMQ has reengineered Kafka for the cloud by separating storage to object storage while ensuring 100% compatibility with Apache Kafka® and delivering users up to 10 times the cost advantage and a hundredfold scalability advantage.

Design Philosophy​

Storage Decoupled to Shared Cloud Storage Services​

The benefits of decoupling storage and compute are well acknowledged across the industry. However, most implementations of this principle involve dividing storage into a self-managed distributed storage system, which significantly increases the complexity of software deployment, operations, and management. AutoMQ contends that separating storage from the software and shifting it to shared cloud storage services represents the best approach in today's cloud-native landscape.

By developing S3Stream, a streaming storage solution based on S3, AutoMQ diverts storage to shared cloud storage provided by cloud providers like EBS and S3. This strategy fully utilizes their storage features to offer cost-effective, low-latency, highly available, highly reliable, and infinitely scalable streaming storage solutions. For further technical information, please see the technical architecture chapter.

Shared Storage Is Superior to the Shared Nothing Architecture​

Apache Kafka uses local disks to deliver highly dependable storage, creating an infinite stream storage abstraction for business processes. Data is stored on each node's disks following specific logic, typically known as the Shared Nothing architecture.

Local disks lack scalability, so the Shared Nothing architecture often achieves higher throughput by scaling horizontally. However, shared cloud storage has become highly elastic and nearly "infinite" in capacity, making it simpler and more cost-effective to leverage the full potential of cloud storage with a shared storage architecture.

Separation of Durability and Availability​

Apache Kafka natively supports a multi-replica mechanism that ensures high data reliability through replica redundancy, while also enhancing system availability via a leader election process among different replicas for fast failover.

In cloud environments, EBS incorporates a 3-replica storage mechanism. Implementing an additional 3-replica storage in Kafka would result in data being stored nine times, significantly increasing storage, bandwidth, and computational costs. AutoMQ contends that cloud-native Kafka does not need a multi-replica mechanism to simultaneously provide both reliability and availability. By adopting a divide and conquer strategy, delegating reliability to cloud storage and providing availability independently embodies true cloud-native practice.

Technical Advantages​

Compared to Apache Kafka, AutoMQ offers the following benefits:

10x Cost Advantage​

AutoMQ's entirely new cloud-native architecture leverages Object storage for its high availability and elastic provisioning capabilities, offering customers a 10-fold cost advantage over Apache Kafka.

  • Utilizing object storage as the primary storage can significantly lower storage costs.

  • High availability is achieved without the need for multi-replica duplication, saving two-thirds of the traffic and replication costs.

  • Natively supports Spot instances and AutoScaling, eliminating the need to reserve resources for peak demand.

Extreme Elasticity​

AutoMQ decouples state storage from business logic by utilizing Object storage services, rendering the business logic layer completely stateless. AutoMQ clusters can complete partition reassignment and self-balancing in seconds, effectively resolving the slow self-balancing and difficult partition reassignment issues that occur during Apache Kafka's scaling. Paired with Cloud providers' elastic scaling group policies, it seamlessly achieves adaptive elasticity for clusters.

Cold Reads Do Not Impact Message Writing​

Each server in Apache Kafka provides a fixed amount of IOPS for message reading and writing, and if there is a large-scale cold read, it can cause IOPS to reach its limit, potentially leading to message write queuing and timeouts. This is due to the limitations of Kafka's integrated storage-compute architecture. In contrast, AutoMQ adopts a compute-storage separation architecture where hot and cold reads do not interfere with each other. The throughput of cold reads depends on the throughput capability of the Object storage, and with EBS as WAL, which is exclusively occupied by the write message process, this issue is effectively avoided in the AutoMQ architecture.

Zero Service Interruption​

AutoMQ stores data entirely on S3, allowing for rapid response to sudden traffic spikes during cluster scaling without the need for data duplication. In contrast, Apache, after scaling, requires substantial bandwidth for data replication, making it difficult to handle sudden traffic. Through automatic scaling, self-balancing, and automatic failure recovery, AutoMQ achieves a high degree of system autonomy, requiring no manual intervention for higher levels of availability.

100% Compatible with Apache Kafka®​

Unlike other providers that re-implement the Kafka protocol, AutoMQ employs a minimally invasive strategy at the storage layer. It modifies only the underlying LogSegment implementation while preserving most of the original Apache Kafka code. This approach allows AutoMQ to maintain 100% compatibility with Apache Kafka and seamlessly adapt to new releases.

Related Content​