Skip to Main Content

Overview

This article provides a comprehensive overview of the advantages and core capabilities of the AutoMQ for Kafka product, hereafter referred to as AutoMQ Kafka.

Introduction to AutoMQ for Kafka​

When enterprises deploy large-scale Apache Kafka® clusters in production environments, they inevitably encounter the following challenges:

  • High Cost and Rapid Expansion: Big data scenarios often involve high throughput and the need to store large amounts of data. This necessitates reserving disk space, which is inflexible and costly.

  • Slow Partition Reassignment: Scaling up or down requires partition reassignment and historical data migration, which can take several hours. During this period, cold data reads may also impact services.

  • Lack of Self-Balancing: Different partitions and loads across nodes lack a self-balancing mechanism, making it easy for local nodes to become idle or suffer from hotspot failures.

These pain points are rooted in Apache Kafka's design philosophy, which targets IDC and integrates storage and computation, making them fundamentally unsolvable. Today, cloud computing has redefined the earliest hardware and software with deterministic services. It is necessary to redesign Kafka based on the cloud to fully leverage the underlying cloud products' capabilities for service-orientation and infinite resource pools, thereby thoroughly addressing the issues of high costs, lack of elasticity, and low operational efficiency.

AutoMQ Kafka is a cloud-native redesign of the next-generation Kafka distribution. While maintaining 100% compatibility with Apache Kafka, AutoMQ Kafka offers users up to 10x cost savings and 100x elasticity improvements. It also supports second-level partition reassignment and automatic traffic self-balancing, addressing operational pain points.

AutoMQ for Kafka Advantages​

10x Cost Optimization​

The new cloud-native architecture of AutoMQ Kafka leverages cloud services such as object storage and Spot instances for high availability and elastic provisioning. This offers customers a 10x cost advantage compared to Apache Kafka.

  • By using object storage as the primary storage, the storage cost per unit is significantly reduced.

  • The single-replica high availability architecture saves two-thirds of the traffic replication cost.

  • Fully utilizing Spot instances and combining with elastic scaling strategies helps reduce computation costs.

Stability Enhancement​

Apache Kafka requires substantial bandwidth for data replication during scaling, making it incapable of handling sudden traffic spikes. In the shared storage architecture of AutoMQ Kafka, partition data is completely stored on S3, allowing the cluster to quickly handle sudden traffic surges without the need for data replication during scaling.

On the foundation of a fully disaggregated storage and compute architecture, AutoMQ Kafka can fully leverage cloud-native capabilities. Through automatic scaling, self-balancing of traffic, and automated fault recovery, the system becomes highly autonomous, eliminating the stability risks associated with traditional manual governance.

Second-Level Partition Reassignment and Self-Balancing of Traffic​

AutoMQ Kafka completely separates storage state into object storage services, with the business logic layer being entirely stateless. The cluster can complete partition reassignment and self-balancing of traffic within seconds, thoroughly addressing the slow scaling and balancing, and difficult partition reassignment pain points of Apache Kafka. Combined with cloud providers' elastic scaling group strategies, clusters can easily achieve adaptive elastic scaling.

100% Compatibility with Apache Kafka​

Unlike other vendors who re-implement the Kafka protocol, AutoMQ Kafka opts for a minimally invasive storage layer replacement approach, only modifying the underlying LogSegment implementation while reusing the major codebase of Apache Kafka across various versions. AutoMQ Kafka can effortlessly achieve 100% compatibility with Apache Kafka and quickly adapt to new versions.

Previously, AutoMQ Kafka has completed the following certifications for compatibility:

  • Passed over 900 functional use case verifications.

  • The functional scope covers over 1000 KIPs of Apache Kafka.

  • Compatible with versions 0.9.x to 3.4.x, and capable of catching up with the community version at T+1 month speed.