Skip to Main Content

Overview

This article offers an in-depth look at the product advantages and essential capabilities of AutoMQ for Kafka, also known as AutoMQ Kafka.

Overview of AutoMQ for Kafka​

When deploying Apache Kafka® clusters on a large scale in production settings, enterprises commonly encounter several challenges:

  • High Costs, Rapid Inflation: Big data environments typically require high throughput and extensive data storage, necessitating reserved disk space that is both inflexible and expensive.

  • Slow Partition Reassignment: Scaling operations often involve partition reassignment and the transfer of historical data, processes that can take several hours and may impact service performance due to cold data reads.

  • Lack of Self-balancing Traffic: Nodes frequently have unequal partition loads and lack an effective self-balancing mechanism, leading to either idle nodes or hotspot failures.

These challenges stem from the design of Apache Kafka®, which is optimized for IDC and integrated storage-compute scenarios and cannot be fundamentally resolved. With the advent of cloud computing, which has redefined the basic constructs of hardware and software with deterministic services, it is crucial to redesign Kafka based on the cloud to fully exploit the capabilities of service-orientation and infinite resource pools provided by underlying cloud products, thus effectively addressing the issues of high costs, limited scalability, and poor operational efficiency.

AutoMQ Kafka is a next-generation Kafka distribution that is redesigned for cloud-native environments. Maintaining 100% compatibility with Apache Kafka®, AutoMQ Kafka provides users with up to tenfold cost advantages and hundredfold scalability benefits, along with second-level partition reassignment and automated traffic self-balancing, effectively resolving operational challenges.

Advantages of AutoMQ for Kafka​

10x Cost Optimization​

AutoMQ Kafka's innovative cloud-native architecture fully harnesses cloud services like Object Storage and Spot Instances, offering features such as high availability and elastic provisioning, which provide customers with a 10x cost reduction compared to Apache Kafka.

  • Utilizing object storage as the primary storage option drastically cuts storage expenses.

  • Its single-replica high availability design reduces traffic replication costs by two-thirds.

  • Leveraging Spot Instances, combined with flexible scaling strategies, significantly lowers computing expenses.

Enhanced Stability​

Apache Kafka demands substantial bandwidth for data replication during scaling processes, making it ill-equipped to manage abrupt traffic spikes. In AutoMQ Kafka's shared storage framework, partition data is completely stored on S3, enabling the cluster to scale swiftly without the need for data replication to accommodate sudden traffic increases.

Built on a fully decoupled storage-compute architecture, AutoMQ Kafka fully exploits cloud-native capabilities, achieving high autonomy through automatic scaling, self-balancing traffic, and auto-failure recovery, thereby mitigating the stability risks associated with traditional manual management.

Instant Reassignment of Partitions and Self-balancing of Traffic​

AutoMQ Kafka completely isolates storage state into object storage services, rendering the business logic layer entirely stateless. Clusters can execute partition reassignment and traffic self-balancing in seconds, effectively resolving the issues of slow scalability, self-balancing, and challenging partition reassignment that plague Apache Kafka. Paired with cloud providers' elastic scaling group strategies, it seamlessly achieves adaptive elastic scaling of clusters.

100% Compatible with Apache Kafka​

Unlike other vendors that reimplement the Kafka protocol, AutoMQ Kafka opts for a minimal aspect replacement at the storage layer, only modifying the underlying LogSegment implementation, while still reusing the core code from various versions of Apache Kafka. AutoMQ Kafka ensures 100% compatibility with Apache Kafka and can quickly adapt to new versions.

AutoMQ Kafka has already achieved the following compatibilities:

  • Passed verification with over 900+ functional test cases.

  • Functionality spans over 1000+ Kafka Improvement Proposals (KIPs) of Apache Kafka.

  • Compatible with versions 0.9.x to 3.4.x, and maintains pace with the Community Edition, updating at a T+1 month interval.