Skip to Main Content

Overview

AutoMQ is a cutting-edge Kafka distribution, reimagined with cloud-native principles and fully compatible with Apache Kafka®. This article offers a comprehensive guide on transitioning from Apache Kafka or other Kafka distributions to AutoMQ.

Reassignment Scenarios​

AutoMQ is compatible with Apache Kafka and facilitates seamless transitions from Kafka distributions offered by other providers, as detailed below:

Source Cluster
Target Cluster
Reassignment Support
Apache Kafka
AutoMQ
Supported
AWS MSK
AutoMQ
Supported
Confluent Platform
AutoMQ
Supported
Alibaba Cloud Kafka
AutoMQ
Supported
Tencent Cloud CKafka Edition
AutoMQ
Supported

If the Kafka release version you are currently using is not listed above, please contact the community for support and assistance.

Reassignment Tool​

AutoMQ recommends using the MirrorMaker2 tool for cluster reassignments, enabling transfers from Apache Kafka or other Kafka versions to AutoMQ.

MirrorMaker2 reassignment tool has the following capabilities:

  • Supports batch reassignment of clusters: MirrorMaker2 facilitates the batch reassignment of Topics and Groups from the source cluster to the target cluster, enabling phased rollouts and thorough validation.

  • Supports synchronization of consumption progress: MirrorMaker2 ensures real-time synchronization of consumption progress in the source cluster, permitting consumers to continue their consumption seamlessly in the target cluster after the transfer.

  • Supports synchronization of dynamic updates for Topics and Groups: MirrorMaker2 actively tracks the dynamic updates of Topics and Groups, ensuring comprehensive coverage during the reassignment process.

The technical architecture of MirrorMaker2 is as follows:

Core Concepts​

MirrorMaker2 introduces several Connectors that enable data synchronization tasks between two Apache Kafka® clusters, complete with necessary configurations and real-time operational status monitoring.

  • MirrorSourceConnector: Facilitates the synchronization of message data from the source cluster.

  • MirrorCheckpointConnector: Manages the synchronization of Consumer Group consumption progress data from the source cluster.

  • MirrorHeartbeatConnector: Monitors the operational status of the synchronization link.

Deploying MirrorMaker2​

To install and deploy MirrorMaker2, consult the relevant resources in the Apache Kafka® community:

Constraints and Limitations​

MirrorMaker 2 employs Connectors to synchronize messages across source and destination clusters, a process that does not ensure perfect data consistency between the source and destination clusters. Consequently, users should be aware of the following constraints and limitations when utilizing this replication tool:

Minor Message Duplication

Consumer progress data is synchronized asynchronously and sporadically, potentially causing consumer offset rollbacks and resulting in duplicate messages when consumers are switched.

Message synchronization is an asynchronous process; due to machine anomalies or recovery from failures, some data may be duplicated.

Minor Consumption Disordering

When switching consumers, if there's a position rollback, a small number of messages may be consumed multiple times, potentially resulting in messages being out of order.

Subsequent Steps​

After exploring the background of the reassignment, it's advisable to choose the appropriate solution based on the current cluster conditions: