Skip to Main Content

Overview

AutoMQ is a next-generation Kafka release redesigned based on cloud-native concepts, and it is compatible with Apache Kafka®. This article will detail how to migrate from Apache Kafka® or other Kafka distributions to AutoMQ.

Migration Scenarios​

AutoMQ is compatible with Apache Kafka®, supporting seamless reassignment from other Kafka distributions provided by different vendors. The relevant information is listed below:

Source Cluster
Target Cluster
Reassignment Support
Apache Kafka
AutoMQ
Supported
AWS MSK
Supported
Confluent Platform
Supported
Alibaba Cloud Message Queue Kafka Edition
Supported
Tencent Cloud CKafka Edition
Supported

If the Kafka release you are currently using is not listed above, please contact the community for support and assistance.

Migration Tool​

AutoMQ recommends using the MirrorMaker2 tool for cluster reassignment, supporting reassignment from Apache Kafka® or other Kafka distributions to AutoMQ.

MirrorMaker2 migration tool has the following capabilities:

  • Supports batch migration of clusters: MirrorMaker2 facilitates batch migration of Topics and Groups from the source cluster to the target cluster, enabling phased rollout and validation.

  • Supports synchronization of consumption progress: MirrorMaker2 ensures real-time synchronization of consumption progress from the source cluster, allowing consumers to continue consuming from the target cluster after migration.

  • Supports dynamic updates of Topics and Groups: MirrorMaker2 enables real-time monitoring of dynamic updates of Topics and Groups, ensuring that no updates are missed during the migration process.

The technical architecture of MirrorMaker2 is as follows:

Core Concepts​

MirrorMaker2 defines several Connectors to implement data synchronization tasks between two Apache Kafka® clusters, including necessary configurations and real-time running states.

  • MirrorSourceConnector: Used for synchronizing message data from the source cluster.

  • MirrorCheckpointConnector: Used to synchronize the consumption progress data of the source cluster ConsumerGroup.

  • MirrorHeartbeatConnector: Used for system detection of the synchronization link's operational status.

Deploying MirrorMaker2​

For the installation and deployment of MirrorMaker2, refer to the relevant links from the Apache Kafka community:

Constraints and Limitations​

MirrorMaker2 uses Connectors to synchronize messages between the source and target clusters. This mechanism cannot guarantee complete data consistency between the source and target clusters. Therefore, when using the reassignment tool, please note the following constraints and limitations:

Minor Message Duplication

Consumer progress data is asynchronously and intermittently synchronized, which may cause offset rollbacks and result in duplicate consumption when switching consumers.

Message synchronization is an asynchronous operation, and due to machine anomalies or fault recovery, partial data may be duplicated.

Minor Message Reordering

If offset rollbacks occur when switching consumers, a small number of messages will be reconsumed, leading to disorder in the message sequence.

Next Steps​

After understanding the background of cluster reassignment, it is recommended to choose the corresponding solution based on the current cluster situation: