Overview
AutoMQ is a next-generation Kafka distribution redesigned based on cloud-native principles and is compatible with Apache Kafka. This article will provide a detailed guide on how to reassign from Apache Kafka or other Kafka distributions to AutoMQ.
Reassignment Scenarios
AutoMQ is compatible with Apache Kafka and supports seamless reassignment from other vendor-provided Kafka distributions. The relevant information is listed below:
| Source Cluster | Target Cluster | Reassignment Support |
|---|---|---|
| Apache Kafka | AutoMQ Cloud | Supported |
| AWS MSK | Supported | |
| Confluent Platform | Supported | |
| Alibaba Cloud Message Queue Kafka Edition | Supported | |
| Tencent Cloud CKafka Edition | Supported |
If the current Kafka release version you are using is not listed above, please contact us immediately for support and assistance.
Reassignment Tool
AutoMQ Cloud provides an out-of-the-box Kafka reassignment tool based on Kafka MirrorMaker2, which supports reassignment from Apache Kafka® or other Kafka releases to AutoMQ.
AutoMQ Cloud migration tool offers the following capabilities:
Adaptive Elastic Scaling: AutoMQ Cloud utilizes cloud providers' elastic scaling groups to manage the capacity of migration components. It can automatically scale according to the throughput pressure of synchronized data, ensuring the migration is completed as quickly as possible without requiring manual configuration of concurrency and node numbers.
Batch Migration of Clusters: AutoMQ Cloud supports the batch migration of Topics and Groups from the source cluster to the target cluster, enabling canary testing and validation.
Support for Synchronizing Consumption Progress: The AutoMQ Cloud migration tool supports real-time synchronization of consumption progress from the source cluster, allowing consumers to continue consumption from the target cluster after migration.
Support for Dynamic Updates of Topics and Groups: The AutoMQ Cloud migration tool supports real-time monitoring of dynamic updates to Topics and Groups, ensuring that no updates are missed during the migration process.
The technical architecture of the AutoMQ Cloud migration tool is as follows:
Core Concepts
A migration task defines a data synchronization task for a set of Topics from an external Apache Kafka cluster (or another Kafka distribution instance) to an AutoMQ instance. It includes the necessary configurations for synchronization and the real-time running status.
The migration tool is implemented based on the Kafka MirrorMaker2 tool. Each migration task is equivalent to a set of Connectors defined by MirrorMaker2:
MirrorSourceConnector: Used for synchronizing message data from the source cluster.
MirrorCheckpointConnector: Used for synchronizing ConsumerGroup consumption progress data from the source cluster.
MirrorHeartbeatConnector: Used for system detection of the operational status of the synchronization link.
Each migration task's Connectors are independently allocated resources and operate in isolation from each other. The system dynamically scales the number of Workers underlying each migration task based on the scale and pressure of the synchronized data.
AutoMQ's migration tool is free to use, but running each migration task independently allocates virtual machine resources and incurs cloud resource costs.
It is recommended to reasonably partition migration task batches and quantities, and to delete migration tasks as soon as they are completed to avoid continuous cloud resource costs.
Each migration task transitions through the following states from creation to deletion:
Creating: The initial reassignment task is an asynchronous process. When the creation operation is completed, the MirrorMaker2 component will be deployed asynchronously in the background. During this process, no changes can be made by the user until the creation is complete.
Running: The current reassignment task is active and is synchronizing data.
Modifying: The current reassignment task is performing configuration changes, state changes, or other operations. Since the execution process is asynchronous, one needs to wait for completion. During this period, no changes are allowed.
Service Exception: The current reassignment task cannot provide normal service due to issues arising from creation, operational changes, or underlying infrastructure exceptions.
Paused: The current reassignment task has been temporarily stopped by the user, retaining its configuration but not synchronizing data. This is generally used for inspection during the reassignment process.
Deleting: Once the reassignment task has completed data synchronization and the application system has switched clusters, the delete operation is performed. Deletion is an asynchronous process where underlying resources are gradually released.
Deleted: The reassignment task has completed the deletion process, and all resources and configurations have been released.
Constraints and Limitations
MirrorMaker2 uses Connectors to synchronize messages between the source and target clusters. This mechanism cannot guarantee that the data in the source and target clusters will be completely consistent. Therefore, when using reassignment tools, the following constraints and limitations should be noted:
Minor Message Duplication
Since the consumption progress data is asynchronously and intermittently synchronized, switching consumers may cause offset rollback, leading to duplicate consumption.
Message synchronization is an asynchronous operation. Due to machine anomalies or fault recovery, some data may be duplicated.
Minor Consumption Out-of-Order
If an offset rollback occurs when switching consumers, a small number of messages may be consumed repeatedly, causing out-of-order consumption for these messages.
Next Steps
After understanding the operational background and features of the AutoMQ reassignment tool, it is recommended to proceed in the following order:
Prerequisites: Identify the characteristics and business relationships of the source cluster that need reassignment, and plan reassignment batches along with preliminary resource checks. Prerequisites▸
Executing Migration: After completing the prerequisite checks, follow the reassignment plan step-by-step. Executing Migration ▸