Overview
AutoMQ is a next-generation Kafka distribution, redesigned from the ground up with cloud-native principles, and maintains compatibility with Apache Kafka®. This article offers detailed instructions on transitioning from Apache Kafka or other Kafka distributions to AutoMQ.
Reassignment Scenarios
AutoMQ is fully compatible with Apache Kafka and supports seamless transitions from Kafka distributions offered by other vendors. The relevant details are outlined below:
| Source Cluster | Target Cluster | Reassignment Support |
|---|---|---|
| Apache Kafka | AutoMQ Cloud | Supported |
| AWS MSK | AutoMQ Cloud | Supported |
| Confluent Platform | AutoMQ Cloud | Supported |
| Alibaba Cloud Message Queue for Kafka | AutoMQ Cloud | Supported |
| Tencent Cloud CKafka Version | AutoMQ Cloud | Supported |
If the Kafka release version you are currently using is not listed above, please contact us immediately for support and assistance.
Reassignment Tool
AutoMQ Cloud, based on Kafka MirrorMaker2, offers an out-of-the-box Kafka reassignment tool that supports reassigning from Apache Kafka or other Kafka releases to AutoMQ.
AutoMQ Cloud reassignment tool has the following capabilities:
Self-Adapting Elastic Scaling: AutoMQ Cloud manages the capacity of reassignment components through cloud provider's elastic scaling group services, enabling automatic scaling based on the throughput pressure of synchronization data to ensure rapid completion of reassignments without the need for manual configuration of concurrency and node numbers.
Support for Batched Cluster Reassignments: AutoMQ Cloud supports batched reassignments of Topics and Groups from the source cluster to the target cluster, facilitating phased implementation and validation.
Supports Synchronization of Consumption Progress: AutoMQ Cloud reassignment tool supports real-time synchronization of consumption progress from the source cluster, allowing consumers to continue consuming from the target cluster after reassignment.
Supports Synchronous Update of Topics and Groups: The AutoMQ Cloud reassignment tool facilitates real-time monitoring of dynamic updates to Topics and Groups, effectively addressing concerns about potential omissions during the reassignment process.
The technical architecture of the AutoMQ Cloud reassignment tool is outlined below:
Core Concepts
A reassignment task is defined as a set of data synchronization tasks from an external Apache Kafka® cluster (or other Kafka distribution instance) to an AutoMQ instance, complete with necessary configurations for synchronization and tracking of real-time operational status.
The reassignment tool is built upon the Kafka MirrorMaker2 framework. Each reassignment task corresponds to a series of Connectors established by MirrorMaker2:
MirrorSourceConnector: Utilized for synchronizing message data from the source cluster.
MirrorCheckpointConnector: Employed to synchronize the consumption progress data of the Consumer Group from the source cluster.
MirrorHeartbeatConnector: Engaged to monitor the operational status of the synchronization link.
Each reassignment task's Connector is allocated its own set of resources, operating in isolation from others, with the system dynamically adjusting the number of underlying Workers based on the data synchronization demands and workload.
The AutoMQ reassignment tool is free to use, yet each reassignment task requires separate virtual machine resources, thus incurring cloud resource expenses.
It is recommended to strategically segment reassignment tasks into manageable batches and sizes, and to promptly delete completed tasks to mitigate ongoing cloud resource expenses.
From inception to removal, each reassignment task progresses through the following status changes:
Creating: The initial setup of the reassignment is an asynchronous process; upon completion, the MirrorMaker2 component is set up in the background also asynchronously, during which modifications are not permitted until the setup is finalized.
Running: The reassignment task is currently active, engaged in data synchronization.
Changing: The reassignment task is undergoing configuration and status updates; since this process is asynchronous, changes must be completed before any operations can be modified.
Service Exception: The reassignment task is currently unable to function normally due to complications from its setup, operational adjustments, or underlying infrastructure issues.
Paused: The reassignment task is currently on hold, maintaining its settings but not transmitting data. This state is typically used for evaluations during the reassignment phase.
Deleting: After the reassignment task has finished transferring data and the application system has switched to the new cluster, the task is then marked for deletion. This removal is an asynchronous operation where resources are progressively freed up.
Deleted: The reassignment task is entirely complete, with all associated resources and settings fully released.
Constraints and Limitations
MirrorMaker2 employs a Connector to facilitate message synchronization between the source and destination clusters, a process that does not ensure data consistency across the clusters. Therefore, it's important to be aware of the following constraints and limitations when utilizing reassignment tools:
Minor Message Duplication
The data regarding consumption progress is synced intermittently and asynchronously, and switching consumers can cause an offset rollback, leading to duplicate consumption.
Message synchronization operates asynchronously; due to irregularities in machine performance or during fault recovery, some messages may be duplicated.
Consuming Minimal Out-of-Order Messages
When switching consumers, a position rollback can cause the re-consumption of several messages, leading to disorder.
Next Steps
After familiarizing yourself with the operational context and functionality of the AutoMQ reassignment tool, it is advised to proceed in the following sequence:
Prerequisites Review: Examine the characteristics and business connections of the source cluster that requires reassignment, and plan the reassignment phases and initial resource assessments. Prerequisites▸
Implementing the Reassignment Plan: Once the prerequisites review is complete, implement the reassignment plan step by step. Executing Migration ▸