Skip to main content
With the Prerequisites▸ completed, you can proceed with the subsequent migration process. This document provides a detailed overview of the strategies and implementation procedures for migrating from Apache Kafka to AutoMQ. Before starting a production migration, review the Kafka Linking best practices and incorporate the applicable endpoint, client cutover, offset, promotion, and rollback checks into your migration runbook.

​
Migration plan

Using Kafka Linking to migrate from Apache Kafka® to AutoMQ, the following tasks should be considered:
  • Message data synchronization: Kafka stores historical data of both consumed and unconsumed messages. Migrating clusters must ensure that message data is replicated to the new cluster as needed, without losing any messages.
  • Producer switching: In addition to data synchronization, the migration process requires switching producer applications at an appropriate time so that producers connect to the target cluster to produce new messages.
  • Consumer switching: In addition to data synchronization, the migration process requires switching consumer applications at an appropriate time so that consumers connect to the target cluster and continue consuming messages from the previous consumption progress.
Refer to the following flowchart for the overall migration plan: Overall migration plan flowchart showing message data sync, producer switch, and consumer switch steps

​
Steps

After completing the Prerequisites▸, the source cluster, target cluster, and the scope of Topics and Consumer Groups to be migrated have been clearly defined. The next step is to start creating the migration task. Create a Kafka Link by clicking on Target Instance >> Kafka Links and filling in the relevant parameters as guided. The console builds the candidate lists from the source cluster’s Kafka APIs. After you enter a search keyword, the console matches resource names by containment. For example, order matches prod-order-v1. Each search displays up to 100 matching resources. The following filters also apply:
  • Source Topic: Includes Topics visible to the Kafka Linking source identity that are not identified as internal Topics. Excludes Topics marked as internal in Kafka metadata and Topic names that start with __ or ., or end with -internal or .internal.
  • Source Consumer Group: Includes Groups visible to the Kafka Linking source identity whose protocolType is consumer or empty and whose Group IDs do not match a product-internal prefix. Excludes other protocol types, such as connect used by Kafka Connect or connector reported by some implementations. It also excludes product-internal Groups whose IDs start with sys-cmp or karapace-autogenerated.
Appearing in a candidate list means only that the source identity can discover the resource. It does not guarantee successful creation. Conflicts such as a same-name Topic on the target or an existing Mirror resource are not prefiltered from the source candidate list and are still validated during creation. If an expected Topic or Consumer Group does not appear, shorten the search keyword, then check its name, Group protocol type, Group ID prefix, and source Kafka ACLs. Internal Topics and coordination Groups used by components such as Kafka Connect are outside the Kafka Linking business Topic and Consumer Group migration scope; migrate them by following the component-specific procedure. See Kafka Linking best practices for the complete checklist.
If the target instance already contains a Topic with the same name as a selected Source Topic, Kafka Linking cannot create the corresponding Mirror Topic. Resolve the naming conflict before adding that Source Topic to the Kafka Link.
Kafka Link creation form showing source cluster, target cluster, and sync configuration fields Select the target topic and Consumer Group. Topic and Consumer Group selection interface for Kafka Link synchronization After creation, you can enter the Kafka Link details to check that the specified Topic and Group have already entered the synchronization status. Once Kafka Link is created, it also supports adding new Topics and Consumer Groups. You can add the required business resources for migration as needed.
If you delete Mirror Topics or Consumer Groups in Kafka Link, the corresponding Topics and Consumer Groups will be deleted from the target cluster (AutoMQ instance). This operation cannot be undone; you will need to recreate them afterward.
Kafka Link details page showing Topic and Consumer Group synchronization status

​
Step 2: Switch producers and consumers, execute migration process

When the Kafka Link is created, users need to update the Producer and Consumer configurations, switching the Producer and Consumer from the source cluster to the target cluster. This mainly involves the following three operations:
  • Producer Cutover: Update the producer’s access parameters to point to the target cluster.
  • Consumer Cutover: Update the consumer’s access parameters to point to the target cluster.
  • Promote Mirror Topic Status: In the AutoMQ console, select the Mirror Topic for status promotion. The promotion operation essentially controls the Kafka Linking component to stop proxying Producer write traffic and replicating data from the source cluster.
Specific steps for switching the Producer and Consumer stream can be found below:

​
Phase 1: Switch the producer to the target cluster

During the migration process, first switch the Producer’s access configuration so that the Producer connects to the target cluster (AutoMQ instance). The traffic topology during the switching process is shown in the figure below: Traffic topology diagram during Phase 1: producers switching to the target AutoMQ cluster Operations steps:
  • The producer updates the access parameters in batches and restarts the application, redirecting the production traffic to the designated AutoMQ instance.
This rolling cutover applies only to producers that do not use Kafka transactions. For applications that set transactional.id or depend on exactly-once semantics, stop source transactional producers, wait for the target to catch up, promote the related Mirror Topics, and then start the producers on the target. See Kafka Linking best practices for the full procedure.
Expected Outcome:
  • Production traffic is migrated to the target instance in phases, ensuring no downtime or disruption in production traffic.
  • Consumers in the source cluster continue to consume all messages without interruption.
  • Messages in the source cluster are synchronized to AutoMQ using replication tasks.
Rollback Operation:
  • Producer rollback configuration: switch back to the source cluster.

​
Phase 2: Switch the consumer to the target cluster

The second stage involves switching the Consumer’s access configuration, allowing the Consumer to connect to the target cluster (AutoMQ instance). The traffic topology during the switch is shown in the figure below: Traffic topology diagram during Phase 2: consumers switching to the target AutoMQ cluster Operations steps:
  • Consumers update their connection parameters and use a rolling deployment so that instances gradually connect to the AutoMQ instance.
  • For standard Consumers that participate in Group management by using subscribe and group.id, instances moved to the target cannot receive partition assignments or start consuming while the target Mirror Group is LINKING. As source instances leave in batches, the remaining source instances must carry the full consumption workload. Control the rollout batch size and monitor source processing capacity and lag.
  • After every source Consumer instance exits, the data plane detects that the source Group has no active members and automatically promotes the Consumer Group. The data plane schedules registered Groups every 10 seconds. If the source Group still had active members at the previous check, the same Group is checked again after a default minimum interval of 30 seconds.
  • A brief consumption pause exists until Consumer Group promotion, target rebalance, and partition assignment complete. Wait for the Group to enter PROMOTED, then confirm that target Consumers have assignments and their consumption progress is advancing.
Before switching Consumers, the following conditions should be metFor every partition, confirm that the Consumer’s actual startup offset is within the readable range of the current Topic on the AutoMQ instance:
For example, if a Consumer’s actual startup offset for Partition X is 100, it can cut over when the target readable range is [80, 150]. A range of [120, 150] means that the historical record at offset 100 is no longer readable. A range of [80, 90] means that the target has not replicated through offset 100, so wait for the target to catch up.The actual startup offset can come from a committed Kafka Consumer Group offset, a Flink checkpoint or savepoint, or application-managed storage. See Kafka Linking best practices for the validation procedure.
Expected Outcome:
  • Production traffic remains in the same state as the previous phase, continuing to proxy back to the source cluster and synchronize to the target cluster.
  • The Consumer Group will continue to consume from the offsets of the source cluster and will not be affected.
Rollback Operation:
  • For the consumer rollback configuration, switch back to the source cluster.
Since the consumption offsets of the source cluster will not be updated automatically, it is recommended to reset the offsets before rolling back to avoid potential duplicate consumption.

​
Phase 3: Promote Mirror Topic status

Once all Producers and Consumers have successfully switched and are functioning as expected, you can elevate the status of the Mirror Topic, stop the traffic proxy, and synchronization. AutoMQ console showing the Mirror Topic promotion step Operations steps:
  • After the Consumer switch is completed, it is recommended to monitor for a certain period to ensure that Producer and Consumer applications are running as expected.
  • Once confirmed, in the AutoMQ console, click on “Promote Mirror Topic” and stop traffic proxy and replication.
Prerequisites for Promoting Mirror Topics:
  • Every standard Producer instance connects to the target AutoMQ instance, and no workload Producer still writes directly through a source-cluster endpoint. Kafka Linking itself remains connected to the source during LINKING, so source-cluster connections alone do not indicate an incomplete workload cutover.
  • For transactional Producers, source instances have stopped, transactions have ended, the target has caught up, and target transactional Producers have not started.
  • Every Consumer instance connects to the target, the source Consumer Group has no active members, Consumer Group promotion is complete, and target consumers are stable.
  • Replication lag has converged, no sustained network, authentication, or request errors remain, and the post-promotion rollback and reconciliation plan is recorded.
See Kafka Linking best practices for the complete gates and validation procedure.
Expected Outcome:
  • All read and write operations for mirror topics are centralized in the target cluster, without proxying back to the source cluster.
Rollback Operation:
  • If the message data in the target cluster exceeds that of the source cluster after promoting the Mirror Topic, you should be cautious about data inconsistencies during the rollback.
After every Mirror Topic and Consumer Group in the Kafka Link is promoted, and production, consumption, and critical business results on the target pass acceptance and the observation window, delete the Kafka Link itself. Deleting the Kafka Link marks the migration as fully complete. AutoMQ console showing the Kafka Link deletion step after migration is complete
To complete the migration, delete only the Kafka Link itself. Do not delete promoted Mirror Topics or Consumer Groups from the Kafka Link; doing so deletes the corresponding topic or Group from the target AutoMQ instance.
After deleting the Kafka Link, stop new workload writes to the source and retain it in an idle state for an observation period, preserving its data and required access. Reclaim the source cluster resources only after the target remains stable and rollback or data reconciliation is no longer needed. See Kafka Linking best practices for the full exit criteria and checklist.