> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Kafka Linking best practices
> Plan and run Kafka Linking migrations with reliable endpoints, safe client cutovers, offset validation, promotion gates, and rollback boundaries.
This guide provides production best practices for migrating Apache Kafka workloads to AutoMQ with Kafka Linking. Complete the [prerequisite review](/automq-cloud/migrate-to-automq/prerequisites), then use this guide to create a runbook for each migration batch.
Kafka Linking synchronizes Kafka topic data and committed Kafka Consumer Group offsets. It does not rewrite offsets stored in Flink checkpoints, savepoints, application databases, or other external systems. Identify the offset source that each consumer actually uses before cutover.
## Plan each migration batch
Start with a non-critical but representative workload. Expand the scope only after the pilot passes. Keep dependent producers, topics, and consumers in the same batch so that a business data flow is not split across migration stages.
Record the following information before migration:
| Object | Information to confirm |
| ------------------ | --------------------------------------------------------------------------------------------------- |
| Topic | Partition count, cleanup policy, retention, maximum message size, peak throughput, and dependencies |
| Producer | Client version, idempotence setting, `transactional.id` usage, instance inventory, and owner |
| Consumer | `group.id`, client version, offset source, acceptable replay window, and owner |
| External component | Flink, Kafka Connect, Kafka Streams, Schema Registry, and application-managed offset stores |
| Security | Listener protocol, credentials, certificate lifetime, Kafka ACLs, firewalls, and security groups |
| Rollback | Steps before and after Mirror Topic promotion, stop conditions, and decision owner |
Avoid unrelated source-cluster upgrades, broker replacements, network changes, authentication changes, and topic recreation during the migration window. Limiting concurrent changes makes failures easier to attribute.
## Configure reliable source endpoints
A Kafka client uses `bootstrap.servers` to obtain cluster metadata. It then connects to the broker addresses returned through `advertised.listeners`. A successful connection to only the bootstrap address does not prove that Kafka Linking can reach the cluster.
Enter a stable bootstrap DNS name or the formal endpoint provided by the source Kafka service. Do not use Kubernetes Pod IPs, node IP and NodePort combinations, or other temporary addresses that change during Pod recreation, node replacement, or scaling.
Validate the source connection as follows:
* Use stable DNS names or the source Kafka service's formal endpoint. When you configure multiple endpoints, distribute them across failure domains.
* Confirm that the AutoMQ data plane can resolve and connect to **every** broker address and port returned in metadata.
* For a Kubernetes source cluster, use the Kafka Operator or platform-provided stable bootstrap service and stable per-broker external listeners.
* Allow both the bootstrap endpoints and all advertised broker endpoints through firewalls, security groups, routes, and network access controls.
* For TLS or mTLS, verify the trust chain, the hostname against the certificate SAN, and the certificate lifetime.
* Use a dedicated migration identity with the permissions required to read the selected topics, inspect Consumer Groups, and discover metadata. Do not rotate or delete this identity during migration.
Run a metadata check from an environment with a network path equivalent to the AutoMQ data plane. For example:
```bash theme={null}
kafka-broker-api-versions.sh \
--bootstrap-server :,: \
--command-config source-client.properties
```
The command must return the expected broker list. Then validate DNS resolution, port connectivity, and the TLS handshake for every returned endpoint. Fix `advertised.listeners` or network access before creating the Kafka Link if any broker is unreachable.
## Prepare the source and target clusters
Complete these checks before creating a Kafka Link:
* Reserve enough target capacity for migration traffic as described in [Prerequisites](/automq-cloud/migrate-to-automq/prerequisites).
* Make sure that a topic with the same name does not already exist on the target. Kafka Linking must create a Mirror Topic for the source topic.
* Keep source topic identity and partition structure stable within a migration batch. Do not delete and recreate a topic with the same name.
* Keep target data long enough to cover the migration and observation windows. The target must not expire records that a consumer still needs before cutover.
* Validate Kafka ACLs, certificates, Schema Registry, connectors, and other external dependencies on the target. Do not assume that topic data synchronization migrates every surrounding configuration or state.
* Save a baseline of topic configurations, partition ranges, Consumer Group offsets, and business traffic on both clusters.
### Verify the Mirror Topic and Consumer Group candidate lists
When you create a Mirror Topic or Consumer Group, the console retrieves candidate resources through the source cluster's Kafka APIs. After you enter a search keyword, the console matches resource names by containment. For example, `order` matches `prod-order-v1`. Each search displays up to 100 matching resources. The candidate lists include only resources visible to the Kafka Linking source identity and then apply these filters:
| Resource | Product filter |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Source Topic | Excludes Topics marked as internal in Kafka metadata. It also excludes Topic names that start with `__` or `.`, or end with `-internal` or `.internal` |
| Source Consumer Group | Includes only Groups whose `protocolType` is `consumer` or empty. It excludes other protocol types, such as `connect` used by Kafka Connect or `connector` reported by some implementations. It also excludes product-internal Groups whose IDs start with `sys-cmp` or `karapace-autogenerated` |
If an item in the migration inventory does not appear, check the following in order:
1. Shorten the search keyword, confirm that the resource name matches by containment, and check whether the result falls outside the 100 items displayed per search.
2. Confirm that the name or Group ID does not match an internal-resource rule above.
3. For a Consumer Group, use a Kafka administration tool to inspect its `protocolType`. A Group with a non-Consumer protocol cannot be created as a Kafka Linking Consumer Group.
4. Confirm that the Kafka Linking source identity has the Kafka ACLs required to discover the Topic or Consumer Group. Resources hidden from that identity do not appear in the candidate lists.
5. Separately check whether the target already has a Topic with the same name and whether the resource is already a Mirror resource. These conflicts are not prefiltered from the source candidate list, but they can cause creation to fail.
Internal Topics, coordination Groups, and external state used by components such as Kafka Connect are outside the business Topic and Consumer Group migration scope supported by these candidate lists. Follow the component-specific migration procedure instead of renaming resources to bypass the product filters.
From the time a Mirror Topic is created until it is promoted to `PROMOTED`, do not increase the partition count of either the source topic or the target Mirror Topic. The source and target partition counts must remain unchanged while Kafka Linking is in progress.
After the Kafka Link enters the linking state, monitor replication lag and errors. Start client cutover only when lag converges and no sustained authentication, network, or request errors remain.
Keep the source cluster, its network path, DNS endpoints, and migration identity available until every Mirror Topic and Consumer Group is promoted and business acceptance passes. Do not stop, scale to zero, delete, or release the source cluster early, and do not revoke Kafka Linking's access permissions.
## Cut over standard producers and consumers
This workflow applies to producers that do not use Kafka transactions. Producers with a `transactional.id` must use the [transactional producer workflow](#cut-over-transactional-producers).
1. Move standard producers to the target AutoMQ instance in batches, then recreate or restart the clients. While a Mirror Topic is `LINKING`, Kafka Linking routes these writes to the source and replicates them back to the target.
2. Check every producer deployment and send callback. Confirm that no untracked producer instance continues to write directly to the source.
3. Before moving a Consumer Group, confirm that the actual startup offset for every partition is within the target topic's readable range. See [Validate consumer offset sources](#validate-consumer-offset-sources) for the offset sources used by different clients.
4. Change the Consumer connection configuration and perform a rolling deployment so that instances connect to the target AutoMQ instance in batches. For standard Consumers that participate in Group management by using `subscribe` and `group.id`, target instances cannot complete JoinGroup or receive partition assignments while the target Mirror Group is `LINKING`, so they do not start consuming. The source instances that have not yet moved continue to carry the consumption workload.
5. Control the rollout batch size and continuously monitor source Consumer capacity and lag. As source instances leave in batches, effective source consumption capacity decreases. Make sure that the remaining source instances can carry the full consumption workload at that point.
6. Confirm that every source Consumer instance has exited. After Kafka Linking detects that the source Consumer Group has no active members, it automatically promotes the target Consumer Group. A brief consumption pause exists until promotion completes.
7. Wait for the Consumer Group to enter `PROMOTED`. Confirm that target Consumers successfully join the Group, receive partition assignments, and resume consumption. Validate consumption rate, lag, business results, and errors, and complete the observation window before moving the next workload.
The rolling cutover must converge with every instance connected to the target. Do not run the same `group.id` on the source and target for an extended period. The Consumer Group cannot be automatically promoted while the source Group still has active members. A target Consumer process being started or connected to a broker does not mean that it is consuming. Treat the cutover as complete only after the Group is `PROMOTED`, target Consumers have partition assignments, and consumption progress is advancing.
### Understand Consumer Group auto-promotion and the consumption pause
Data-plane auto-promotion uses the following flow:
1. When a target Consumer attempts to join a Mirror Group that is still `LINKING`, the data plane registers the Group for auto-promotion checks. This JoinGroup attempt receives a retriable error, so the target Consumer temporarily has no partition assignment.
2. The data plane queries the source Group. It attempts auto-promotion only when the source Group does not exist, or when it has no active members and its state is `EMPTY` or `DEAD`. If source members remain, the target Group stays `LINKING` and target Consumers remain unable to join.
3. After the condition is met, Kafka Linking fetches the source Group's committed offsets, validates that they are within the readable ranges of the target topics, commits them to the target Group, and changes the Group to `PROMOTED`.
4. The target Consumer retries according to its client configuration. It resumes consumption on AutoMQ only after it joins the Group, completes the rebalance, and receives partition assignments.
The data plane schedules registered Groups every 10 seconds. After the first JoinGroup registration, a source check can normally start on the next scheduler run. If the source Group still has active members at that check, subsequent source-state checks for the same Group are eligible at a default minimum interval of 30 seconds. Therefore, consumption does not resume immediately after the last source instance exits. The workflow must still wait for the next eligible source check, offset synchronization and validation, Group promotion, and the client retry and rebalance. Plan for a consumption pause measured in tens of seconds. Network, authentication, out-of-range offsets, or promotion failures can extend this window.
## Cut over transactional producers
Kafka transactions cannot use the rolling proxy workflow for standard producers. Use the following order for any application that sets `transactional.id` or depends on exactly-once semantics:
1. Keep the transactional producer on the source while target replication lag converges.
2. Stop transactional producers on the source. Confirm that every open transaction has committed or aborted.
3. Compare source and target end offsets for every partition. Confirm that the target has caught up with the source.
4. Promote every Mirror Topic involved in the transactions and wait for the `PROMOTED` state.
5. Confirm that old producer instances cannot restart. Then start the transactional producers on the target.
6. Use `read_committed` consumers to verify committed records from before and after promotion, and validate the end-to-end transactional result.
Do not point a transactional producer at the target while its Mirror Topic is still `LINKING`. For transactions across multiple topics, place all involved topics in one migration batch and stop, catch up, and promote them together.
## Validate consumer offset sources
The consumer startup position depends on how the client manages state:
| Offset source | Kafka Linking behavior | Pre-cutover check |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Committed Kafka Consumer Group offset | After all source instances exit, Kafka Linking synchronizes and promotes the Group offset | Confirm that the source Group committed offset for every partition is within the target readable range, and ensure that the rollout eventually removes every source instance |
| Flink checkpoint or savepoint | Kafka Linking does not rewrite Flink state | Inspect the restored offset for every partition |
| Application-managed offset | Kafka Linking does not rewrite a database, Redis, file, or business snapshot | Identify the source of truth and inspect the offset passed to `seek` |
| Kafka Connect or other component state | State can reside in internal topics or an external store | Follow the component's migration procedure instead of checking only a business Consumer Group |
For every topic-partition, verify:
```text theme={null}
target earliest offset <= client startup offset <= target end offset
```
Pause the consumer cutover if any partition fails this check. Options include continuing on the source until the checkpoint advances, changing the synchronization or retention range, or resetting client state after assessing the business impact.
For example, if a Consumer's actual startup offset for Partition X is 100, it can cut over when the target readable range is `[80, 150]`. A range of `[120, 150]` means that the historical record at offset 100 is no longer readable. A range of `[80, 90]` means that the target has not replicated through offset 100, so wait for the target to catch up.
`auto.offset.reset` applies only when the client has no valid starting offset. It does not override a Flink restore or an explicit `assign` and `seek` from an external offset store.
## Promote Mirror Topics
Promotion stops write routing and replication from the source for a topic. Pass every gate before promotion:
| Gate | Pass condition |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Standard producers | Every workload Producer instance connects to the target, and no instance still writes directly through a source-cluster endpoint |
| Transactional producers | Source instances have stopped, transactions have ended, the target has caught up, and target instances have not started |
| Consumer Groups | Source Groups have no active members, and target consumers run normally |
| External-offset consumers | Every partition startup offset is readable on the target |
| Kafka Link | Replication lag has converged, with no sustained network, authentication, or request errors |
| Business acceptance | Produce, consume, lag, error rate, and critical business results meet the acceptance criteria |
| Rollback readiness | The rollback owner, reconciliation method, and stop conditions are recorded |
Kafka Linking remains connected to the source and routes writes during `LINKING`, so source-cluster connections alone do not indicate that workload Producers have not switched. For a low-frequency topic, do not use a short zero-traffic window as the only proof that a Producer has stopped. Check the workload Producer inventory and deployment configuration, and compare source and target write requests over a window longer than the normal message interval.
Promote topics by business batch instead of promoting the entire migration at once. Validate each batch before continuing.
During Mirror Topic promotion, a small number of writes that still reach the old routing path can briefly receive `OUT_OF_ORDER_SEQUENCE_NUMBER`. For non-transactional producers with idempotence enabled and normal retry settings, Kafka clients typically reset the sequence state and retry automatically, so applications usually do not need to intervene. Monitor final send failures. Investigate client configuration, Topic state, and residual routed traffic only if the error is ultimately returned to the application or continues after promotion; compensate only records confirmed as failed according to the application's idempotency policy.
## Define rollback boundaries
Rollback semantics change after promotion:
* **Before promotion:** Standard producers can return to the source. Before moving consumers back, reconcile the source Consumer Group offset or expect records already processed on the target to be replayed.
* **After promotion:** New target records no longer replicate to the source. Changing only `bootstrap.servers` would return clients to a source cluster that is missing post-promotion records. Stop the cutover and use a data reconciliation or reverse-migration plan.
## Complete the migration
End a Kafka Linking migration only after all of these conditions are met:
* Every Mirror Topic in the Kafka Link is `PROMOTED`.
* Every Consumer Group in the Kafka Link has completed promotion.
* Production, consumption, lag, error rate, and critical business results on the target AutoMQ instance pass acceptance, including the agreed observation window.
After all conditions pass, delete the **Kafka Link itself** in AutoMQ Console. Deleting the Kafka Link marks the Kafka Linking migration as fully complete.
Do not delete the promoted Mirror Topics or Consumer Groups one by one. Deleting these resources deletes the corresponding topic or Group from the target AutoMQ instance and can cause message data or consumer offsets to be lost, or interrupt the workload. To complete the migration, delete only the Kafka Link itself.
After deleting the Kafka Link, retain the source cluster in an idle state for an observation period. Stop new workload writes, but preserve the original data and required access. Reclaim the source cluster's network, storage, compute resources, and migration identity only after the target remains stable and rollback or data reconciliation is no longer needed. See [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) for the operational steps.
## Use the migration checklist
Keep at least the following evidence for each batch:
* Stable source bootstrap endpoints and connectivity results for every advertised broker endpoint.
* Migration identity permissions, certificate lifetime, and firewall approvals.
* Topic, producer, Consumer Group, external offset source, owner, and dependency inventories.
* Partition earliest offsets, end offsets, committed Group offsets, and replication lag before and after cutover.
* The selected standard or transactional producer cutover order.
* Evidence that source Groups became `Empty`, target consumers stabilized, and business acceptance passed.
* Mirror Topic promotion approval, execution time, observation result, and rollback boundary.
* Promotion results for every Mirror Topic and Consumer Group, the Kafka Link deletion record, and approval to reclaim the source cluster.
After completing these checks, follow [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) to create the Kafka Link in AutoMQ Console and run the migration.