> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Kafka Linking best practices
> Plan and run Kafka Linking migrations with reliable endpoints, safe client cutovers, offset validation, promotion gates, and rollback boundaries.
This guide provides production best practices for migrating Apache Kafka workloads to AutoMQ with Kafka Linking. Complete the [prerequisite review](/automq-cloud/migrate-to-automq/prerequisites), then use this guide to create a runbook for each migration batch.
Kafka Linking synchronizes Kafka topic data and committed Kafka Consumer Group offsets. It does not rewrite offsets stored in Flink checkpoints, savepoints, application databases, or other external systems. Identify the offset source that each consumer actually uses before cutover.
## Plan each migration batch
Start with a non-critical but representative workload. Expand the scope only after the pilot passes. Keep dependent producers, topics, and consumers in the same batch so that a business data flow is not split across migration stages.
Record the following information before migration:
| Object | Information to confirm |
| ------------------ | --------------------------------------------------------------------------------------------------- |
| Topic | Partition count, cleanup policy, retention, maximum message size, peak throughput, and dependencies |
| Producer | Client version, idempotence setting, `transactional.id` usage, instance inventory, and owner |
| Consumer | `group.id`, client version, offset source, acceptable replay window, and owner |
| External component | Flink, Kafka Connect, Kafka Streams, Schema Registry, and application-managed offset stores |
| Security | Listener protocol, credentials, certificate lifetime, Kafka ACLs, firewalls, and security groups |
| Rollback | Steps before and after Mirror Topic promotion, stop conditions, and decision owner |
Avoid unrelated source-cluster upgrades, broker replacements, network changes, authentication changes, and topic recreation during the migration window. Limiting concurrent changes makes failures easier to attribute.
## Configure reliable source endpoints
A Kafka client uses `bootstrap.servers` to obtain cluster metadata. It then connects to the broker addresses returned through `advertised.listeners`. A successful connection to only the bootstrap address does not prove that Kafka Linking can reach the cluster.
Enter a stable bootstrap DNS name or the formal endpoint provided by the source Kafka service. Do not use Kubernetes Pod IPs, node IP and NodePort combinations, or other temporary addresses that change during Pod recreation, node replacement, or scaling.
Validate the source connection as follows:
* Use stable DNS names or the source Kafka service's formal endpoint. When you configure multiple endpoints, distribute them across failure domains.
* Confirm that the AutoMQ data plane can resolve and connect to **every** broker address and port returned in metadata.
* For a Kubernetes source cluster, use the Kafka Operator or platform-provided stable bootstrap service and stable per-broker external listeners.
* Allow both the bootstrap endpoints and all advertised broker endpoints through firewalls, security groups, routes, and network access controls.
* For TLS or mTLS, verify the trust chain, the hostname against the certificate SAN, and the certificate lifetime.
* Use a dedicated migration identity with the permissions required to read the selected topics, inspect Consumer Groups, and discover metadata. Do not rotate or delete this identity during migration.
Run a metadata check from an environment with a network path equivalent to the AutoMQ data plane. For example:
```bash theme={null}
kafka-broker-api-versions.sh \
--bootstrap-server :,: \
--command-config source-client.properties
```
The command must return the expected broker list. Then validate DNS resolution, port connectivity, and the TLS handshake for every returned endpoint. Fix `advertised.listeners` or network access before creating the Kafka Link if any broker is unreachable.
## Prepare the source and target clusters
Complete these checks before creating a Kafka Link:
* Reserve enough target capacity for migration traffic as described in [Prerequisites](/automq-cloud/migrate-to-automq/prerequisites).
* Make sure that a topic with the same name does not already exist on the target. Kafka Linking must create a Mirror Topic for the source topic.
* Keep source topic identity and partition structure stable within a migration batch. Do not delete and recreate a topic with the same name.
* Keep target data long enough to cover the migration and observation windows. The target must not expire records that a consumer still needs before cutover.
* Validate Kafka ACLs, certificates, Schema Registry, connectors, and other external dependencies on the target. Do not assume that topic data synchronization migrates every surrounding configuration or state.
* Save a baseline of topic configurations, partition ranges, Consumer Group offsets, and business traffic on both clusters.
From the time a Mirror Topic is created until it is promoted to `PROMOTED`, do not increase the partition count of either the source topic or the target Mirror Topic. The source and target partition counts must remain unchanged while Kafka Linking is in progress.
After the Kafka Link enters the linking state, monitor replication lag and errors. Start client cutover only when lag converges and no sustained authentication, network, or request errors remain.
Keep the source cluster, its network path, DNS endpoints, and migration identity available until every Mirror Topic and Consumer Group is promoted and business acceptance passes. Do not stop, scale to zero, delete, or release the source cluster early, and do not revoke Kafka Linking's access permissions.
## Cut over standard producers and consumers
This workflow applies to producers that do not use Kafka transactions. Producers with a `transactional.id` must use the [transactional producer workflow](#cut-over-transactional-producers).
1. Move standard producers to the target AutoMQ instance in batches, then recreate or restart the clients. While a Mirror Topic is `LINKING`, Kafka Linking routes these writes to the source and replicates them back to the target.
2. Check every producer deployment and send callback. Confirm that no untracked producer instance continues to write directly to the source.
3. Before moving a Consumer Group, confirm that the actual startup offset for every partition is within the target topic's readable range. See [Validate the consumer's actual offset source](#validate-the-consumers-actual-offset-source) for the offset sources used by different clients.
4. Change the Consumer connection configuration and perform a rolling deployment so that instances connect to the target AutoMQ instance in batches. Briefly running the same `group.id` on the source and target is expected during the rollout. Kafka Linking completes Consumer Group promotion after all source instances have exited.
5. Confirm that every Consumer instance connects to the target and Consumer Group promotion is complete. Validate consumption rate, lag, business results, and errors, and complete the observation window before moving the next workload.
The rolling cutover must converge with every instance connected to the target. Do not run the same `group.id` on the source and target for an extended period; this prevents Consumer Group promotion and makes consumer progress and rollback positions ambiguous.
## Cut over transactional producers
Kafka transactions cannot use the rolling proxy workflow for standard producers. Use the following order for any application that sets `transactional.id` or depends on exactly-once semantics:
1. Keep the transactional producer on the source while target replication lag converges.
2. Stop transactional producers on the source. Confirm that every open transaction has committed or aborted.
3. Compare source and target end offsets for every partition. Confirm that the target has caught up with the source.
4. Promote every Mirror Topic involved in the transactions and wait for the `PROMOTED` state.
5. Confirm that old producer instances cannot restart. Then start the transactional producers on the target.
6. Use `read_committed` consumers to verify committed records from before and after promotion, and validate the end-to-end transactional result.
Do not point a transactional producer at the target while its Mirror Topic is still `LINKING`. For transactions across multiple topics, place all involved topics in one migration batch and stop, catch up, and promote them together.
## Validate the consumer's actual offset source
The consumer startup position depends on how the client manages state:
| Offset source | Kafka Linking behavior | Pre-cutover check |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Committed Kafka Consumer Group offset | After all source instances exit, Kafka Linking synchronizes and promotes the Group offset | Confirm that the source Group committed offset for every partition is within the target readable range, and ensure that the rollout eventually removes every source instance |
| Flink checkpoint or savepoint | Kafka Linking does not rewrite Flink state | Inspect the restored offset for every partition |
| Application-managed offset | Kafka Linking does not rewrite a database, Redis, file, or business snapshot | Identify the source of truth and inspect the offset passed to `seek` |
| Kafka Connect or other component state | State can reside in internal topics or an external store | Follow the component's migration procedure instead of checking only a business Consumer Group |
For every topic-partition, verify:
```text theme={null}
target earliest offset <= client startup offset <= target end offset
```
Pause the consumer cutover if any partition fails this check. Options include continuing on the source until the checkpoint advances, changing the synchronization or retention range, or resetting client state after assessing the business impact.
For example, if a Consumer's actual startup offset for Partition X is 100, it can cut over when the target readable range is `[80, 150]`. A range of `[120, 150]` means that the historical record at offset 100 is no longer readable. A range of `[80, 90]` means that the target has not replicated through offset 100, so wait for the target to catch up.
`auto.offset.reset` applies only when the client has no valid starting offset. It does not override a Flink restore or an explicit `assign` and `seek` from an external offset store.
## Promote Mirror Topics
Promotion stops write routing and replication from the source for a topic. Pass every gate before promotion:
| Gate | Pass condition |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Standard producers | Every workload Producer instance connects to the target, and no instance still writes directly through a source-cluster endpoint |
| Transactional producers | Source instances have stopped, transactions have ended, the target has caught up, and target instances have not started |
| Consumer Groups | Source Groups have no active members, and target consumers run normally |
| External-offset consumers | Every partition startup offset is readable on the target |
| Kafka Link | Replication lag has converged, with no sustained network, authentication, or request errors |
| Business acceptance | Produce, consume, lag, error rate, and critical business results meet the acceptance criteria |
| Rollback readiness | The rollback owner, reconciliation method, and stop conditions are recorded |
Kafka Linking remains connected to the source and routes writes during `LINKING`, so source-cluster connections alone do not indicate that workload Producers have not switched. For a low-frequency topic, do not use a short zero-traffic window as the only proof that a Producer has stopped. Check the workload Producer inventory and deployment configuration, and compare source and target write requests over a window longer than the normal message interval.
Promote topics by business batch instead of promoting the entire migration at once. Validate each batch before continuing.
During Mirror Topic promotion, a small number of writes that still reach the old routing path can briefly receive `OUT_OF_ORDER_SEQUENCE_NUMBER`. For non-transactional producers with idempotence enabled and normal retry settings, Kafka clients typically reset the sequence state and retry automatically, so applications usually do not need to intervene. Monitor final send failures. Investigate client configuration, Topic state, and residual routed traffic only if the error is ultimately returned to the application or continues after promotion; compensate only records confirmed as failed according to the application's idempotency policy.
## Define rollback boundaries
Rollback semantics change after promotion:
* **Before promotion:** Standard producers can return to the source. Before moving consumers back, reconcile the source Consumer Group offset or expect records already processed on the target to be replayed.
* **After promotion:** New target records no longer replicate to the source. Changing only `bootstrap.servers` would return clients to a source cluster that is missing post-promotion records. Stop the cutover and use a data reconciliation or reverse-migration plan.
## Complete the migration
End a Kafka Linking migration only after all of these conditions are met:
* Every Mirror Topic in the Kafka Link is `PROMOTED`.
* Every Consumer Group in the Kafka Link has completed promotion.
* Production, consumption, lag, error rate, and critical business results on the target AutoMQ instance pass acceptance, including the agreed observation window.
After all conditions pass, delete the **Kafka Link itself** in AutoMQ Console. Deleting the Kafka Link marks the Kafka Linking migration as fully complete.
Do not delete the promoted Mirror Topics or Consumer Groups one by one. Deleting these resources deletes the corresponding topic or Group from the target AutoMQ instance and can cause message data or consumer offsets to be lost, or interrupt the workload. To complete the migration, delete only the Kafka Link itself.
After deleting the Kafka Link, retain the source cluster in an idle state for an observation period. Stop new workload writes, but preserve the original data and required access. Reclaim the source cluster's network, storage, compute resources, and migration identity only after the target remains stable and rollback or data reconciliation is no longer needed. See [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) for the operational steps.
## Use the migration checklist
Keep at least the following evidence for each batch:
* Stable source bootstrap endpoints and connectivity results for every advertised broker endpoint.
* Migration identity permissions, certificate lifetime, and firewall approvals.
* Topic, producer, Consumer Group, external offset source, owner, and dependency inventories.
* Partition earliest offsets, end offsets, committed Group offsets, and replication lag before and after cutover.
* The selected standard or transactional producer cutover order.
* Evidence that source Groups became `Empty`, target consumers stabilized, and business acceptance passed.
* Mirror Topic promotion approval, execution time, observation result, and rollback boundary.
* Promotion results for every Mirror Topic and Consumer Group, the Kafka Link deletion record, and approval to reclaim the source cluster.
After completing these checks, follow [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) to create the Kafka Link in AutoMQ Console and run the migration.