Skip to Main Content

Prerequisites

Before migrating to AutoMQ, ensure to verify the following conditions and constraints to avoid any disruption to the production environment.

Compatibility Assessment​

Due to differences in version support, AutoMQ Cloud currently only offers Kafka Server services and does not provide managed services for other ecosystem components. Therefore, before migrating to AutoMQ, please confirm if the following features are in use and refer to the following guidelines for handling them:

  • Kafka Server: AutoMQ offers a more cost-effective and flexible Kafka Server, compatible with Apache Kafka 0.9.x to 3.4.x versions. If the currently used Apache Kafka Server version is not within this support range, please contact us for support and updates.

  • Kafka Client: AutoMQ is compatible with the original Apache Kafka Client SDK. Simply confirm that the original SDK version is between 0.9.x and 3.4.x to ensure compatibility.

  • Kafka Connector: AutoMQ currently does not provide managed Kafka Connector services. If you are using Kafka Connector, it is recommended to maintain the existing service and only replace the Kafka Server access point in the Connector configuration with the AutoMQ instance access point.

  • Kafka Schema Registry: AutoMQ currently does not provide managed Schema Registry services. If you are using Kafka Schema Registry, it is recommended to maintain the existing service and only replace the Kafka Server access point in the Registry configuration with the AutoMQ instance access point.

Apart from the differences in service components, before migrating to AutoMQ, please confirm if the following features are in use and refer to the following guidelines for handling them.

  • ACL Feature: The current version of AutoMQ does not support the ACL feature. If your application heavily relies on ACL for access control, it is crucial to evaluate the risks associated with making modifications.

  • Access Point Protocol: The current version of AutoMQ supports the PLAINTEXT protocol for internal network access. If your application uses other protocols to access Kafka, it is recommended to consult AutoMQ technical personnel to assess the risks of making modifications.

Business Scope Assessment​

When migrating from Apache Kafka® to AutoMQ in a production environment, it is generally advised to migrate in batches. This means gradually selecting Topics and Consumer Groups from the source cluster and migrating them according to business chains and scopes.

Therefore, it is recommended to follow the process below for business scope assessment:

  1. Inventory Source Cluster Resources: It is advisable to use Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster and classify them according to business systems. Clean up invalid or unclaimed Topics and Consumer Groups in advance.

It is recommended to check the Topics in the source cluster and clean up those that have not been written to or subscribed to for a long time, thus narrowing the scope of the migration.

  1. Batch Division by Business System: In a production environment, it is recommended to plan the migration batches according to business systems. Start by migrating offline and non-core system businesses, and gradually expand the migration scope to avoid unexpected risks.

  2. Coordinate Business Personnel for Reassignment:

In the process of migrating from Apache Kafka to AutoMQ, it is necessary for business personnel to cooperate in changing the access points and restarting the applications. Therefore, before executing the reassignment plan, it is essential to coordinate with the business personnel.

Once the Apache Kafka Client establishes a connection to the server, it does not disconnect from the source cluster. Therefore, even if domain names are used to access the server, the client applications still need to be restarted to connect to the new AutoMQ cluster.

Cluster Environment Preparation​

Create AutoMQ Instance​

Before executing the reassignment tasks, it is necessary to create an AutoMQ instance in advance and ensure that the AutoMQ instance is in a running state. The following considerations should be noted when creating an AutoMQ instance:

  • Network Connectivity Check: When creating an AutoMQ instance, you need to select a VPC and subnet. Please ensure that the VPC subnet where the AutoMQ instance is located can communicate with the current Apache Kafka. Otherwise, subsequent data reassignment and synchronization will not be possible.

  • Instance Specification Check: When creating an AutoMQ instance, be sure to confirm the scale of the source cluster and choose the appropriate AutoMQ instance specification based on the source cluster's peak read/write throughput. This helps avoid the risk of system failures due to insufficient capacity during the later stages of reassignment. For the definition of AutoMQ instance specifications, please refer to Billing Instructions for BYOC▸.

For detailed instance creation instructions, please refer to Experience AutoMQ for Kafka▸.

Retrieve Source Cluster Configuration​

The principle of migrating from Apache Kafka® to AutoMQ is to use a reassignment tool to sync the existing data and configurations from the source cluster to the new cluster and then switch the client configurations at an appropriate time to achieve access to the new cluster.

During the process of syncing data and configurations from the source cluster, it is necessary to retrieve the source cluster configurations. Therefore, please ensure the following configurations are prepared in advance:

  • Source Cluster Endpoints: The reassignment tool needs to access the source cluster, requiring the configuration of the source cluster's bootstrap endpoint list.

  • Source Cluster Access Protocol: The reassignment tool needs to access the source cluster, requiring confirmation of the access protocol. Currently, PLAINTEXT and SASL_PLAINTEXT protocols are supported.

  • Source Cluster Username and Password (Optional): If the source cluster requires the SASL_PLAINTEXT protocol for access, the username and password must be configured, and ensure that the username has permission to subscribe to all topics.

  • Source Cluster Network Firewall Policy (Optional): If the source cluster is configured with network firewall or security group policies, the subnet of AutoMQ must be allowed.

Next Steps​

Once the prerequisites have been reviewed and met, you can proceed with the reassignment process. It is recommended to follow the steps in the following order:

  • Implement the reassignment plan: After reviewing the prerequisites, implement the reassignment plan step by step. Executing Migration ▸