Skip to Main Content

Prerequisites

Before migrating to AutoMQ, make sure to verify the following prerequisites and constraints to ensure that your production environment remains unaffected.

Function Compatibility Assessment​

Due to differences in version support, AutoMQ Cloud currently only offers Kafka Server services and does not provide managed services for other ecosystem components. Therefore, before migrating to AutoMQ, confirm whether the following functionalities are in use and handle them according to the guidance below:

  • Kafka Server: AutoMQ offers a more cost-effective and flexible Kafka Server, compatible with Apache Kafka 0.9.x ~ 3.4.x versions. If your current Apache Kafka Server version is not within the supported range, please contact us for support and updates.

  • Kafka Client: AutoMQ is compatible with the existing Apache Kafka Client SDK. Simply ensure that the original SDK version is between 0.9.x and 3.4.x, and it will be compatible.

  • Kafka Connector: AutoMQ does not currently provide managed Kafka Connector services. If you are using Kafka Connector, it is recommended to continue using the existing service and just replace the Kafka Server endpoint in the Connector configuration with the AutoMQ instance endpoint.

  • Kafka Schema Registry: AutoMQ does not currently provide managed Schema Registry services. If you are using Kafka Registry, it is recommended to continue using the existing service and just replace the Kafka Server endpoint in the Registry configuration with the AutoMQ instance endpoint.

In addition to the differences in service components, confirm whether the following functionalities are in use before migrating to AutoMQ and handle them according to the guidance below.

  • ACL Functionality: The current version of AutoMQ does not support ACL functionality. If your application heavily relies on ACL for access control, it is essential to evaluate the risks involved in making modifications.

  • Access Protocol: The current version of AutoMQ supports the PLAINTEXT protocol for internal network access. If your application utilizes other protocols to access Kafka, it is recommended to contact AutoMQ technical support to help assess the modification risks.

Business Scope Sorting​

For migrating from Apache Kafka to AutoMQ in a production environment, it is generally recommended to perform the migration in batches. This involves gradually selecting Topics and Consumer Groups from the source cluster and migrating them in batches based on business chains and business scope.

Therefore, it is recommended to perform business scope sorting according to the following process:

  1. Inventory of Source Cluster Resources: It is advisable to use the Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster and classify and organize them by business system. Clean up invalid or unclaimed Topics and Consumer Groups in advance.

It is recommended to check the Topics in the source cluster and clean up Topics and Consumer Groups that have had no writes or subscriptions for a long time in advance to reduce the scope of the migration.

  1. Batch Division by Business System: In a production environment, it is recommended to plan migration batches by business system. Start by migrating offline and non-core business systems, and gradually expand the migration scope to avoid unexpected risks.

  2. Coordinate with Business Personnel for Reassignment: During the process of migrating Apache Kafka® to AutoMQ, it is necessary for business personnel to cooperate in changing access points and restarting applications. Therefore, before executing the reassignment plan, it is important to coordinate with the business personnel for their cooperation.

Apache Kafka Client does not disconnect from the source cluster after establishing a server connection. Therefore, even if domain names are used to access the server, it is still necessary to restart the client application to connect to the new AutoMQ cluster.

Cluster Environment Preparation​

Create AutoMQ Instance​

Before executing the reassignment tasks, an AutoMQ instance needs to be created in advance, and it must be ensured that the AutoMQ instance is in a running state. When creating an AutoMQ instance, please pay attention to the following considerations:

  • Network Connectivity Check: When creating an AutoMQ instance, it is necessary to select a VPC and subnet. Please ensure that the VPC subnet where the AutoMQ instance is located can communicate with the current Apache Kafka. Otherwise, data reassignment and synchronization cannot be executed subsequently.

  • Instance Specification Check: When creating an AutoMQ instance, it is essential to confirm the scale of the source cluster and choose the appropriate AutoMQ instance specification according to the peak read/write throughput of the source cluster. This helps avoid risks of system failures due to insufficient capacity during the latter stages of reassignment. For definitions of AutoMQ instance specifications, please refer to Billing Instructions for BYOC▸.

For detailed instructions on instance creation, please refer to Experience AutoMQ for Kafka▸.

Retrieve Source Cluster Configuration​

The principle of migrating from Apache Kafka® to AutoMQ is to use migration tools to synchronize existing data and configurations from the source cluster to the new cluster, then switch client configurations at the appropriate time to enable access to the new cluster.

In the process described above, synchronizing data and configurations from the source cluster requires obtaining the source cluster's configuration. Therefore, it is necessary to confirm in advance whether the following configuration information is prepared:

  • Source Cluster Access Points: The migration tool needs to access the source cluster, requiring the configuration of the source cluster's Bootstrap access point list.

  • Source Cluster Access Protocol: The migration tool needs to access the source cluster, requiring confirmation of the access protocol for the source cluster. Currently, PLAINTEXT and SASL_PLAINTEXT protocols are supported.

  • Source Cluster Username and Password (Optional): If the source cluster requires access using the SASL_PLAINTEXT protocol, it is necessary to configure the username and password and ensure that the username has permission to subscribe to all topics.

  • Source Cluster Network Firewall Policy (Optional): If the source cluster is configured with network firewall or security group policies, the subnet of AutoMQ needs to be allowed.

Next Steps​

Once the prerequisites are sorted out and met, you can proceed with the subsequent reassignment process. It is recommended to handle it in the following order:

  • Execute the Reassignment Plan: After completing the prerequisites, follow the reassignment plan step by step. Executing Migration ▸