Skip to Main Content

Prerequisites

Before migrating to AutoMQ, ensure all conditions and constraints are checked to avoid any disruptions in your production environment.

Feature Compatibility Review​

Given the version support differences, AutoMQ Cloud only provides services for Kafka Server at this time and does not yet offer managed services for other components in the ecosystem. Hence, it's crucial to verify the use of the following functionalities and adhere to these guidelines before migrating:

  • Kafka Server: AutoMQ offers a cost-effective and flexible Kafka Server, compatible with Apache Kafka versions 0.9.x ~ 3.4.x. If your current Apache Kafka Server version falls outside this supported range, please contact us for assistance and updates.

  • Kafka Client: AutoMQ supports the original Apache Kafka Client SDK; ensure your SDK version is within 0.9.x ~ 3.4.x for compatibility.

  • Kafka Connector: AutoMQ does not provide managed services for Kafka Connector yet. If you're using Kafka Connector, we recommend continuing with your existing service and simply updating the Kafka Server endpoint in the Connector to point to an AutoMQ instance.

  • Kafka Schema Registry: Currently, AutoMQ does not provide managed Schema Registry services. If you use Kafka Registry, we suggest maintaining your current service and updating the Registry’s Kafka Server endpoint to an AutoMQ instance.

In addition to these component differences, verify the following functionalities and adhere to the guidelines provided before proceeding with migration to AutoMQ.

  • ACL Features: The current version of AutoMQ does not support ACL features. If your application heavily relies on ACL for permission control, you need to assess the risks associated with any modifications.

  • Access Protocol: The current version of AutoMQ supports the PLAINTEXT protocol for internal network access. If your application uses other protocols to access Kafka, it is advisable to contact AutoMQ technical staff to assist in assessing the risks of modifications.

Business Scope Review​

For production environments transitioning from Apache Kafka to AutoMQ, it is generally recommended to migrate in phases, gradually selecting Topics and Consumer Groups from the source cluster, and migrating them according to the business links and scope.

Therefore, the following process is recommended for business scope review:

  1. Inventory of Source Cluster Resources: It is recommended to use the Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster, categorizing them by business system. Clean up any ineffective or unclaimed Topics and Consumer Groups in advance.

It is advised to check the Topics in the source cluster, cleaning up those that have long been inactive in terms of writing and subscriptions to minimize the migration scope.

  1. Batch Division by Business System: In production environments, it is recommended to plan migration batches based on business systems, starting with the migration of offline and non-core system businesses, gradually expanding the migration scope to avoid unforeseen risks.

  2. Coordinate with Business Units for Reassignment: The reassignment process from Apache Kafka® to AutoMQ requires the cooperation of business units to change access points and restart the applications. Therefore, coordinating with business units is crucial prior to implementing the reassignment plan.

Once the Apache Kafka Client connects to the server, it maintains its connection to the original cluster. Consequently, even when accessing the server via a domain name, it is still necessary to restart the client application to establish a connection with the new AutoMQ cluster.

Cluster Environment Preparation​

Create AutoMQ Instance​

Before initiating the reassignment task, it is essential to create an AutoMQ instance and verify that it is operational. Consider the following when creating an AutoMQ instance:

  • Network Connectivity Check: Select a VPC and subnet when creating an AutoMQ instance. It is essential to ensure that the VPC subnet for the AutoMQ instance and the existing Apache Kafka are mutually accessible; otherwise, data reassignment and synchronization cannot proceed.

  • Instance Specification Check: It is crucial to assess the scale of the original cluster and select an AutoMQ instance specification that aligns with the peak read-write throughput of the original cluster. This step is vital to avoid the risk of system failures due to inadequate capacity during the reassignment. For definitions of AutoMQ instance specifications, please refer to Billing Instructions for BYOC▸.

For detailed instructions on creating an instance, please refer to Experience AutoMQ for Kafka▸.

Retrieve Source Cluster Configuration​

The migration principle from Apache Kafka® to AutoMQ involves utilizing a migration tool to synchronize the data and configurations from the original cluster to the new cluster, and to timely switch the client configurations to access the new cluster.

During this process, synchronizing data and configurations from the source cluster involves retrieving the configuration of the source cluster. Therefore, it is essential to ensure that the following configuration information is ready:

  • Source Cluster Entry Point: The migration tool needs access to the source cluster, requiring the setup of a list of Bootstrap entry points for the source cluster.

  • Source Cluster Access Protocol: The migration tool must verify the access protocol for the source cluster, currently supporting PLAINTEXT and SASL_PLAINTEXT protocols.

  • Source Cluster Username and Password (optional): If access to the source cluster requires the SASL_PLAINTEXT protocol, configuring a username and password is necessary, with the stipulation that this username has permissions to subscribe to all topics.

  • Source Cluster Network Firewall Policy (optional): Should the source cluster have network firewalls or security group policies in place, allowing traffic from the subnet where AutoMq is situated becomes necessary.

Subsequent Steps​

Once the prerequisites are satisfied and established, the subsequent reassignment process can begin. It is recommended to follow this sequence:

  • Implementing the Reassignment Plan: Once the prerequisites are in place, execute the reassignment plan step by step. Executing Reassignment ▸