This document presents the Kafka CLI tools for verifying automatic partition reassignment and data balancing while scaling an AutoMQ cluster. The Kafka CLI tools run via a Docker image offered by AutoMQ.
-
Create a topic with 16 partitions and distribute a balanced load.
-
As you start and stop brokers, check to see if partitions automatically reassign themselves across different brokers.
This automatic data balancing is an intrinsic feature of AutoMQ, ensuring that data is distributed automatically and evenly throughout the cluster. By observing the distribution of partitions and broker load, you can confirm whether automatic partition balancing functions as anticipated.
Prerequisites
Before conducting automated partition data rebalance tests, the following conditions must be met:
Complete the installation and deployment of the AutoMQ cluster. You can refer to the following methods for installing and deploying AutoMQ:
Additionally, the host running the test program needs to meet the following conditions:
-
Linux/Mac/Windows Subsystem for Linux
-
Docker
Experience Partition Reassignment Triggered by Cluster Node Changes.
If the previous AutoMQ cluster was deployed by following the guide Deploy Multi-Nodes Test Cluster on Docker▸, the cluster bootstrap address you would have acquired might look like “server1:9092,server2:9092,server3:9092”, and the AutoMQ cluster would be in the “automq_net” Docker network.
Please replace the bootstrap-server address below with the actual address of the cluster based on your deployment configuration.
Create Topic
View Partition Distribution
Launch Producer
Start the Consumer
Stop the Broker
Stop a server, causing its partitions to be reassigned to other nodes. After stopping, you can observe how producers and consumers recover.
After stopping, the producer logs will appear as follows:
After waiting a few seconds, you will see that production and consumption return to normal.
Review Partition Distribution Again
After the producer resumes writing, we examine the partition distribution once more and observe that all partitions are located on broker1. AutoMQ efficiently and quickly completes the reassignment of partitions and rebalancing of traffic from the stopped node.
Restart the Broker
Restart automq-server3 to trigger the automatic reassignment of partitions. After several seconds of retrying, the producer and consumer can resume operations.
At this stage, if we review the partition distribution again, we can confirm that the partitions have been automatically reassigned.
