Example: Partition Reassignment in Seconds

This document outlines the process for using Kafka CLI tools to verify AutoMQ’s sub-second partition reassignment capability. The Kafka CLI tool is executed via a Docker image provided by AutoMQ. First, create a new Kafka topic with a single partition and initiate a producer to write a certain amount of data into it. Next, attempt to reassign the partition among different AutoMQ brokers using the reassign command. Observe the time it takes to complete the partition reassignment.

Prerequisites

Before carrying out a partition reassignment test, the following conditions must be met: Complete the installation and deployment of the AutoMQ cluster. You can refer to the following methods to install and deploy AutoMQ:

Deploy Multi-Nodes Test Cluster on Docker▸

Additionally, the host running the test program must meet the following conditions:

Linux/Mac/Windows Subsystem for Linux
Docker

Experience Partition Reassignment

If the previous AutoMQ cluster was deployed following the Deploy Multi-Nodes Test Cluster on Docker▸ guide, the obtained cluster Bootstrap address will resemble “server1:9092,server2:9092,server3:9092,” and the AutoMQ cluster is on the “automq_net” Docker network.

Please replace the bootstrap-server address below with the actual cluster’s address based on the deployment configuration.

Create Topic

docker run --network automq_net automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

Send Messages

Execute the following command to send a specified amount of data.

docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

View Partition Distribution

docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"

Reassign Partitions Between Brokers

In this step, it’s assumed that partition 0 of the test topic “reassign-topic” is currently on node 1, and the goal is to reassign this partition to node 2. Depending on the actual partition distribution during testing, you might opt to reassign to different nodes.

Create the partition reassignment plan.

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json

Execute the partition reassignment plan.

docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --execute"

3. Check if the Reassignment Is Complete.

docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --verify"

What is AutoMQ?

Getting Started

Deployment

Migration

Observability

Architecture

Table Topic

Eliminate Inter-Zone Traffics

Integrations

Configuration

Releases

Benchmarks

Reference

Example: Partition Reassignment in Seconds

Prerequisites

Experience Partition Reassignment

Create Topic

Send Messages

View Partition Distribution

Reassign Partitions Between Brokers

3. Check if the Reassignment Is Complete.

​Prerequisites

​Experience Partition Reassignment

​Create Topic

​Send Messages

​View Partition Distribution

​Reassign Partitions Between Brokers

​3. Check if the Reassignment Is Complete.

Prerequisites

Experience Partition Reassignment

Create Topic

Send Messages

View Partition Distribution

Reassign Partitions Between Brokers

3. Check if the Reassignment Is Complete.