Skip to Main Content

Example: Partition Reassignment in Seconds

This guide aims to walk you through the process of verifying the sub-second reassignment capability of AutoMQ. Initially, create a new Kafka Topic with one partition and start a producer to write a certain amount of data into it. Then, attempt to reassign the partition between different AutoMQ brokers using the reassign command, and observe the time taken to complete the reassignment.

After successfully installing AutoMQ using the Cluster Deployment on Linux▸ method, you will obtain a list of Bootstrap Server addresses similar to the following:

192.168.0.1:9092,192.168.0.2:9092

If you installed using the Deploy Locally▸ method, the Bootstrap Server address will be:

broker1:9092,broker2:9092

Throughout all the steps, please ensure to replace the Bootstrap Server address with the one you actually obtained.

Prerequisites

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

If the download speed of the container image is slow, please refer to Docker Hub Mirror Configuration▸

Create Topic


CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Send Message

Run the following command to transmit a specified amount of data:


CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Observe the Partition Distribution


CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Redistribute Partitions Across Brokers


echo '{
"partitions": [
{
"topic": "reassign-topic",
"partition": 0,
"replicas": [
2
]
}
],
"version": 1
}' > move.json && (CMD='docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092,broker2:9092 --reassignment-json-file /move.json --execute"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json


Verify the Completion of the Reassignment


echo '{
"partitions": [
{
"topic": "reassign-topic",
"partition": 0,
"replicas": [
2
]
}
],
"version": 1
}' > move.json &&(CMD=' docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092,broker2:9092 --reassignment-json-file /move.json --verify"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json