Skip to Main Content

Example: Partition Reassignment in Seconds

This guide aims to walk you through verifying AutoMQ's second-level partition reassignment functionality. First, create a new Kafka topic with one partition and start a producer to write a specified amount of data into it. Then, reassign the partition between different AutoMQ brokers using the reassign command, and observe the time it takes to complete the partition reassignment.

After successfully installing AutoMQ via Cluster Deployment on Linux▸, you will obtain a list of Bootstrap Server addresses similar to the following format:

192.168.0.1:9092,192.168.0.2:9092

If you installed it through Deploy Locally▸, the Bootstrap Server addresses will be:

broker1:9092,broker2:9092

Throughout all steps, ensure you replace the Bootstrap Server addresses with the actual addresses you obtained.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

If downloading the container image is slow, please refer to Docker Hub Mirror Configuration▸

Create Topic​


CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Send Message​

Execute the following commands to send a certain amount of data:


CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

View Partition Distribution​


CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Reassign Partitions Among Brokers​


echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json && (CMD='docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  broker1:9092,broker2:9092 --reassignment-json-file /move.json --execute"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json


Check if Reassignment Is Complete​


echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json &&(CMD=' docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092,broker2:9092 --reassignment-json-file /move.json --verify"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json