Skip to Main Content

Example: Partition Reassignment in Seconds

This guide is designed to walk you through the verification of AutoMQ's second-level partition reassignment feature. First, create a new Kafka Topic with one partition and start a producer to write a certain amount of data into it. Then, attempt to reassign the partition between different AutoMQ brokers using the reassign command, and observe the time it takes to complete the partition reassignment.

After successfully installing AutoMQ using Cluster Deployment on Linux▸, you will receive a list of Bootstrap Server addresses similar to the following:

192.168.0.1:9092,192.168.0.2:9092

If you installed via Deploy Locally▸, the Bootstrap Server address will be:

broker1:9092,broker2:9092

In all steps, please ensure to replace the Bootstrap Server address with the one you actually obtained.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

If the download speed of the container image is slow, refer to Docker Hub Mirror Configuration▸

Create Topic​


CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Send Messages​

Run the following command to transmit a specific quantity of data:


CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Observe the Distribution of Partitions​


CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Redistribute Partitions Across Brokers​


echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json && (CMD='docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  broker1:9092,broker2:9092 --reassignment-json-file /move.json --execute"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json


Verify the Completion of the Reassignment​


echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json &&(CMD=' docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092,broker2:9092 --reassignment-json-file /move.json --verify"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json