Skip to Main Content

Example: Self-Balancing when Cluster Nodes Change

You can establish a Topic with 16 partitions and initiate a balanced load to check if partitions automatically redistribute among various Brokers when a Broker is started or stopped. This automatic reassignment is a core feature of AutoMQ, aimed at ensuring data is evenly and autonomously spread throughout the cluster. By monitoring how partitions and Broker loads are distributed, you can confirm the effectiveness of the self-balancing functionality.

After successfully deploying AutoMQ using Cluster Deployment on Linux▸, you will be provided with a series of Bootstrap Server addresses, which may look like this:

192.168.0.1:9092,192.168.0.2:9092

If your installation was through Deploy Locally▸, your Bootstrap Server address will be:

broker1:9092,broker2:9092

Make sure to substitute the Bootstrap Server address with the actual one you received at all stages of the setup.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

If you are experiencing slow download speeds for container images, consider consulting Docker Hub Mirror Configuration▸

  • Verify that the self-balancing feature is activated on your AutoMQ cluster. It is enabled by default in local deployments. For other types of deployments, you need to include the parameter --override autobalancer.controller.enable=true when launching the Controller.

Create Topic​


CMD='docker run  --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh *--partitions* 16 --create --topic self-balancing-topic --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

View Partition Distribution​


CMD='docker run  --network automq_net   automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic self-balancing-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD


Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 1        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 4        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 5        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 8        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 9        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 12        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 13        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 2        Replicas: 2        Isr: 2

Start the Producer​


CMD='docker run  --network automq_net  automqinc/automq:latest /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic self-balancing-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Start the Consumer​


CMD='docker run --network automq_net  automqinc/automq:1.0.4 /bin/bash -c "/opt/kafka/kafka/bin/kafka-consumer-perf-test.sh --topic self-balancing-topic --show-detailed-stats --timeout 300000 --messages=1024000 --reporting-interval 1000 --bootstrap-server=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Shutdown Broker​

Shut down a Broker, causing the partitions on it to be reassigned to other Brokers. After the shutdown, monitor the recovery of producers and consumers.


docker stop broker2

When stopping the broker, the following logs can be observed from the producer:


[2024-04-29 05:00:03,436] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 49732 on topic-partition self-balancing-topic-7, retrying (2147483641 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-04-29 05:00:03,438] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-7 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)

Wait a few seconds and then note that production and consumption have resumed normally.


2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-3 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 42141 on topic-partition self-balancing-topic-3, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-3 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,588] 25693 records sent, 5138.6 records/sec (5.02 MB/sec), 8.9 ms avg latency, 1246.0 ms max latency.
[2024-05-07 11:56:13,589] 25607 records sent, 5120.4 records/sec (5.00 MB/sec), 1.8 ms avg latency, 44.0 ms max latency.
[2024-05-07 11:56:18,591] 25621 records sent, 5121.1 records/sec (5.00 MB/sec), 1.6 ms avg latency, 10.0 ms max latency.

Check the Partition Distribution Again.​

After the producer resumed writing, we revisited the partition distribution and observed that all partitions were located on broker1. AutoMQ swiftly and automatically completed the reassignment of partitions from the stopped broker, as well as the self-balancing of traffic.


CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic self-balancing-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD


Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 1        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 4        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 5        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 8        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 9        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 12        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 13        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 1        Replicas: 1        Isr: 1

Restart the broker

Restarting broker2 triggers the automatic reassignment of partitions, allowing producers and consumers to resume operations after several seconds of retries.


docker start broker2

Upon rechecking the partition distribution, it is evident that the automatic reassignment has been completed.


Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 1        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 4        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 5        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 8        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 9        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 12        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 13        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 1        Replicas: 1        Isr: 1