Skip to Main Content

Cluster Deployment on Linux

This article outlines the rapid deployment and startup of an AutoMQ cluster in a Public Cloud environment, featuring 3 CONTROLLER nodes and 2 BROKER nodes, and tests the core features of AutoMQ.

AutoMQ supports deployment on Private Clouds, with options to create a storage system compatible with AWS EBS and AWS S3, such as Ceph, CubeFS, or MinIO.

Prerequisites​

  • Prepare 5 hosts for deploying the AutoMQ cluster. In a Public Cloud environment, it is recommended to choose network-optimized Linux amd64 hosts with 2 cores and 16GB of memory, ensuring that the system disk has at least 10GB of storage space and the data volume disk has no less than 10GB. Configurations may be reduced for testing environments. Example as follows:

    Role
    		IP
    		Node ID
    		System Volume
    		Data Volume
    CONTROLLER
    		192.168.0.1
    		0
    		EBS 20GB
    		EBS 20GB
    CONTROLLER
    		192.168.0.2
    		1
    		EBS 20GB
    		EBS 20GB
    CONTROLLER
    		192.168.0.3
    		2
    		EBS 20GB
    		EBS 20GB
    BROKER
    		192.168.0.4
    		3
    		EBS 20GB
    		EBS 20GB
    BROKER
    		192.168.0.5
    		4
    		EBS 20GB
    		EBS 20GB

    When purchasing computing resources, it is advisable to specify the same subnet and IP address as in this example to facilitate the direct copying of command operations.

  • Download the binary installation package for installing AutoMQ. Refer to Software Artifact▸.

  • Create two custom-named object storage buckets, such as automq-data and automq-ops.

  • Create an IAM user and generate both an Access Key and Secret Key for them. Ensure that this IAM user has comprehensive read and write permissions for the previously established Object storage bucket.

It is recommended to consult the official website for more detailed guidance.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws-cn:s3:::automq-data/*",
                "arn:aws-cn:s3:::automq-ops/*"
            ]
        }
    ]
}

Install and Start the AutoMQ Cluster​

Step 1: Generate an S3 URL​

AutoMQ offers the automq-kafka-admin.sh tool for quick initialization of AutoMQ. Just input an S3 URL with the necessary access point and authentication details to start AutoMQ instantly, bypassing the steps of manual cluster ID creation or storage configuration.


bin/automq-kafka-admin.sh generate-s3-url \
--s3-access-key=xxx  \
--s3-secret-key=yyy \
--s3-region=cn-northwest-1  \
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \
--s3-data-bucket=automq-data \
--s3-ops-bucket=automq-ops

For settings such as Endpoint and Region in the command line, consult the configuration guides provided by various cloud service providers.

Output Results​

Once the command is executed, the process will automatically advance through the following phases:

  1. Check the core features of S3 using the provided accessKey and secretKey to confirm the compatibility of AutoMQ with S3.

  2. Create an s3url using the identity and access point details.

  3. Retrieve a command example to launch AutoMQ using the s3url. In this command, replace --controller-list and --broker-list with the actual CONTROLLER and BROKER details to be deployed.

The execution results are as follows:


############ Ping S3 ########################

[ OK ] Write s3 object
[ OK ] Read s3 object
[ OK ] Delete s3 object
[ OK ] Write s3 object
[ OK ] Upload s3 multipart object
[ OK ] Read s3 multipart object
[ OK ] Delete s3 object
############ String of S3url ################

Your s3url is:

s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=xxx&s3-secret-key=yyy&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA


############ Usage of S3url ################
To start AutoMQ, generate the start commandline using s3url.
bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093"  \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"

TIPS: Please replace the controller-list and broker-list with your actual IP addresses.

Step 2: Generate the Startup Command List​

Replace the --controller-list and --broker-list in the commands generated in the previous step with your host information, specifically, replace them with the IP addresses of the 3 CONTROLLER and 2 BROKER hosts mentioned in the environment preparation, using the default ports 9092 and 9093.


bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093"  \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"

Parameter Description​

Parameter Name
Required
Description
--s3-url
Yes
Generated by the bin/automq-kafka-admin.sh generate-s3-url command-line tool, containing authentication, cluster ID, etc.
--controller-list
Yes
At least one address is required, serving as the IP and port list for CONTROLLER hosts. Format: IP1:PORT1; IP2:PORT2; IP3:PORT3
--broker-list
Yes
At least one address is required, serving as the IP and port list for BROKER hosts. Format: IP1:PORT1; IP2:PORT2; IP3:PORT3
--controller-only-modeNoDetermines whether the CONTROLLER node acts solely as a CONTROLLER. The default is false, meaning the deployed CONTROLLER node also serves as a BROKER role.

Output Result​

After running the command, it will produce the necessary commands to initiate AutoMQ.


############ Start Commandline ##############
To start an AutoMQ Kafka server, please navigate to the directory where your AutoMQ tgz file is located and run the following command.

Before running the command, make sure that Java 17 is installed on your host. You can verify the Java version by executing 'java -version'.

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=1 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.2:9092,CONTROLLER://192.168.0.2:9093 --override advertised.listeners=PLAINTEXT://192.168.0.2:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=2 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.3:9092,CONTROLLER://192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.3:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=3 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.4:9092 --override advertised.listeners=PLAINTEXT://192.168.0.4:9092

bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=4 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.5:9092 --override advertised.listeners=PLAINTEXT://192.168.0.5:9092


TIPS: Start controllers first and then the brokers.

node.id is automatically assigned starting from 0 by default.

Step 3: Start AutoMQ​

To initiate the cluster, systematically execute the commands from the previous step on the designated CONTROLLER or BROKER host. For example, to launch the first CONTROLLER process at 192.168.0.1, run the first command in the generated list of startup commands.


bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092

Parameter Explanation​

When deploying the startup command, unspecified parameters will default to Apache Kafka's default configuration. For parameters that are newly introduced in AutoMQ, AutoMQ's default settings will apply. To modify these defaults, append --override key=value parameters at the end of the command.

Parameter NameMandatoryDescription
s3-urlYesGenerated by the bin/automq-kafka-admin.sh generate-s3-url command line tool, includes authentication and cluster ID information.
process.rolesYesOptions are CONTROLLER or BROKER. If a single host serves both as CONTROLLER and BROKER, the configuration value should be CONTROLLER, BROKER.
node.idYesAn integer used to uniquely identify a BROKER or CONTROLLER in a Kafka cluster, must be unique within the cluster.
controller.quorum.votersYesInformation about hosts participating in the KRAFT election, includes nodeid, ip, and port information, for example: 0@192.168.0.1:9093, 1@192.168.0.2:9093, 2@192.168.0.3:9093
listenersYesIP and port that are listened on.
advertised.listenersYesThe address provided by the BROKER for Client access.
log.dirsNoDirectory for storing KRAFT and BROKER metadata.
s3.wal.pathNoIn production environments, it is recommended to store AutoMQ WAL data on a newly mounted bare device data volume. This setup can yield enhanced performance as AutoMQ supports writing directly to bare devices, thereby reducing latency. Ensure the path is correctly configured to store WAL data.
autobalancer.controller.enableNoThe default value is false, meaning traffic self-balancing is not enabled. Once traffic self-balancing is activated, the AutoMQ auto balancer component will automatically reassign partitions to ensure overall traffic is balanced.

When activating continuous traffic self-balancing or managing Example: Self-Balancing When Cluster Nodes Change▸, it's recommended to explicitly set the parameter --override autobalancer.controller.enable=true during startup.

Background Execution​

To execute in the background, append the following code at the end of your command:


command > /dev/null 2>&1 &

Data Volume Path​

To check local data volumes, use the Linux lsblk command; unpartitioned block devices should be treated as data volumes. For instance, vdb represents an unpartitioned bare block device.


vda    253:0    0   20G  0 disk
├─vda1 253:1    0    2M  0 part
├─vda2 253:2    0  200M  0 part /boot/efi
└─vda3 253:3    0 19.8G  0 part /
vdb    253:16   0   20G  0 disk

By default, AutoMQ stores metadata and WAL data in the /tmp directory. However, it’s crucial to understand that directories like /tmp on Alibaba Cloud and Tencent Cloud typically mount on tmpfs, which is unsuitable for production use.

For optimal performance in production or formal testing environments, it is advised to adjust the configuration by setting the metadata directory log.dirs and the WAL data directory s3.wal.path (the dedicated device for data writing) to distinct locations.


bin/kafka-server-start.sh ...\
--override  s3.telemetry.metrics.exporter.type=prometheus \
--override  s3.metrics.exporter.prom.host=0.0.0.0 \
--override  s3.metrics.exporter.prom.port=9090 \
--override  log.dirs=/root/kraft-logs \
--override  s3.wal.path=/dev/vdb \
> /dev/null 2>&1 &

Please update s3.wal.path to reflect the actual name of your local raw device.

Run the Demo Program​

Once the AutoMQ cluster is operational, execute the demo program below to test its capabilities.

  1. Example: Produce & Consume Message▸

  2. Example: Simple Benchmark▸

  3. Example: Partition Reassignment in Seconds▸

  4. Example: Self-Balancing When Cluster Nodes Change▸

  5. Example: Continuous Data Self-Balancing▸

Stop and Uninstall the AutoMQ Cluster​

Upon completing your test, follow these instructions to cease operations and remove the AutoMQ cluster.

  1. Run the command listed below on each node to terminate the process.

bin/kafka-server-stop.sh

  1. Set the lifecycle rules for object storage to automatically purge data from s3-data-bucket and s3-ops-bucket, and then proceed to delete these storage buckets.

  2. Remove the created compute instance, including its associated system and data volumes.

  3. Delete the test user and their associated AccessKey and SecretId.

Related Content​

Try AutoMQ Cloud on the Cloud Marketplace▸