Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.automq.com/llms.txt

Use this file to discover all available pages before exploring further.

This document outlines how to deploy a multi-node AutoMQ cluster on a Linux host (utilizing object storage services from cloud vendors or S3-compatible storage software). In this development environment, users can test cluster-related features such as partition reassignment and automatic data balancing. In addition to the Linux host deployment plan, users can refer to the following documents to explore other deployment options:
Deploying AutoMQ for production workloads and optimizing its parameters can be relatively complex. You can reach out to the AutoMQ team through this form for necessary assistance and best practices.
Moreover, if you want to completely eliminate the installation and deployment effort, you can try the fully managed cloud services provided by the AutoMQ team through the link below. Currently, all cloud markets offer a free 2-week trial.

​
Prerequisites

This documentation example is intended to deploy a 5-node AutoMQ cluster, where 3 nodes run both the Controller and Broker, and the remaining 2 nodes run only the Broker. Ensure the following prerequisites:
  • Five Linux hosts, preferably network-optimized virtual machines with 4 CPUs and 16GB of RAM, for deploying the AutoMQ cluster. Make sure the system disk storage space is at least 20GB, and data volume disk space is no less than 20GB (required only for deploying Controller nodes). An example is as follows:
    RoleIPNode IDSystem VolumeMetadata Volume
    Controller + Broker
    		192.168.0.1
    		0
    		EBS 20GB
    		EBS 20GB
    Controller + Broker
    		192.168.0.2
    		1
    		EBS 20GB
    		EBS 20GB
    Controller + Broker
    		192.168.0.3
    		2
    		EBS 20GB
    		EBS 20GB
    Broker192.168.0.41000EBS 20GBNot applicable
    Broker192.168.0.51001EBS 20GBNot applicable
  • Download the binary installation package to set up AutoMQ. Refer to Software Artifact▸.
  • Create two custom-named object storage buckets, such as automq-data and automq-ops, as used in this document.
  • Create an IAM user and generate an Access Key ID and Access Key Secret for it. Then, ensure that the IAM user has full read and write permissions to the previously created object storage bucket.
  • Install jemalloc to optimize memory allocation. Refer to Getting-started.
AutoMQ relies on the Kafka KRaft component for maintaining cluster metadata; therefore, in production scenarios, each cluster requires the deployment of at least 3 nodes (running both Controller and Broker).

​
Install and Launch the AutoMQ Cluster

​
Step 1: Create a Cluster Deployment Project

AutoMQ provides the automq-cli.sh tool for managing AutoMQ clusters. By using the command automq-cli.sh cluster create [project], a cluster configuration template will be automatically created in the directory clusters/[project]/topo.yaml. 
bin/automq-cli.sh cluster create poc
An example of the execution result is as follows: 
Success create AutoMQ cluster project: poc
========================================================
Please follow the steps to deploy AutoMQ cluster:
1. Modify the cluster topology config clusters/poc/topo.yaml to fit your needs
2. Run ./bin/automq-cli.sh cluster deploy --dry-run clusters/poc , to deploy the AutoMQ cluster

​
Step 2: Edit the Cluster Configuration Template

Edit the configuration template generated in Step 1 during cluster creation. An example of the configuration template is as follows: 
global:
  clusterId: ''
  # Bucket URI Pattern: 0@s3://$bucket?region=$region&endpoint=$endpoint
  # Bucket URI Example:
  # AWS : 0@s3://xxx_bucket?region=us-east-1
  # OCI: 0@s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
  config: |
    s3.data.buckets=0@s3://xxx_bucket?region=us-east-1
    s3.ops.buckets=1@s3://xxx_bucket?region=us-east-1
    s3.wal.path=0@s3://xxx_bucket?region=us-east-1
    log.dirs=/root/kraft-logs
  envs:
    - name: KAFKA_S3_ACCESS_KEY
      value: 'xxxxx'
    - name: KAFKA_S3_SECRET_KEY
      value: 'xxxxx'
controllers:
  # The Controllers Default Are Combined Nodes Which Roles Are Controller and Broker.
  # The Default Controller Port Is 9093 and the Default Broker Port Is 9092
  - host: 192.168.0.1
    nodeId: 0
  - host: 192.168.0.2
    nodeId: 1
  - host: 192.168.0.3
    nodeId: 2
brokers:
  - host: 192.168.0.5
    nodeId: 1000
  - host: 192.168.0.6
    nodeId: 1001
  • global.clusterId: A randomly generated unique ID, which does not require modification.
  • global.config: This is the custom incremental configuration for all nodes within the cluster. You need to specify actual values for s3.data.buckets, s3.ops.buckets, and s3.wal.path. Additional configuration items can be added with line breaks. For details on configuring object storage parameters for different cloud providers, refer to Object Storage Configuration▸.
  • global.envs: These are the environment variables for the node. Replace KAFKA_S3_ACCESS_KEY and KAFKA_S3_SECRET_KEY with actual values.
  • controllers: This is a list of Controller nodes that must be updated with actual values.
  • brokers: This is a list of Broker nodes that also need to be updated with actual values.

​
Step 3: Start AutoMQ

Execute the cluster pre-check command to generate the final startup command. 
bin/automq-cli.sh cluster deploy --dry-run clusters/poc
This command will initially verify the S3 configuration for accuracy, ensure successful access to S3, and then output the startup commands for each node. An example output is as follows: 
Host: 192.168.0.1
KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'

...
To start the cluster, sequentially execute the list of commands provided in the previous step on the designated CONTROLLER or BROKER hosts. For instance, on 192.168.0.1, begin the first CONTROLLER process by executing the corresponding command from the generated startup command list for that host. 
KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'

​
Test Message Sending and Receiving

After installing and starting the AutoMQ cluster, you can test message sending, consumption, and other functionalities by executing the Kafka CLI commands available in the bin directory of the installation package.
  1. Execute the following command to run kafka-topics.sh and create a Topic.
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
  1. Execute the following command to run kafka-console-producer.sh and send test messages.
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
  1. Execute the following command to run kafka-console-consumer.sh and consume test messages.
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

​
Stop and Uninstall the AutoMQ Cluster.

After completing the tests, you can refer to the following steps to stop and uninstall the AutoMQ cluster.
  1. Run the following command on each node to terminate the process.
bin/kafka-server-stop.sh
  1. You can automatically clear data in s3-data-bucket and s3-ops-bucket by setting lifecycle rules for object storage, and then remove these buckets.
  2. Delete the created compute instances along with their respective system volumes and data volumes.
  3. Remove the IAM user set up for testing along with its Access Key ID and Access Key Secret.
Table Topic is a core feature designed by AutoMQ for modern data lake architectures. It integrates streaming data with static data lakes and addresses traditional stream-batch separation, complex ETL processes, and high costs through architectural innovation. For instructions on enabling Table Topic in the cluster, refer to Overview▸.