Deploy AutoMQ Software Via Helm Chart

This document details the process for deploying AutoMQ Software in a private Kubernetes data center using a Helm Chart. To deploy AutoMQ in a public cloud environment, refer to Overview▸.

Prerequisites

Before installing AutoMQ with a Helm Chart, ensure the following prerequisites are satisfied:

Prepare a Kubernetes Environment: Establish an available Kubernetes cluster in advance, ensuring it meets the conditions below:
1. Allocate Resources for AutoMQ: It is recommended to allocate 4 cores and 16GB of memory for each AutoMQ Pod. Deploying on a dedicated Node is advisable for stable network throughput performance.
2. Storage Plugin: If your Kubernetes is provided by a cloud vendor, it is advisable to install the storage plugin offered by the vendor to manage EBS volume resources effectively.
Prepare Object Storage Buckets: Each AutoMQ cluster requires two separate object storage buckets: one Ops Bucket for system logs and metrics data, and one Data Bucket for message data. Please refer to the object storage product documentation for guidance on creating them.
Install the Helm Chart Tool: It is recommended to install version 3.6 or higher. You can refer to the documentation for detailed instructions.

Obtain the Software Chart.

The AutoMQ Software Chart image is published and made available to the public through an Azure Container Registry (East US). You can test the pull with the following command.

helm pull oci://automq.azurecr.io/helm/automq-enterprise-chart --version 5.3.4

Install AutoMQ

AutoMQ Software offers two types of WAL storage options: EBSWAL and S3WAL. A comparison of the two storage engines is as follows; it is recommended to choose based on your needs. For detailed principles, please refer to the Technical Architecture.

EBSWAL Mode: WAL storage uses high-speed EBS volumes to deliver low-latency send performance, supported only in public cloud environments like AWS, GCP, and Azure. When using, you need to assign EBS volumes to AutoMQ’s Pods via a StorageClass.
S3WAL Mode: Deployment is relatively simple, as WAL storage writes directly to object storage, offering sub-100 ms send RT performance. It supports all public cloud environments as well as private data centers (as long as they provide S3-compatible object storage). Deployment is relatively straightforward, with no need to allocate EBS volumes.

The following sections provide a simplified deployment scenario example for various cloud vendor environments. This example scenario deploys in S3WAL mode, uses static credentials to access cloud resources, and supports accessing AutoMQ within a Kubernetes cluster via a Headless service access point or from outside the Kubernetes cluster using a LoadBalancer access point.The AutoMQ team also provides a variety of advanced feature examples, including TLS, authentication and authorization, Auto-Scaler, and more. For details, please refer to the Advanced Feature Examples ▸.

Step 1: Create Credentials and Perform Authorization.

AutoMQ clusters require access to external services such as object storage and storage volumes. Therefore, before installation, you need to create credentials for AutoMQ and complete the authorization process.

AWS
Azure
OCI

If AutoMQ is deployed in the AWS public cloud environment using AWS S3 storage, you must access the IAM product to create an authorization policy. AutoMQ must be granted permission for the following operations to access AWS S3:

- actions:
    - s3:GetLifecycleConfiguration
    - s3:PutLifecycleConfiguration
    - s3:ListBucket
    - s3:PutObject
    - s3:GetObject
    - s3:AbortMultipartUpload
    - s3:PutObjectTagging
    - s3:DeleteObject

If you deploy using the EBSWAL mode, additional authorization for the following policy is required:

- actions:
    - ec2:DescribeVolumes
    - ec2:DetachVolume
    - ec2:DescribeAvailabilityZones

After creating an IAM authorization policy, credentials can be generated using two methods.

Using IAM Subaccount Static AccessKey: In this approach, attach the authorization policy to the IAM subaccount and utilize the subaccount’s static AccessKeyId and AccessKeySecret as credentials to access AutoMQ.
Using IAM Role Dynamic Credentials: For this approach, create an IAM Role and attach the authorization policy to the Role. Dynamic credentials allow access to AutoMQ through a Pod assuming the EC2 Role in EKS.

To clarify illustrations, the following configuration file example employs a static AccessKey as credentials.

Deploying AutoMQ in the Azure public cloud environment requires using Azure Blob Storage. Authorization credentials must be configured within the Storage Account’s access control (IAM). Users can opt for either of the following permission entities to assign the system role to Blob: Storage Blob Data Owner.

Using Service Principal Static Secret: In this setup, the AccessKey corresponds to the App’s clientId, and the SecretKey corresponds to the Secret value along with the tenantId. Together, these serve as static credentials for accessing AutoMQ.
Using Managed Identity Dynamic Credentials: With this approach, in addition to having the ServiceAccount assigned the necessary roles, it is vital to associate it with the corresponding VMSS (virtual machine scale set) for AutoMQ. This way, you can access AutoMQ on AKS using the clientId of this dynamic credential.

In an OCI environment, you need to generate a Customer Secret Key for a specific OCI user to serve as static credentials. This user must be granted appropriate permissions to access Object Storage.First, create the following IAM authorization policies for the user or user group managing AutoMQ:

Allow group <your-group-name> to manage objects in compartment <your-compartment> where any {
  target.bucket.name = '<your-data-bucket>',
  target.bucket.name = '<your-ops-bucket>'
}

Allow group <your-group-name> to use buckets in compartment <your-compartment> where any {
  target.bucket.name = '<your-data-bucket>',
  target.bucket.name = '<your-ops-bucket>'
}

The manage objects permission set includes operations like creating, reading, deleting, and listing objects.
The use buckets permission set includes permissions to manage bucket lifecycle policies.

If you plan to use EBSWAL mode, OKE worker nodes require permissions to manage Block Volumes. Ensure you create a Dynamic Group for your node pool and grant it the following permissions:

Allow dynamic-group <your-node-dynamic-group> to manage block-volumes in compartment <your-compartment>
Allow dynamic-group <your-node-dynamic-group> to manage volume-attachments in compartment <your-compartment>
Allow dynamic-group <your-node-dynamic-group> to use subnets in compartment <your-compartment>
Allow dynamic-group <your-node-dynamic-group> to inspect instances in compartment <your-compartment>

After creating the credentials, record the Access Key and Secret Key, as they will be used as static credentials in the subsequent Helm Chart configuration.

Step 2: Create Storage Class

Before installing AutoMQ, you must declare a Storage Class in the Kubernetes cluster for allocating storage volumes. These storage volumes serve several purposes:

Storing AutoMQ Controller Metadata: In the AutoMQ cluster, the Controller Pod responsible for metadata management must mount the storage volumes to store KRaft metadata.
EBSWAL Mode Storage for WAL Data (Optional): If you plan to deploy using the EBSWAL mode, each Broker Pod will also require a mounted data volume for writing WAL data.

Please specify the Storage Class based on the Kubernetes storage plugin from your cloud provider or private data center, then keep a record of the Storage Class name for later parameter configuration.

AWS
Azure
GCP
OCI

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: automq-disk-eks-gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3  # EBS Volume Type
allowVolumeExpansion: true

# Example StorageClass Definition (Azure Premium SSD Example)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: automq-disk-storage-class
provisioner: disk.csi.azure.com
parameters:
  skuName: PremiumV2_LRS
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: automq-disk-gcp-pd-balanced
provisioner: pd.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: pd-balanced  # High Performance Cloud Disk.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: automq-block-volume
provisioner: blockvolume.csi.oraclecloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  # vpusPerGB determines the IOPS and throughput performance of the volume
  # 10: Provides basic performance
  # 20: Provides higher performance, recommended for production environments
  vpusPerGB: "20"
  # attachmentType determines the type of volume attachment
  # iscsi: iSCSI attachment, general purpose
  # paravirtualized: Paravirtualized attachment, offers higher performance
  attachmentType: "paravirtualized"

Step 3: Initialize the Configuration File

The configuration information for the AutoMQ Software Chart is composed of multiple parts, allowing for user customization via the values.yaml file.

Required: Set a Unique Instance IDA critical parameter you must define is global.automqInstanceId. This serves as a required unique identifier for your AutoMQ cluster.Why is it important? This ID ensures that resources are properly isolated, preventing data corruption or conflicts. Each cluster must have its own distinct automqInstanceId.

First, create an empty file named automq-values.yaml. You can copy the example configuration below and edit it.

global:
  cloudProvider:
    name: "Replace With Your True Cloud Provider Name"
    credentials: "Replace With Your True Your Credentials"
  config: |
    s3.ops.buckets=Replace With Your True Ops Bucket URL
    s3.data.buckets=Replace With Your True Data Bucket URL
    s3.wal.path=Replace With Your True WAL PATH

controller:
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"
  persistence:
    metadata:
      storageClass: "Replace With Your True StroageClass"
    wal:
      enabled: false

  annotations:

  env:
    - name: "KAFKA_JVM_PERFORMANCE_OPTS"
      value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5"
    - name: "KAFKA_OPTS"
      value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
    - name: "KAFKA_HEAP_OPTS"
      value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

broker:
  replicas: 0
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"

  persistence:
    wal:
      enabled: false

  annotations:

  env:
    - name: "KAFKA_JVM_PERFORMANCE_OPTS"
      value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5"
    - name: "KAFKA_OPTS"
      value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
    - name: "KAFKA_HEAP_OPTS"
      value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      annotations:
        <Add Cloud Provider Specific Annotations Here>

      extraPorts:
        - name: "tcp-9092"
          port: 9092
          protocol: "TCP"
          targetPort: 9092

In the configuration file generated in the previous step, certain parameters must be updated according to your actual conditions.

Modify Common Parameters.

global.cloudProvider.name This parameter specifies the deployment cloud environment. Please insert the enumerated value according to the name of the cloud provider. If it is a private data center, you’ll also need to fill it in with the enumerated value.

Deployment Environment	Parameter Enumerated Value
AWS	aws
Google Cloud	gcp
Azure	azure
OCI	noop
Alibaba Cloud	aliyun

global.cloudProvider.credentials This parameter details the public credentials used by the AutoMQ cluster to access cloud resources. The current example utilizes static credentials of the AccessKey type. To use the IAM Role method, please refer to the advanced parameter documentation for guidance on modifications.

global:
  cloudProvider:
    credentials: static://?accessKey=<your-accesskey>&secretKey=<your-secretkey>

KAFKA_S3_ACCESS_KEY and KAFKA_S3_SECRET_KEY Environment Variables The example configuration file employs static credentials of the AccessKey type. Therefore, in addition to the global.cloudProvider.credentials parameter, you need to update the Controller and Broker environment variables with the correct credentials. Refer to the credentials created in Step 1 to update the credentials in the example:

controller:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

broker:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

global.config This parameter specifies the configuration for accessing object storage using S3URL, which includes three components: s3.ops.buckets, s3.data.buckets, and s3.wal.path.

Below is an example configuration using the S3WAL mode with static credentials. If you need to use EBSWAL mode, please refer to the advanced configuration section for modification instructions.

AWS
Azure
Google Cloud
OCI
Alibaba Cloud

Fill in the Ops Bucket and Data Bucket created in the prerequisites according to your actual scenario.

config: |
  s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static
  s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static
  s3.wal.path=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static

Fill in the Ops Bucket, Data Bucket, region, and Endpoint parameters according to your actual scenario.

config: |
  s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static
  s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static
  s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static

AccessKey-type static credentials are not supported on GCP, requiring a different format. You need to fill in the Ops Bucket, Data Bucket, region, Endpoint, and GCP ServiceAccount parameters.

config: |
  s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>
  s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>
  s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>

Fill in the Ops Bucket, Data Bucket, region, and Endpoint parameters according to your actual scenario.

config: |
  s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true
  s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true
  s3.wal.path=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true

Input the Ops Bucket, Data Bucket, region, and Endpoint parameters according to the actual requirements.

config: |
  s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static
  s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static
  s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static

Set LoadBalancer Annotations to Enable External Kubernetes Cluster Access

If you need to access AutoMQ from outside the Kubernetes cluster, you need to enable externalAccess. To configure an internal LoadBalancer, you should modify the values.yaml file, specifically the externalAccess.controller.service.loadBalancerAnnotations section, and add the following annotations based on your cloud provider:

AWS
Azure
GCP
OCI

To create an internal Network Load Balancer (NLB), add the following annotations:

service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
# If you need to specify subnets, add:
# service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxxx,subnet-yyyy"

To create an internal standard Load Balancer, add the following annotations:

service.beta.kubernetes.io/azure-load-balancer-internal: "true"
# If you need to specify a subnet, add:
# service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "your-subnet-name"

To create an internal TCP/UDP Load Balancer, add the following annotations:

networking.gke.io/load-balancer-type: "Internal"
# If you need to specify a subnet, add:
# cloud.google.com/load-balancer-subnet: "your-subnet-name"

To create an internal Load Balancer, add the following annotations:

oci.oraclecloud.com/load-balancer-type: "nlb"
oci-network-load-balancer.oraclecloud.com/internal: "true"
oci-network-load-balancer.oraclecloud.com/subnet: "<Replace With Your True Subnet OCID>"

controller.persistence.metadata.storageClass Substitute this parameter with the name of the Storage Class created in step 2, which is designated for storing metadata in the AutoMQ Controller Pod.

Revise the Cluster Topology and Resource Request Parameters.

Adjust the cluster topology and resource request parameters based on the resources allocated to AutoMQ Node. The parameters that need modification are as follows: broker.replicas The AutoMQ Software Chart will start with three Controller Pods by default. These Controller Pods also provide data read and write capabilities. If users wish to horizontally scale more Brokers, they can set the broker.replicas parameter.

Default value: 0, which represents a three-node cluster without the need for additional Brokers.
Setting range: >= 0, configured as needed.

Resource Request Parameters AutoMQ Software Controller and Broker Pods need to have the Request and Limit parameters adjusted, along with the corresponding JVM HEAP settings. The configuration files mentioned earlier default to a 4Core16GB specification. Please update these parameters based on the actual allocated computing resources.

controller.resources.requests.cpu
controller.resources.requests.memory
controller.resources.limits.cpu
controller.resources.limits.memory
controller.env.[KAFKA_HEAP_OPTS]
broker.resources.requests.cpu
broker.resources.requests.memory
broker.resources.limits.cpu
broker.resources.limits.memory
broker.env.[KAFKA_HEAP_OPTS]

Step 4: Install Chart and Access the Cluster

After customizing the values.yaml configuration file to suit your deployment requirements, proceed with the installation of AutoMQ.

helm upgrade --install <release-name> oci://automq.azurecr.io/helm/automq-enterprise --version 5.3.4 -f <your-custom-values.yaml> --namespace <namespace> --create-namespace

Once the installation is complete, users can access AutoMQ within the Kubernetes cluster, either through a Headless service or a LoadBalancer.

Note: We recommend deploying an Internal LoadBalancer to prevent changes in Pod IP addresses.

Step 5: Connect and Test the Cluster

Headless Service

Locate the Headless service

kubectl get svc --namespace <namespace> -l "app.kubernetes.io/component=controller" -w

Connecting and Testing Using Kafka Clients

Use the Headless Service for your Kafka client’s --bootstrap-server option to send and receive messages. Here’s the command you can use:

./kafka-console-producer.sh \
  --bootstrap-server <release-name>-automq-enterprise-controller-0.<release-name>-automq-enterprise-controller-headless.<namespace>.svc.cluster.local:9092 \
  --topic test-topic

LoadBalancer

Find External Address

Wait for the EXTERNAL-IP to be assigned. Use the following command:

kubectl get svc automq-release-automq-enterprise-controller-loadbalancer -n automq -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

You can obtain the external IP of the LoadBalancer.

Connect and Test Using Kafka Clients

Port 9092 is used for client access.

# Replace <EXTERNAL-IP> with the Address from the Previous Step
./kafka-console-producer.sh \
  --bootstrap-server <EXTERNAL-IP>:9092 \
  --topic test-topic

Other Advanced Configurations

The deployment document above provides a basic example of deploying AutoMQ in S3WAL mode. In real-world production environments, users can choose more advanced configurations like EBSWAL and integrate Auto-Scaler support. For the full configuration file, refer to Helm Chart Values Readme▸.

Configuring the WAL Type

In the previously mentioned installation steps, S3WAL was used as an example. AutoMQ supports deployment options for both EBSWAL and S3WAL modes.

S3WAL Mode
EBSWAL Mode

In S3WAL mode, there’s no need to mount a WAL data volume, making the configuration relatively straightforward. First, configure the global.config.s3.wal.path parameter.

config: |
  s3.wal.path=0@s3://<xxx-data-bucket>?region=<region>&endpoint=<endpoint>&authType=<authType>

Then, disable controller.persistence.wal.enabled and broker.persistence.wal.enabled.

# Applying StorageClass in Controller/broker
controller:
  persistence:
    metadata:
      storageClass: "your-storage-class"
    wal:
      enabled: false
broker:
  persistence:
    wal:
      enabled: false

In EBSWAL mode, each controller and broker requires a mounted WAL data volume. Therefore, you need to set controller.persistence.wal.storageClass and broker.persistence.wal.storageClass, ensuring these values are set to the previously created Storage Class.

# Applying StorageClass in Controller/broker
controller:
  persistence:
    metadata:
      storageClass: "your-storage-class"
    wal:
      storageClass: "your-storage-class"
broker:
  persistence:
    wal:
      storageClass: "your-storage-class"

Next, configure the global.config.s3.wal.path parameter.

config: |
  s3.wal.path=0@block:///dev/waliobandwidth=131072000&iodepth=8&iops=3000&capacity=2147483648

Setting Credentials

AutoMQ supports accessing external resources using either static AccessKeys or dynamic IAM Roles. To prevent the leakage of static AccessKey configurations in production environments, it is recommended to use dynamically generated credentials provided by the cloud provider’s IAM Roles.

IAM Role Credentials
AccessKey Credentials

When using IAM Role Credentials, it is necessary to attach the authorization policy to the Role in Step 1. Then, refer to the example below to modify the Credentials configuration.

global:
  cloudProvider:
    credentials: instance://?role=<your-instance-profile>

  config: |
    s3.data.buckets=0@s3://<your-bucket>?authType=instance&role=<role-id>
    s3.ops.buckets=1@s3://<your-bucket>?authType=instance&role=<role-id>

The format for filling out credentials parameters is outlined in the following table:**

Deployment Environment	Parameter Values
AWS	instance://?role=<your-instance-profile> For Role, enter the IAM instance profile, not the Role ARN.
Google Cloud	instance://?role=<your-service-account-name> For Role, enter the name of the GCP ServiceAccount.
Azure	instance://?role=<your-managed-identity-client-id> For Role, enter the Azure Managed Identity Client ID.
Alibaba Cloud	instance://?role=<your-role-id> For Role, enter the RAM Role name of Alibaba Cloud.

When using AccessKey Credentials, you need to attach the authorization policy to the sub-account or service account in Step 1. Then modify the Credentials configuration as shown in the example below, setting authType to static.

global:
  cloudProvider:
    credentials: static://?accessKey=<your-accesskey>&secretKey=<your-secretkey>

  config: |
    s3.data.buckets=0@s3://<your-bucket>?authType=static
    s3.ops.buckets=1@s3://<your-bucket>?authType=static

After configuring static AccessKey credentials, you must also set these credentials in the environment variables of the Controller and Broker as required by your cloud provider.

controller:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

broker:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

If deploying in Azure, you need to set an additional environment variable: AZURE_TENANT_ID, which corresponds to the current tenant ID (tenantId).

Set Fine-grained Scheduling Policies

In Kubernetes, AutoMQ’s fine-grained scheduling policy is implemented using node affinities and tolerations. Users are advised to customize label matching rules based on their node types:

Tolerations

It’s recommended to add a taint to the Kubernetes node group with the key “dedicated,” operator “Equal,” value “automq,” and effect “NoSchedule.” Then, configure the corresponding toleration rules in global.tolerations to schedule Pods:

global:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "automq"
    effect: "NoSchedule"

Node Affinities

Override default values in the controller/agent configuration to align with node labels (e.g., node-type: automq-worker):

controller:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: "node-type"
          operator: In
          values: ["automq-worker"]

Set up Auto-scaling

Number of Controllers

By default, the cluster deploys 3 Controller Pods, but users can customize the number of Controller replicas.

Note: Once the cluster is deployed, adjusting the number of Controller replicas is not supported to avoid unforeseen risks.

Number of Brokers

The number of brokers is managed by the broker.replicas parameter, which allows for horizontal scaling. By default, there are 0 brokers.

Auto-scaling Configuration

By default, HPA (Horizontal Pod Autoscaler) is disabled. To activate it, two conditions must be fulfilled:

broker.replicas > 0
Enable and configure parameters in global.autoscaling.hpa:

global:
  autoscaling:
    hpa:
      enabled: true        # Enable HPA
      minReplicas: "1"     # Minimum Replicas
      maxReplicas: "3"     # Maximum Replicas
      targetCPU: "60"      # Target CPU Utilization (%)
      targetMemory: ""     # Target Memory Utilization (%) (optional)

Identity Recognition Configuration

AutoMQ allows overriding of protocol listeners and enabling secure authentication. By default, it uses the following ports:

Client to server access: 9092 (PLAINTEXT).
Internal communication between Controllers: 9093 (PLAINTEXT).
Internal communication between Brokers: 9094 (PLAINTEXT).

AutoMQ also enables secure authentication by configuring listener overrides (e.g., enabling SASL authentication) for custom ports and protocols. Allowed values include ‘PLAINTEXT’, ‘SASL_PLAINTEXT’, ‘SASL_SSL’, and ‘SSL’.

listeners:
  client:
    - containerPort: 9092
      protocol: SASL_PLAINTEXT
      name: BROKER_SASL
  controller:
    - containerPort: 9093
      protocol: SASL_PLAINTEXT
      name: CONTROLLER_SASL
  interbroker:
    - containerPort: 9094
      protocol: SASL_PLAINTEXT
      name: BROKER_SASL

Additionally, you can set a password for it, which is randomly generated by default.

sasl:
  controller:
    user: "user1"
    password: "PWDxxx"
  interbroker:
    user: "user2"
    password: "PWDxxx"
  client:
    user: "user3"
    password: "PWDxxx"

Configure Prometheus RemoteWrite Metrics Integration

AutoMQ Server supports pushing cluster metrics directly to a user-defined Prometheus instance via the Prometheus RemoteWrite protocol. This approach eliminates the need to deploy additional Prometheus scraping components in the Kubernetes cluster, simplifying the monitoring architecture. Configure the s3.telemetry.metrics.exporter.uri parameter in global.config within your values.yaml to enable this feature. Choose the appropriate configuration format based on your Prometheus endpoint’s authentication method:

No Auth
Basic Auth
Bearer Token
AWS SigV4

For Prometheus endpoints that do not require authentication:

global:
  config: |
    s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>

Using username and password for basic authentication:

global:
  config: |
    s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=basic&username=<your-prometheus-username>&password=<your-prometheus-password>

Using Bearer Token for authentication:

global:
  config: |
    s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=bearer&token=<your-prometheus-token>

Using AWS SigV4 signature authentication (suitable for Amazon Managed Prometheus and similar services):

global:
  config: |
    s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=sigv4&region=<your-prometheus-region>

Once RemoteWrite is enabled, AutoMQ will automatically push core metrics from Brokers and Controllers to the specified Prometheus endpoint. For detailed information on metrics monitoring and alerting configuration, refer to Prometheus Monitoring & Alerting.

Security and Access Control

AutoMQ supports multiple security configurations to protect your data in transit and control client access. This section covers the two primary security models for client authentication when deploying with the Helm chart: SASL_SSL and SSL (mutual TLS). These two paths are mutually exclusive. Choose the one that aligns with your organization’s security policies.

Starting with the latest Helm chart, you only need to provide one TLS secret containing ca.crt, tls.crt, and tls.key. AutoMQ reuses this PEM bundle for all listeners and internal clients (AutoBalancer/admin) so there is no need to prepare separate certificates, and hostname verification for these built-in clients is automatically disabled when mTLS is enabled.

The Helm values tls.keystorePassword and tls.truststorePassword are used only when loading JKS/PKCS12 keystores. For PEM-based deployments (the default when tls.type=PEM or certificates are auto-generated) the chart mounts the PEM files directly, so these passwords are typically left blank unless your private key itself is protected.

Path 1: Configuring SASL_SSL Authentication

This is a common security model where clients authenticate using a username and password, and the communication channel is encrypted with TLS.

Step 1: Configure `values.yaml` for SASL_SSL

You need to define a SASL_SSL listener, enable ACLs, and configure SASL users and their passwords. The server will present a TLS certificate to clients, but clients do not need their own certificate to authenticate. Example values.yaml configuration:

# 1. Define a listener for SASL_SSL clients
listeners:
  client:
    - containerPort: 9112
      protocol: SASL_SSL
      name: CLIENT_SASL_SSL
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

# 2. Reference the secret containing the server's TLS certificate
tls:
  type: PEM
  existingSecret: automq-server-tls # Secret must contain server.crt, server.key, ca.crt

# 3. Expose the SASL_SSL port externally
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      extraPorts:
        - name: "sasl-client"
          port: 9112
          protocol: "TCP"
          targetPort: 9112
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60
      annotations:
        external-dns.alpha.kubernetes.io/evaluate-target-health: "false"

# 4. Enable ACLs and define the SASL superuser
acl:
  enabled: true
  superUsers:
    - "_automq"

# 5. Define credentials for the superuser and regular client users
sasl:
  controller:
    password: "\<strong-password-for-automq\>"
    user: _automq
  interbroker:
    password: "\<strong-password-for-automq\>"
    user: _automq
  client:
    users:
      - "my-user" # A regular application user
    passwords:
      - "<password-for-my-user>"

#### (Optional) Automate Route 53 binding with external-dns

If you deploy on AWS and want Helm to publish the bootstrap DNS record automatically:

1. Install [external-dns](https://github.com/kubernetes-sigs/external-dns) in your cluster with `--source=service --provider=aws --policy=upsert-only --registry=txt`. Grant its IAM role `route53:ListHostedZones`, `route53:ListResourceRecordSets`, and `route53:ChangeResourceRecordSets`, and enable IRSA/credentials accordingly.
2. Ensure the base domain you configured under `listeners.client[].advertisedHostnames.baseDomain` exactly matches the Route 53 hosted zone (for example `automq.private`) and the `privateZoneId` fields reference the same zone ID.
3. Add/verify the following block: the controller Service keeps owning the bootstrap hostname, while the listener block tells Kafka which FQDNs to advertise.

```yaml
listeners:
  client:
    - name: CLIENT_SASL_SSL
      containerPort: 9112
      protocol: SASL_SSL
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60
```yaml

Once the controller Service obtains an NLB and renders those annotations, external-dns will upsert `automq-bootstrap.automq.private` automatically. Set `externalAccess.controller.externalDns.enabled=false` if you prefer to copy the hostname and create the record manually.

Step 2: Post-Deployment ACL Management

After deploying the cluster, you must use the superuser (_automq) to grant permissions to regular users like my-user.

Configure an Admin Client (superuser.properties): This file allows you to run admin tools by authenticating as _automq.

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="_automq" \
  password="\<strong-password-for-automq\>";
# The client needs to trust the server's certificate
ssl.truststore.certificates=/path/to/your/ca.crt

Grant Permissions: Use kafka-acls.sh with the admin configuration to grant WRITE and READ permissions to my-user.

kafka-acls.sh --bootstrap-server <your-load-balancer-dns>:9112 \
  --command-config superuser.properties \
  --add --allow-principal User:my-user --operation WRITE --topic my-topic

Step 3: Client Configuration

A regular application client (my-user) would use the following configuration. The client only needs to trust the server, which can be done by providing the CA certificate in a PEM file or a JKS truststore. Example using PEM Truststore:

# client.properties for 'my-user'
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="my-user" \
  password="<password-for-my-user>";

# Path to the PEM file containing the CA certificate
ssl.truststore.certificates=/path/to/your/ca.crt

Example using JKS Truststore:

# client.properties for 'my-user'
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="my-user" \
  password="<password-for-my-user>";

ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=your_truststore_password

Path 2: Configuring SSL (mTLS) Authentication

In this model, clients authenticate by presenting a TLS certificate that is trusted by the cluster. This is known as mutual TLS (mTLS).

Step 1: Prepare Certificates

You will need a hierarchy of certificates:

Server Certificate: For the AutoMQ brokers.
Admin Client Certificate: A certificate with a specific Common Name (e.g., CN=automq-admin) for an administrator who will be designated as a superuser.
Application Client Certificate: A unique certificate for each client application (e.g., CN=my-app).

Step 2: Configure `values.yaml` for mTLS

You need to define an SSL listener, require client authentication, and set the admin certificate’s principal as the superuser. Example values.yaml configuration:

# 1. Define a listener for mTLS clients and require client certs
listeners:
  client:
    - containerPort: 9122
      protocol: SSL
      name: CLIENT_MTLS
      # This enforces mTLS
      sslClientAuth: required
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

# 2. Reference the secret containing the server's TLS certificate
tls:
  type: PEM
  existingSecret: automq-server-tls

# 3. Expose the SSL port externally
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      extraPorts:
        - name: "tls-client"
          port: 9122
          protocol: "TCP"
          targetPort: 9122
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60

# 4. Enable ACLs and define the certificate-based superuser
acl:
  enabled: true
  # The principal is 'User:' + the full Subject of the admin certificate
  superUsers:
    - "User:CN=automq-admin"

# 5. SASL is still used for internal communication
sasl:
  # ... configuration for _automq ...

Step 3: Post-Deployment ACL Management

After deployment, use the admin certificate to grant permissions to regular application principals.

Configure an Admin Client (admin.properties): This file uses the admin certificate (CN=automq-admin) to authenticate. Example using PEM files (Recommended):

security.protocol=SSL
ssl.truststore.certificates=/path/to/your/ca.crt
ssl.keystore.key=/path/to/your/admin.key
ssl.keystore.certificate.chain=/path/to/your/admin.crt

Example using JKS files:

security.protocol=SSL
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=your_truststore_password
ssl.keystore.location=/path/to/your/admin-keystore.jks
ssl.keystore.password=your_keystore_password

Grant Permissions: Use kafka-acls.sh to grant permissions to the application principal User:CN=my-app.

kafka-acls.sh --bootstrap-server <your-load-balancer-dns>:9122 \
  --command-config admin.properties \
  --add --allow-principal "User:CN=my-app" --operation WRITE --topic my-topic

Step 4: Client Configuration

A regular application client would use its own unique certificate (CN=my-app) to connect. Example using PEM files (Recommended):

# client.properties for 'my-app'
security.protocol=SSL
ssl.truststore.certificates=/path/to/your/ca.crt
ssl.keystore.key=/path/to/your/myapp.key
ssl.keystore.certificate.chain=/path/to/your/myapp.crt

Example using JKS files:

# client.properties for 'my-app'
security.protocol=SSL
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=your_truststore_password
ssl.keystore.location=/path/to/your/myapp-keystore.jks
ssl.keystore.password=your_keystore_password

# Create a topic
./kafka-topics.sh --create --bootstrap-server <bootstrap-server> \
  --replication-factor 1 --partitions 3 --topic <your-topic> \
  --command-config client-mtls.properties

Performance Tuning

For more details on performance tuning, refer to the AutoMQ Performance Tuning Guide.

​Prerequisites

​Obtain the Software Chart.

​Install AutoMQ

​Step 1: Create Credentials and Perform Authorization.

​Step 2: Create Storage Class

​Step 3: Initialize the Configuration File

​Modify Common Parameters.

​Set LoadBalancer Annotations to Enable External Kubernetes Cluster Access

​Revise the Cluster Topology and Resource Request Parameters.

​Step 4: Install Chart and Access the Cluster

​Step 5: Connect and Test the Cluster

​Headless Service

​LoadBalancer

​Other Advanced Configurations

​Configuring the WAL Type

​Setting Credentials

​Set Fine-grained Scheduling Policies

​Tolerations

​Node Affinities

​Set up Auto-scaling

​Number of Controllers

​Number of Brokers

​Auto-scaling Configuration

​Identity Recognition Configuration

​Configure Prometheus RemoteWrite Metrics Integration

​Security and Access Control

​Path 1: Configuring SASL_SSL Authentication

​Step 1: Configure values.yaml for SASL_SSL

​Step 2: Post-Deployment ACL Management

​Step 3: Client Configuration

​Path 2: Configuring SSL (mTLS) Authentication

​Step 1: Prepare Certificates

​Step 2: Configure values.yaml for mTLS

​Step 3: Post-Deployment ACL Management

​Step 4: Client Configuration

​Performance Tuning

Prerequisites

Obtain the Software Chart.

Install AutoMQ

Step 1: Create Credentials and Perform Authorization.

Step 2: Create Storage Class

Step 3: Initialize the Configuration File

Modify Common Parameters.

Set LoadBalancer Annotations to Enable External Kubernetes Cluster Access

Revise the Cluster Topology and Resource Request Parameters.

Step 4: Install Chart and Access the Cluster

Step 5: Connect and Test the Cluster

Headless Service

LoadBalancer

Other Advanced Configurations

Configuring the WAL Type

Setting Credentials

Set Fine-grained Scheduling Policies

Tolerations

Node Affinities

Set up Auto-scaling

Number of Controllers

Number of Brokers

Auto-scaling Configuration

Identity Recognition Configuration

Configure Prometheus RemoteWrite Metrics Integration

Security and Access Control

Path 1: Configuring SASL_SSL Authentication

Step 1: Configure `values.yaml` for SASL_SSL

Step 2: Post-Deployment ACL Management

Step 3: Client Configuration

Path 2: Configuring SSL (mTLS) Authentication

Step 1: Prepare Certificates

Step 2: Configure `values.yaml` for mTLS

Step 3: Post-Deployment ACL Management

Step 4: Client Configuration

Performance Tuning