# Deploy to AWS EKS
Source: https://docs.automq.com/automq-cloud/deploy-automq-on-kubernetes/deploy-to-aws-eks

AutoMQ offers full Kafka compatibility and a cloud-native architecture based on AWS EKS, providing an efficient and scalable message queue service.

Refer to [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), AutoMQ supports deployment on Kubernetes. This article outlines the installation process for deploying AutoMQ on the AWS EKS platform.

<Info>
  Throughout the product's **full lifecycle**, AutoMQ will maintain deep and continuous integration with Amazon Elastic Kubernetes Service (EKS) and EKS Anywhere (EKS-A).

  To uphold this commitment, we ensure:

  **Continuous version compatibility:** We guarantee that each new version of AutoMQ will be compatible with both current and future versions of EKS and EKS-A. Additionally, we proactively adapt to updates on the AWS platform, offering users a forward-compatible experience.

  **Timely response and resolution:** We have established a clear response mechanism to quickly identify and resolve any configuration, security, and non-security issues reported by customers or AWS.
</Info>

## Prepare EKS Cluster and Compute Resources

If you expect to run the AutoMQ cluster on Kubernetes, you need to prepare an EKS cluster for AutoMQ to use. If you expect to deploy the AutoMQ cluster on EC2, you do not need to prepare an EKS cluster.

### Step 1: Create the Required IAM Roles

Before creating the EKS cluster and compute resources, create a dedicated IAM Role. This Role is used for the EKS cluster's own service authorization.

1. Access the IAM console and click Create Role.
2. **Trusted entity type**: Select **AWS Service**.
3. **Service Use case**: Select **EKS-Cluster**.
4. Click Next, enter a custom role name, and create the IAM Role.

If you use EKS managed node groups, also follow [Amazon EKS node IAM role](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) to create the node group IAM Role. If you use Karpenter, prepare the node role referenced by the EC2NodeClass.

### Step 2: Create EKS Cluster

1. Log in to the AWS [EKS console](https://console.aws.amazon.com/eks/home), click **Create cluster**, and select the custom creation mode.
2. **Configure basic cluster information**:
   * Bind the EKS cluster IAM Role created in the previous step.
   * Select an EKS-supported Kubernetes version that meets the AutoMQ release requirements.
3. **Configure VPC network**:
   * Select the VPC you have prepared for AutoMQ; this VPC should be consistent with the VPC where the application is located.
   * Select the private subnets for deploying the cluster.
   * It is recommended to select the default security group for the VPC.
4. Keep other default configurations and create the EKS cluster.
5. **Authorize console access**.
   To allow AutoMQ Console to manage the EKS cluster, authorize the IAM Role used by the console.
   1. Go to the **Access** menu of the EKS cluster and click **Create Access Entry**.
   2. Select the IAM Role used by AutoMQ Console.
   3. Set **Type** to **Standard**.
   4. Click Next, and on the **Add access policy** page, select the **AmazonEKSClusterAdminPolicy** policy.
   5. Select **Cluster** for the **Scope** and click Add.
   6. Complete the creation.
6. **Set security group rules to allow access to Pods and the EKS API Server service from outside the EKS cluster.**

   <Info>
     AutoMQ Console needs to call the EKS cluster API Server, so ensure that the network where the console runs can access the API Server on port 443. In addition, AutoMQ Brokers expose service ports. If components outside the cluster, including AutoMQ Console and Producer/Consumer applications, need to access AutoMQ, add inbound rules to the EKS cluster security group.

     * **Ports**: `443`, `9090`, `9092`, `9102`, `9122`, `9112`, `8083`.
     * **Source**: Set the CIDR ranges or security groups according to where AutoMQ Console and your applications run.
   </Info>

### Step 3: Prepare EKS Compute Capacity

Prepare stable capacity for Kubernetes system components and dedicated capacity for AutoMQ workloads. Dedicated AutoMQ capacity can use EKS managed node groups or Karpenter.

#### 1. Create a System Node Group

1. Go to the EKS cluster details page, click the **Compute** menu, and **Create node group**.
2. **IAM Role**: Select the node group IAM Role you created in **Step 1**.
3. **Instance Configuration**: Select `t3.medium` or equivalent instances, with a recommended quantity of 2-3.
4. **Subnet**: Select all private subnets required for EKS deployment.
5. Complete the node group creation.

#### 2. Prepare Dedicated AutoMQ Capacity

Choose one of the following capacity types.

**EKS managed node group**

1. Go to the **Compute** menu and **Create node group**.
2. **IAM Role**: Select the node group IAM Role created in **Step 1**.
3. **Custom label**: Add `node-type=automq`.
4. **Taints**: Add key `dedicated`, value `automq`, and effect `NO_SCHEDULE`.
5. **Instance Configuration**: Select an instance type supported by AutoMQ Console.
6. **Subnet**: Select private subnets in the one or three Availability Zones planned for the AutoMQ instance.
7. **Scaling Configuration**: Set a maximum size that can satisfy the planned AutoMQ node count. A starting desired size of 3 nodes is recommended.
8. Complete the node group creation.

**Karpenter NodePool**

1. Add the custom label `node-type=automq` and taint `dedicated=automq:NoSchedule` under `spec.template`.
2. Ensure that `spec.template.spec.requirements` permits the instance types and Availability Zones selected in AutoMQ Console.
3. Set CPU and memory limits that can satisfy the planned AutoMQ node count.

For Karpenter installation and EC2NodeClass configuration, see the [Karpenter getting started guide](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/). Managed node groups and Karpenter NodePools use the same [SchedulingSpec](/automq-cloud/deploy-automq-on-kubernetes/overview#schedulingspec).

#### 3. Prepare Kafka Connector Capacity (Optional)

If you plan to use the managed Kafka Connect service provided by AutoMQ, prepare node resources for Connector. Create a separate node pool by referring to the system node group steps, or scale up the system node group to ensure sufficient compute resources.

### Step 4: Initialize Local Tools and Configure EKS Plugins

The newly created EKS cluster needs to have some necessary system plugins installed to work with AutoMQ.

#### 1. Initialize Local Tools

1. Install and configure [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Helm](https://helm.sh/docs/intro/install/) in your local environment.
2. Execute the following command to generate the KubeConfig configuration file so that `kubectl` can connect to your EKS cluster.
   ```bash theme={null}
   # Replace <region> and <cluster-name> with your actual values
   aws eks update-kubeconfig --region <region> --name <cluster-name>
   ```

#### 2. Configure Cluster Autoscaler for managed node groups (optional)

Use Cluster Autoscaler when EKS managed node groups need to scale on demand. Skip this step when you use Karpenter, which provisions nodes through its own controllers.

1. Download the AutoScaler configuration file from the [download link](https://download.automq.com/eks-config/cluster-autoscaler-autodiscover.yaml).
2. Modify the configuration file, replacing `<YOUR CLUSTER NAME>` with your EKS cluster name.
3. Execute the installation command:
   ```bash theme={null}
   kubectl apply -f cluster-autoscaler-autodiscover.yaml
   ```

#### 3. Prepare IAM authorization for EKS add-ons

The Amazon EBS CSI Driver and AWS Load Balancer Controller need IAM permissions to operate AWS resources. The example commands below use IRSA and `eksctl`, so create an IAM OIDC provider for the EKS cluster first. For details, see the AWS documentation [Create an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).

This OIDC provider is used by the EKS add-ons in the following steps. AutoMQ workload Pods can use either EKS Pod Identity or IRSA. Prepare that workload IAM Role and ServiceAccount in [Step 5](#step-5-prepare-the-automq-workload-iam-role-and-serviceaccount).

#### 4. Configure Amazon EBS CSI Driver

The EKS cluster needs this plugin to manage storage volumes.

1. Refer to the official AWS documentation "[Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html)" to create a dedicated IAM Role for the CSI Driver after creating the OIDC Provider above.
2. After completing the IAM Role and Trust Policy configuration, go to the **Add-ons** tab of the EKS cluster console, add the **Amazon EBS CSI Driver**, and select the IAM Role created in the previous step.

#### 5. Install AWS Load Balancer Controller

This plugin is used to manage Load Balancers and provide an external access entry for AutoMQ to the EKS cluster. To install the Load Balancer Controller, you can refer to the [official documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/#option-b-attach-iam-policies-to-nodes).

1. Execute the following command to generate the IAM Policy.
   ```bash theme={null}
   curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.10.0/docs/install/iam_policy.json
   aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam-policy.json
   ```
2. Create an IAM Role and attach the Policy created in the previous step to this Role.
   ```bash theme={null}
   eksctl create iamserviceaccount \
     --cluster=<cluster-name> \
     --namespace=kube-system \
     --name=aws-load-balancer-controller \
     --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
     --override-existing-serviceaccounts \
     --region <region-code> \
     --approve
   ```
3. Add the Helm repository and install the CRD:
   ```bash theme={null}
   helm repo add eks https://aws.github.io/eks-charts
   kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
   ```
4. Execute the installation command, paying attention to replace `<eks-cluster-id>` with your cluster name, `<Your RegionID>` with your region ID, and `<Your VPCID>` with your VPC ID:
   ```bash theme={null}
   helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<eks-cluster-id> --set region=<Your RegionID> --set vpcId=<Your VPCID>
   ```

### Step 5: Prepare the AutoMQ Workload IAM Role and ServiceAccount

AutoMQ data plane Pods need AWS permissions to access cloud resources such as S3. Before you create an AutoMQ instance, prepare an IAM Role and a Kubernetes ServiceAccount for the AutoMQ workload.

You can bind the ServiceAccount to the IAM Role in either of these ways:

| Method           | When to use it                                                                                                                                                                                   | Trust policy difference                                                                                              | ServiceAccount binding                                                                               |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| EKS Pod Identity | Recommended for new EKS clusters when you want EKS to manage the association between the role and the ServiceAccount.                                                                            | The role trusts the AWS service principal `pods.eks.amazonaws.com` and allows `sts:AssumeRole` and `sts:TagSession`. | Create an EKS Pod Identity association between the cluster, namespace, ServiceAccount, and IAM Role. |
| IRSA             | Use this when your cluster already uses IAM roles for service accounts, when you need compatibility with existing IRSA automation, or when Pod Identity is not supported for your workload type. | The role trusts the cluster OIDC provider as a federated principal and allows `sts:AssumeRoleWithWebIdentity`.       | Annotate the Kubernetes ServiceAccount with `eks.amazonaws.com/role-arn=<role-arn>`.                 |

<Tip>
  The IAM permission policy attached to the role can be the same for both methods. The main difference is the IAM Role trust policy and how the ServiceAccount is associated with the role. See [Appendix: Configure EKS Pod Identity for AutoMQ workloads](#appendix-configure-eks-pod-identity-for-automq-workloads) and [Appendix: Configure IRSA for AutoMQ workloads](#appendix-configure-irsa-for-automq-workloads) for the detailed setup.
</Tip>

## Install the Environment Console

After preparing the EKS cluster and compute resources, choose either [Deploy AutoMQ Console on EC2▸](/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws#deploy-automq-console-on-ec2) or [Deploy AutoMQ Console on EKS▸](/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws#deploy-automq-console-on-eks) based on your operational requirements. The console deployment mode does not restrict running the AutoMQ data plane on EKS.

## Create an AutoMQ instance on Kubernetes

Sign in to AutoMQ Console, create an instance, and configure these fields:

1. **Deployment type**: Select **Kubernetes**.
2. **VPC and Availability Zones**: Select the VPC used by EKS and either one or three workload Availability Zones.
3. **Kubernetes cluster**: Select the EKS cluster.
4. **Load balancer subnets**: Select NLB subnets that cover the workload Availability Zones.
5. **Instance type**: Select a type permitted by the managed node group or Karpenter NodePool.
6. **Scheduling YAML**: If the dedicated compute pool has labels or taints, enter the corresponding [SchedulingSpec](/automq-cloud/deploy-automq-on-kubernetes/overview#schedulingspec).
7. Preview the configuration and create the instance.

AutoMQ adds the selected instance type and Availability Zones to the effective Kubernetes scheduling conditions. Don't repeat those two conditions in the Scheduling YAML.

## Appendix: Configure EKS Pod Identity for AutoMQ workloads

For the complete AWS procedure, see [Learn how EKS Pod Identity grants pods access to AWS services](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).

Before you start, make sure the EKS Pod Identity Agent is installed in the cluster. You do not need this step if EKS Auto Mode is enabled. The IAM Role must belong to the same AWS account as the EKS cluster for a direct Pod Identity association. Also make sure the AWS principal that creates the association has `iam:PassRole` permission for the IAM Role.

1. Create an IAM Role and attach the IAM permission policy required by AutoMQ.

2. Configure the role trust policy for EKS Pod Identity:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
         "Effect": "Allow",
         "Principal": {
           "Service": "pods.eks.amazonaws.com"
         },
         "Action": [
           "sts:AssumeRole",
           "sts:TagSession"
         ],
         "Condition": {
           "StringEquals": {
             "aws:RequestTag/kubernetes-namespace": "<automq-namespace>",
             "aws:RequestTag/kubernetes-service-account": "<automq-service-account>"
           }
         }
       }
     ]
   }
   ```

3. Create the namespace and ServiceAccount:

   ```bash theme={null}
   kubectl create namespace <automq-namespace>
   kubectl create serviceaccount <automq-service-account> -n <automq-namespace>
   ```

4. Create the Pod Identity association:

   ```bash theme={null}
   aws eks create-pod-identity-association \
     --cluster-name <cluster-name> \
     --namespace <automq-namespace> \
     --service-account <automq-service-account> \
     --role-arn arn:aws:iam::<aws-account-id>:role/<automq-role-name> \
     --region <region-code>
   ```

5. Before creating the AutoMQ instance, confirm that the association is active and the ServiceAccount exists in the target namespace.

## Appendix: Configure IRSA for AutoMQ workloads

For the complete AWS procedure, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).

1. Create an IAM OIDC provider for the EKS cluster. You only need to do this once for each cluster.

   ```bash theme={null}
   eksctl utils associate-iam-oidc-provider \
     --cluster <cluster-name> \
     --region <region-code> \
     --approve
   ```

2. Create an IAM Role and attach the IAM permission policy required by AutoMQ.

3. Configure the role trust policy for IRSA:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/oidc.eks.<region-code>.amazonaws.com/id/<oidc-provider-id>"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.<region-code>.amazonaws.com/id/<oidc-provider-id>:aud": "sts.amazonaws.com",
             "oidc.eks.<region-code>.amazonaws.com/id/<oidc-provider-id>:sub": "system:serviceaccount:<automq-namespace>:<automq-service-account>"
           }
         }
       }
     ]
   }
   ```

4. Create the namespace and ServiceAccount, and annotate the ServiceAccount with the IAM Role ARN:

   ```bash theme={null}
   kubectl create namespace <automq-namespace>
   kubectl create serviceaccount <automq-service-account> -n <automq-namespace>
   kubectl annotate serviceaccount <automq-service-account> \
     -n <automq-namespace> \
     eks.amazonaws.com/role-arn=arn:aws:iam::<aws-account-id>:role/<automq-role-name>
   ```

5. Before creating the AutoMQ instance, confirm that the annotated ServiceAccount exists in the target namespace.


# Deploy to Azure AKS
Source: https://docs.automq.com/automq-cloud/deploy-automq-on-kubernetes/deploy-to-azure-aks

Deploy AutoMQ on Azure AKS to leverage cloud-native scalability and API compatibility with Kafka, ensuring cost efficiency and seamless Kubernetes integration.

Refer to [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), AutoMQ supports deployment on Kubernetes. This document outlines the installation process for deploying AutoMQ on the Azure AKS platform.

<Info>
  The terms AutoMQ product service provider, AutoMQ service provider, and AutoMQ in this document specifically refer to AutoMQ HK Limited and its subsidiaries.
</Info>

## Choose Installation Method

AutoMQ provides the following two methods to deploy the environment to Azure AKS. Choose the method that best suits your needs:

| Installation Method                                                                              | Description                                                                                                                                                                     |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Option 1: One-Click Installation via Terraform](#option-1-one-click-installation-via-terraform) | Use the Terraform example project provided by AutoMQ to automate the environment deployment. The process is streamlined and efficient. Recommended as the preferred approach.   |
| [Option 2: Manual Installation via Azure Portal](#option-2-manual-installation-via-azure-portal) | Manually create and configure the required cloud resources step by step through the Azure portal. Suitable for scenarios that require fine-grained control over each operation. |

## Option 1: One-Click Installation via Terraform

AutoMQ provides a one-stop Terraform example project that covers all operations including AKS cluster creation, Managed Identity configuration, network setup, Storage Account creation, and permission grants. By running Terraform, you can automate the entire environment deployment without manual step-by-step operations.

<Tip>
  It is recommended to use the Terraform method for deployment. This approach significantly reduces manual steps and lowers the risk of configuration errors.
</Tip>

### Step 1: Run Terraform to Create Cloud Resources and Deploy the Console

Go to the [AutoMQ Terraform Example Project (Azure)](https://github.com/AutoMQ/automq-labs/tree/main/byoc-examples/setup/azure/azure-automq-env), configure the parameters according to the README instructions, and execute Terraform. Terraform will automatically complete the following operations:

* Create the cloud resources required by the AutoMQ console (Storage Account, Private DNS Zone, Managed Identity and permission configurations, etc.).
* Create the AKS cluster and dedicated node pool.
* Deploy the AutoMQ environment console.

After execution, Terraform will output the key information required for subsequent steps. Please record the output values.

### Step 2: Log in to the Console and Complete Initialization

After Terraform execution is complete, log in to the AutoMQ BYOC console and create a deployment configuration to complete initialization.

1. Log in to the console, enter the cluster name and other configurations, then click next.

   1. Deployment Type: Select Kubernetes.

   2. Kubernetes Cluster: Enter the cluster name for the AKS cluster.

   3. AKS Resource Group: Enter the resource group associated with the AKS cluster.

   4. DNS ZoneId: Enter the ZoneId of the Private DNS Zone used for deploying AutoMQ.

   5. DNS Resource Group: Specify the Resource Group for the Private DNS Zone being utilized to deploy AutoMQ.

   6. Bucket Name: Enter the data Bucket designated for storing messages during the AutoMQ deployment. Support for multiple Buckets is available.

   7. Bucket Endpoint: Provide the Blob Service Endpoint of the data Bucket intended for storing messages during the AutoMQ deployment.

<img />

1. After entering the cloud resource information, follow the console guidance to enter the Managed Identity Client ID used by the data-plane node pool, then click next to preview the creation.

2. Preview the deployment configuration information and **complete the creation**. You can then go to the instance management page to create an instance.

<Tip>
  All the information required for the above configuration (including AKS cluster name, Resource Group, DNS Zone ID, Storage Account Bucket name and Endpoint, Managed Identity Client ID, etc.) can be obtained directly from the Terraform Output after execution. There is no need to look up each value individually in the Azure portal. After running `terraform apply`, refer to the Output values printed in the terminal to fill in the configuration.
</Tip>

## Option 2: Manual Installation via Azure Portal

This section describes the detailed steps to manually complete the environment deployment through the Azure portal.

### Step 1: Install Environment Console

Refer to [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), AutoMQ is compatible with deployment on AKS clusters. In the AKS deployment mode, it is first necessary to install the AutoMQ Console, and then use the console interface to operate AKS and deploy the cluster to AKS.

On Azure, it's recommended to install the console using ARM templates. Refer to [Install Env on Azure▸](/automq-cloud/getting-started/install-byoc-environment/install-env-on-azure).

### Step 2: Create Managed Identity

When deploying AutoMQ to AKS, the AutoMQ data plane cluster requires a dedicated node pool. This node pool must be bound to an independent Managed Identity to access cloud resources. Therefore, before creating the AKS node pool, you need to create a Managed Identity in advance and grant the necessary permissions for AutoMQ. The steps are as follows:

1. Visit the [Managed Identity Console](https://portal.azure.com/#browse/Microsoft.ManagedIdentity%2FuserAssignedIdentities). Click "Create." Choose the following parameters:

* Resource Group: It is recommended to keep it consistent with the AutoMQ Console resource group.

* Region: It is recommended that the target region for deploying AutoMQ remains consistent.

<Tip>
  AutoMQ requires that cloud resources such as the environment console, AKS, Storage Account, Private DNS Zone, etc., be located within the same Resource Group. If your Resource Group is not consistent, additional authorization to the AutoMQ console will be necessary.
</Tip>

<img />

1. Click Next, **Create Managed Identity**. Once created, click on the Managed Identity, navigate to Overview, and record the Client ID, which will be needed when creating the deployment configuration in step 4.

<img />

### Step 3: Create an AKS Cluster

Refer to [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview). Users should pre-create a standalone AKS cluster designated for AutoMQ use. Users can visit the Azure AKS product console to follow the steps below.

1. Log in to the Azure [AKS Console](https://portal.azure.com/#browse/Microsoft.ContainerService%2FmanagedClusters). Click **Create Cluster**.

<img />

1. **Configure Basic Cluster Information**, focusing on the specified configuration items, while leaving other options at their default settings.

   1. Region: Choose the correct region.

   2. Resource Group: Select the appropriate resource group, preferably the same as the console, Storage Account, DNS, etc.

   3. Availability Zone: Choose 3 availability zones.

   4. AKS Pricing Tier: Choose `Standard`.

   5. Authentication and Authorization: Select `Local account with Kubernetes RBAC`.

<img />

3. **Configure Node Pool**: Set up a dedicated node pool for AutoMQ (you can modify the default generated node pool or create a new separate node pool). Refer to the reference values below for configuration items to modify, and it is recommended to keep other configurations unchanged.

   4. Mode: Select `User` mode.

   5. Availability zones: It is recommended to choose at least 3 availability zones.

   6. Node size: Refer to [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview) and select the `Standard D4as v5` instance type.

   7. Scale method: Choose the `Autoscale` mode to automatically scale nodes based on deployment needs.

   8. Minimum node count: It is recommended to select at least 3 nodes.

   9. Max pods per node: It is advised to set this to 30.

   10. Taints: Add a taint configuration with the key as `dedicated`, value as `automq`, and effect as `NoSchedule`.

<img />

1. **Configure the Network.** Choose the target **Virtual Network and Subnet.**

   1. Network Configuration: Ensure that you select the `Azure CNI Node Subnet` mode so that Pods can directly use VNet IPs.

   2. Bring your own Azure Virtual Network: Make sure this option is checked and enabled.

   3. Virtual Network: Select the private network where AutoMQ should be deployed.

   4. Cluster subnet: Pick the subnet where AutoMQ should be deployed.

<img />

1. **Maintain the default configurations and create an AKS cluster.**

2. **Go to the** [VMSS Console](https://portal.azure.com/#view/Microsoft_Azure_ComputeHub/ComputeHubMenuBlade/~/virtualMachineScaleSetsBrowse)**. Find the VMSS associated with the AutoMQ dedicated node pool, and associate the Managed Identity created in step 2.**

<Tip>
  The Pods of AutoMQ data plane components access Azure cloud resources by utilizing the Managed Identity linked to the AKS Node.

  It is advisable to use the node pool name and resource group to filter and locate the appropriate VMSS. The VMSS naming format is aks-\{NodePoolName}-xxxx-vmss.
</Tip>

<img />

Once the correct VMSS is located, click on the Security >> Identity page, and associate the previously created Managed Identity.

<img />

1. **Assign the** `Network Contributor` **role to the AKS cluster.** Navigate to the [Virtual Network console](https://portal.azure.com/#view/HubsExtension/AssetMenuBlade/~/virtualnetworks/assetName/NetworkFoundation/extensionName/Microsoft_Azure_Network), find the network where the current AKS is located, click on the IAM option, and add a role assignment.

<Tip>
  Deploying AutoMQ to AKS requires a Load Balancer to be used as the Service IP. To create a Load Balancer, AKS must be granted the Network Contributor role.
</Tip>

<img />

During authorization, first select the `Network Contributor` role, then search for `System-assigned managed identity`, and choose the managed identity used by the current AKS cluster.

<img />

2. Create a Placeholder Deployment for the node pool used by AutoMQ to accelerate failover speed in node failure scenarios.

<Tip>
  **Working principle:**

  Placeholder Deployment functions deploy a low-priority "placeholder" application on the nodes of a Kubernetes cluster, occupying several nodes. When nodes hosting Pods of the AutoMQ cluster experience failures, these Placeholder nodes can be quickly reclaimed for rapid recovery.
</Tip>

Placeholder Deployment can be managed using the kubectl command or through the Kubernetes console.

First, [click the link](http://download.automq.com/k8s-placeholder/automq-low-priority.yaml) to download the priority declaration file named `automq-low-priority.yaml`, and execute the following command to create a priority declaration.

```bash theme={null}
kubectl apply -f automq-low-priority.yaml
```

Then, [click the link](http://download.automq.com/k8s-placeholder/automq-aks-placeholder.yaml) to download the `automq-aks-placeholder.yaml` file. You may need to adjust the parameters within based on the actual node pool deployment:

* `metadata.name`: It is recommended to modify this to a meaningful Placeholder name, such as `placeholder-for-nodegroup-A`.

* `replicas` : This represents the number of placeholder pods reserved, with a default setting of 1. When deploying across multiple availability zones, it's recommended to maintain 1 machine in each zone, setting the number equal to the available zones count.

* `affinity.nodeAffinity` : This is used to select nodes for Placeholder deployment. Adjust the `matchExpressions` within `key` and `values` to accurately match the AutoMQ node pool. The example YAML file provides 2 node selection options.

  * `kubernetes.azure.com/agentpool` : Utilize the `kubernetes.azure.com/agentpool` tag to filter specific node pools within Azure.

  * `node.kubernetes.io/instance-type` : Use the `node.kubernetes.io/instance-type` tag to filter specific node models on Azure.

* `resources` :

  * CPU/memory limits should match the specific node pool specifications, such as 2C16G.

  * The CPU/memory requests are slightly below the specific specifications of the node pool, for example, an actual usage ratio of 3/4. This ensures the Placeholder Pod can be scheduled to an additional node, achieving exclusive occupancy. This avoids unexpected usage by other pods in the cluster, which could lead to scheduling failure due to insufficient resources during an actual failover.

**The parameter segment that needs modification can be found in the following YAML file:**

```yaml theme={null}
metadata:
  # TODO: Replace with Your Custom Name
  name: {Replace with your custom placeholder deployment name}
spec:
  # TODO: Replace with Your Custom Node Nums
  replicas: 1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.azure.com/agentpool
                  operator: In
                  values:
                  # TODO: Replace with Your Custom Node Pool Name
                  - "Replace with your custom Node Pool Name"
                - key: node.kubernetes.io/instance-type
                  operator: In
                  values:
                  # TODO: Replace with Your Custom Node Pool VM Size
                  - "Replace with your custom Node Pool VM Size"
      containers:
        - name: placeholder
          resources:
          # TODO: Replace with Your Custom Memory and CPU Size
            limits:
              cpu: 2000m
              memory: 16Gi
            requests:
              cpu: 1000m
              memory: 12Gi
```

After modification, execute the following command to install the Placeholder.

```bash theme={null}
kubectl apply -f automq-aks-placeholder.yaml
```

Once executed, use the following command to check the status of the Placeholder Pod to ensure it is in the `Running` state and verify whether it has been scheduled to the expected node.

```bash theme={null}
kubectl get pods -l app=low-priority-placeholder -o wide
```

### Step 4: Enter the Environment Console to Create a Deployment Configuration.

When first accessing the AutoMQ BYOC console, you need to prepare resources such as an object storage bucket and private DNS, create deployment configuration, and set Kubernetes cluster information and authorization for proper use.

1. Go to the [Storage Account console](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) and create a Blob Container to save data in the Data Bucket. Ensure that the region of the Storage Account aligns with the AutoMQ console and AKS, etc.

<img />

After creation, record the current container name and Blob Service Endpoint.

<img />

1. Navigate to the [Private DNS Zone product console](https://portal.azure.com/#browse/Microsoft.Network%2FprivateDnsZones) and create a Private Zone for the subsequent access point resolution of AutoMQ instances. Ensure that the Zone's region aligns with the AutoMQ console and AKS, etc.

<img />

Remember to select Virtual Network Links and associate the Virtual Network of AutoMQ and AKS with the current Zone.

<img />

1. Log in to the console, enter the cluster name and other configurations, then click next.

   1. Deployment Type: Select Kubernetes.

   2. Kubernetes Cluster: Enter the cluster name for the AKS (Azure Kubernetes Service) cluster.

   3. AKS Resource Group: Enter the resource group associated with the AKS cluster.

   4. DNS ZoneId: Enter the ZoneId of the private DNS zone used for deploying AutoMQ.

   5. DNS Resource Group: Specify the Resource Group for the Private DNS Zone being utilized to deploy AutoMQ.

   6. Bucket Name: Enter the data Bucket designated for storing messages during the AutoMQ deployment. Support for multiple Buckets is available.

   7. Bucket Endpoint: Provide the data Bucket endpoint intended for storing messages during the AutoMQ deployment. Support for multiple Buckets is available.

<img />

4. Once you have entered the cloud resource information, refer to the console guidance to authorize the Managed Identity used by the data-plane node pool (i.e., the Managed Identity ClientID created in step 2). The authorization process is as follows:

**Navigate to** [Managed Identity console](https://portal.azure.com/#browse/Microsoft.ManagedIdentity%2FuserAssignedIdentities)**,** locate the Managed Identity from step 2, **click on the IAM option**, and enter the Role assignments menu. Grant this Managed Identity **Contributor role permissions**.

<img />

Select the previously created Managed Identity to complete authorization.

<img />

Go to the [Storage Account Console](https://portal.azure.com/#view/HubsExtension/BrowseResource.ReactView/resourceType/Microsoft.Storage%2FStorageAccounts), click on the IAM option, and proceed to grant the current Managed Identity the `Storage Blob Data Contributor` role.

<img />

After completing the authorization, enter the ClientID of the current Managed Identity and click Next to preview the creation.

1. Preview the deployment configuration information and **complete the creation**. You can then go to the instance management page to create an instance.


# Deploy to Google Cloud GKE
Source: https://docs.automq.com/automq-cloud/deploy-automq-on-kubernetes/deploy-to-google-cloud-gke

AutoMQ offers scalable, cost-efficient Kafka-compatible cloud service. Learn how to deploy AutoMQ clusters on Google Cloud GKE for optimal performance.

As described in [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), AutoMQ supports deployment on Kubernetes. This article outlines the installation process for deploying an AutoMQ cluster on the Google Cloud GKE platform.

<Info>
  In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.
</Info>

## Install the Environment Console

As described in [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), AutoMQ supports deployment on GKE clusters. In the GKE deployment mode, you still need to first install the AutoMQ console, and then use the console interface to manage GKE for deploying the cluster on GKE.

On Google Cloud, the 8.x AutoMQ console is installed and started with a Docker image. For console installation instructions, see [Install AutoMQ on Google Cloud▸](/automq-cloud/getting-started/install-byoc-environment/google-cloud/install-automq-on-gcp).

After the AutoMQ console installation is complete, use the console address, initial username, and password to log in to the console, and follow the page instructions to complete permission initialization.

## Prepare the GKE Cluster and Required Node Pools

If you want to run AutoMQ on Kubernetes, prepare a GKE cluster for AutoMQ. If you only want to deploy AutoMQ in IaaS mode, you do not need to prepare a GKE cluster.

### Step 1: Create a GKE Standard cluster

For instructions on creating a GKE Standard cluster, see the Google Cloud documentation [Creating a regional cluster](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster). This document does not repeat the full GKE creation procedure. It only lists the key configuration requirements for AutoMQ deployment.

AutoMQ currently supports only GKE Standard clusters. GKE Autopilot is not supported. When you create the cluster, confirm the following settings:

* **Cluster type**: Select GKE Standard.
* **Cluster location**: A Regional cluster is recommended so that AutoMQ instances can span multiple zones.
* **Network mode**: Enable VPC-native traffic routing and prepare secondary IP ranges for Pods and Services.
* **Dataplane**: Dataplane V2 is recommended.
* **Identity model**: Enable [Workload Identity Federation for GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
* **Node metadata**: Do not disable the GKE Metadata Server. AutoMQ Pods depend on it when a Kubernetes ServiceAccount is bound to a Google Service Account (GSA).

If you use Shared VPC, use the following project boundary:

* **Service project**: Owns the GKE cluster, Console VM, AutoMQ instances, data buckets, and instance GSAs.
* **Host project**: Owns the VPC, subnets, routes, NAT, firewall rules, and other network resources.

In Shared VPC mode, the GKE service agent also needs permissions in the host project to use the shared network. See the Google Cloud documentation [Setting up clusters with Shared VPC](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc).

### Step 2: Configure network and firewall rules

The AutoMQ console needs to access the GKE API Server and operate Kubernetes resources during instance creation, upgrades, and scaling. The AutoMQ console also needs to access the service ports exposed by the AutoMQ data plane for lifecycle management, status checks, and operations. Whether Kafka clients need to access these ports depends on your application access pattern.

If the AutoMQ console, GKE cluster, and Kafka clients are not in the same subnet or cannot communicate by default, add the required VPC firewall rules. At minimum, confirm the following access paths:

* The AutoMQ console can access the GKE API Server.
* The AutoMQ console can access the AutoMQ Pod or Service ports. This access path is required.
* Kafka client networks can access the AutoMQ service ports required by your applications, such as Kafka bootstrap, broker, or related proxy ports.
* If the GKE nodes don't use external IPs and need to access Google APIs such as Cloud Storage, IAM, and Cloud DNS, the node subnet has Private Google Access enabled and the required routes and DNS are configured for Google APIs. Access to external image registries or other non-Google endpoints still requires Cloud NAT or another egress path.

Common AutoMQ service ports include: `8083`, `9090`, `9092`, `9093`, `9102`, `9103`, `9112`, and `9113`. The console subnet is a required source. Add Kafka client source CIDR ranges only when those clients need application access. Restrict the source CIDR ranges based on the console subnet and client networks. If the AutoMQ workload node pool uses a dedicated node VM service account, you can restrict the target to that node pool by service account. Otherwise, use network tags or another targeting method to narrow the target scope.

The following command shows how to allow common AutoMQ ports from the console source CIDR range and optional client source CIDR ranges, while narrowing the target when a dedicated node VM service account is used. In Shared VPC mode, create firewall rules in the host project.

```bash theme={null}
HOST_PROJECT_ID="<host-project-id>"
VPC_NAME="<vpc-name>"
AUTOMQ_NODE_SA_EMAIL="<dedicated-automq-node-service-account-email>"
# If you don't need to allow Kafka client CIDR ranges yet, keep only the console subnet CIDR.
SOURCE_CIDRS="<console-subnet-cidr>,<client-cidr>"

gcloud compute firewall-rules create "automq-gke-allow-automq-service" \
  --project="${HOST_PROJECT_ID}" \
  --network="${VPC_NAME}" \
  --direction=INGRESS \
  --action=ALLOW \
  --source-ranges="${SOURCE_CIDRS}" \
  --target-service-accounts="${AUTOMQ_NODE_SA_EMAIL}" \
  --rules=tcp:8083,tcp:9090,tcp:9092,tcp:9093,tcp:9102,tcp:9103,tcp:9112,tcp:9113
```

### Step 3: Prepare the AutoMQ workload node pool

<Tip>
  When you create a GKE Standard cluster, it has a node pool that can run GKE system components. Prepare a separate AutoMQ workload node pool, and add the `dedicated=automq:NoSchedule` taint only to the AutoMQ workload node pool.
</Tip>

AutoMQ Brokers should run on a dedicated node pool. Prepare this node pool in advance, and select or enter the node pool name during instance creation.

When you prepare the AutoMQ workload node pool, confirm the following settings:

* **GKE Metadata Server**: Must be enabled.
* **Taint**: Set `dedicated=automq:NoSchedule` so that only AutoMQ workloads are scheduled to this node pool.
* **Machine type and size**: Select machine types based on the node pool requirements in [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview), and set the minimum, desired, and maximum node counts based on workload size.
* **Autoscaling**: Cluster Autoscaler with the Balanced policy is recommended.

<Tip>
  AutoMQ on GKE uses Workload Identity to map a Kubernetes ServiceAccount to a Google Service Account (GSA). Make sure the AutoMQ workload node pool has GKE Metadata Server enabled. The node pool VM Service Account and access scopes affect the node itself, but they no longer carry AutoMQ data plane permissions.
</Tip>

### Step 4: Confirm the AutoMQ workload identity

AutoMQ data plane Pods need to access Google Cloud resources such as GCS, Cloud DNS, and Compute. When you create an instance, choose one of the following identity modes:

| Identity mode     | When to use                                                                                          | What you prepare                                                                                                                                                      | What the AutoMQ console does                                                                                                    |
| ----------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Managed identity  | Recommended for new environments where the console should create and maintain the instance identity. | Make sure control-plane permission initialization has passed.                                                                                                         | Creates the instance GSA and configures the required IAM bindings for GCS, Cloud DNS, Workload Identity, and related resources. |
| Existing identity | Use when your organization already has GSA naming, audit, or approval workflows.                     | Create the GSA in advance and grant the permissions shown in the console. The GSA resource name uses `projects/<project-id>/serviceAccounts/<service-account-email>`. | Verifies the GSA is readable and uses it in the instance configuration.                                                         |

If you use an existing identity, also prepare the Kubernetes namespace and Kubernetes ServiceAccount, and make sure the ServiceAccount can use the GSA through Workload Identity. See [Appendix: Configure GKE Workload Identity for AutoMQ workloads](#appendix-configure-gke-workload-identity-for-automq-workloads).

## Create an AutoMQ Instance and Select Kubernetes Deployment

Log in to the AutoMQ console, create an instance, select Kubernetes as the deployment type, and configure the following settings.

1. **Deployment type**: Select Kubernetes.

2. **Kubernetes cluster**: Select the target GKE cluster. If you use an API or automation tool, the GKE cluster ID must be the full resource name: `projects/<project-id>/locations/<location>/clusters/<cluster-name>`.

3. **Node pool**: Select or enter the node pool name for AutoMQ Brokers. The node pool must meet the node pool requirements in [Overview▸](/automq-cloud/deploy-automq-on-kubernetes/overview).

4. **Data bucket**: Let AutoMQ create a managed bucket, or enter an existing GCS bucket.

5. **Private DNS Zone**: Let AutoMQ create a managed Cloud DNS private managed zone, or enter an existing zone.

6. **Instance identity**:
   * For managed identity, the AutoMQ console creates the instance GSA and configures the required IAM bindings.
   * For existing identity, enter the full resource name of the existing GSA and confirm that the permissions shown in the console have been granted.

7. **Namespace and ServiceAccount**: Required only when you manage the Kubernetes identity yourself. The ServiceAccount must be bound to the instance GSA through Workload Identity.

8. Preview the configuration and **complete the creation**.

## Appendix: Configure GKE Workload Identity for AutoMQ workloads

For complete instructions, see the Google Cloud documentation [Authenticate to Google Cloud APIs from GKE workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).

If you use an existing GSA, make sure the Kubernetes ServiceAccount can impersonate the GSA. Example steps:

1. Create the namespace and Kubernetes ServiceAccount.

   ```bash theme={null}
   kubectl create namespace <automq-namespace>
   kubectl create serviceaccount <automq-service-account> -n <automq-namespace>
   ```

2. Add a `roles/iam.workloadIdentityUser` binding to the GSA.

   ```bash theme={null}
   gcloud iam service-accounts add-iam-policy-binding <gsa-email> \
     --role roles/iam.workloadIdentityUser \
     --member "serviceAccount:<project-id>.svc.id.goog[<automq-namespace>/<automq-service-account>]"
   ```

3. Annotate the Kubernetes ServiceAccount with the GSA.

   ```bash theme={null}
   kubectl annotate serviceaccount <automq-service-account> \
     -n <automq-namespace> \
     iam.gke.io/gcp-service-account=<gsa-email>
   ```

When you create the AutoMQ instance, enter the namespace, ServiceAccount, and GSA resource name prepared above.

## Appendix: Deploy AutoMQ Placeholder Deployment (optional)

If you want to reserve failover capacity in the AutoMQ workload node pool, you can deploy a Placeholder Deployment for the AutoMQ node pool. It runs low-priority placeholder Pods that pre-allocate resources. When a node that hosts an AutoMQ Broker fails, the low-priority Pods can be preempted so that the Broker can be scheduled and recovered faster.

You can deploy the Placeholder Deployment by using `kubectl` or the Kubernetes console.

1. Download and create the low-priority class declaration.

   ```bash theme={null}
   curl -fLO https://download.automq.com/k8s-placeholder/automq-low-priority.yaml
   kubectl apply -f automq-low-priority.yaml
   ```

2. Download the GKE Placeholder Deployment example.

   ```bash theme={null}
   curl -fLO https://download.automq.com/k8s-placeholder/automq-gke-placeholder.yaml
   ```

3. Modify the key parameters in `automq-gke-placeholder.yaml` based on the actual AutoMQ workload node pool:

   * `metadata.name`: Use a meaningful name, such as `placeholder-for-automq-nodepool-a`.
   * `replicas`: The number of Placeholder Pods to reserve. For multi-zone deployments, evaluate whether to reserve one Pod per zone.
   * `affinity.nodeAffinity`: Match the AutoMQ workload node pool. On GKE, use `cloud.google.com/gke-nodepool` to match the node pool name, or use `node.kubernetes.io/instance-type` to match the node machine type.
   * `resources`: Set `limits` based on the node size. Set `requests` slightly lower than the node size so that the Placeholder Pods can be scheduled.

4. Apply the Placeholder Deployment.

   ```bash theme={null}
   kubectl apply -f automq-gke-placeholder.yaml
   ```

5. Check whether the Placeholder Pods are running on the expected node pool.

   ```bash theme={null}
   kubectl get pods -l app=low-priority-placeholder -o wide
   ```


# Overview
Source: https://docs.automq.com/automq-cloud/deploy-automq-on-kubernetes/overview

Requirements, supported platforms, and machine types for deploying AutoMQ BYOC on managed Kubernetes.

AutoMQ BYOC (Bring Your Own Cloud) environments support deployment to managed Kubernetes platforms provided by AWS, Google Cloud, Azure, Alibaba Cloud, Huawei Cloud, and Tencent Cloud. This article introduces the basic concepts and constraints of deploying AutoMQ to Kubernetes platforms.

<Info>
  In this article, mentions of AutoMQ product service providers, AutoMQ service providers, and AutoMQ all specifically refer to AutoMQ HK Limited and its subsidiaries.
</Info>

## Constraints and Limitations

When deploying AutoMQ on Kubernetes, users need to follow the constraints and limitations outlined below. Improper user operations may lead to deployment and modification failures of the cluster.

### Constraint 1: Users Must Provide a Dedicated Kubernetes Cluster that Meets the Requirements

AutoMQ requires a dedicated Kubernetes cluster that should not be shared with other application systems for the following reasons:

* **Network isolation risk in Kubernetes:** AutoMQ is designed for high-throughput data transmission scenarios and demands high network throughput. Network isolation between different Pods within a Kubernetes cluster is not complete. AutoMQ requires an independent Kubernetes cluster to avoid interference with other business systems of the user.

* **Operations authorization isolation:** The Kubernetes cluster where AutoMQ resides needs to provide operations authorization (granting AutoMQ service provider access to the cluster). AutoMQ requires an independent Kubernetes cluster to prevent unauthorized access by application systems.

<Tip>
  Multiple AutoMQ instances can be deployed in one Kubernetes cluster. Share a Kubernetes cluster only among workloads in the same business domain and cloud network.
</Tip>

### Constraint 2: Users Must Provide Dedicated Compute Capacity that Meets the Requirements

AutoMQ requires dedicated compute capacity. You can provide it through a managed node pool or an autoscaler such as Karpenter. The compute pool must permit the instance types and Availability Zones selected when you create an AutoMQ instance.

<Tip>
  AutoMQ requires dedicated compute capacity. Provide a separate stable compute pool for Kubernetes system components.
</Tip>

* **Add an exclusive label and taint:** To prevent other workloads from occupying resources on AutoMQ nodes, you can add a custom label such as `node-type=automq` and the taint `dedicated=automq:NoSchedule`. Use the corresponding node affinity and toleration in the [SchedulingSpec](#schedulingspec).

* **Compliant Machine Types:** The compute pool must permit the machine types supported by AutoMQ for the selected cloud provider. As of August 2026, the supported machine types are listed below:

| **Cloud Provider**<br /> | **Allowed Machine Types List**<br />                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------------------------- |
| AWS                      | `r6in.large`, `r6i.large`, `m7g.xlarge`, `m7g.2xlarge`, `c6g.2xlarge`, `m7g.4xlarge`, `m6g.medium`            |
| Google Cloud             | `n4d-standard-2`, `n4d-highmem-2`, `n4a-highmem-1`, `n4a-standard-2`, `n4d-standard-4`                        |
| Azure                    | `Standard_D4as_v5`, `Standard_D4s_v4`, `Standard_E2as_v5`                                                     |
| Alibaba Cloud            | `ecs.r7.large`, `ecs.r6.large`, `ecs.u1-c1m8.large`, `ecs.u1-c1m2.2xlarge`, `ecs.g8y.xlarge`, `ecs.r8y.large` |
| Huawei Cloud             | `c6ne.xlarge.4`, `m7.large.8`, `m7n.large.8`                                                                  |
| Tencent Cloud            | `SA5.2XLARGE16`, `SA5.4XLARGE32`, `SA5.LARGE16`, `S6.LARGE16`, `S6.2XLARGE16`, `SA4.2XLARGE16`                |

* **Workload Identity Authorization Requirements:** AutoMQ clusters need to access services such as object storage and cloud volumes during operation. Grant the required cloud permissions to the cloud identity or role associated with the AutoMQ workload ServiceAccount, and ensure that the authorization is not modified or revoked.

* **Zone (Subnet) Requirements:** AutoMQ supports both single-zone and three-zone instances. The managed node pool or Karpenter NodePool must permit the Availability Zones selected for the instance.

<Tip>
  Example: A user needs to create three AutoMQ instances, namely Instance 1 (Zone A), Instance 2 (Zone B), and Instance 3 (Zones A, B, and C). The compute pool requirements must permit Zone A, Zone B, and Zones A+B+C respectively.
</Tip>

## SchedulingSpec

The **Scheduling YAML** field is optional. Use it when AutoMQ pods need to target dedicated Kubernetes nodes or tolerate taints on those nodes.

### Supported fields

The YAML supports these root fields:

| Field         | Supported configuration                                                                              |
| ------------- | ---------------------------------------------------------------------------------------------------- |
| `affinity`    | `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` |
| `tolerations` | A list of standard Kubernetes toleration objects                                                     |

Node affinity has these restrictions:

* Configure at most one `nodeSelectorTerm`.
* Each term can contain only non-empty `matchExpressions`.
* Each expression must use the `In` operator and include a key and at least one value.

The following fields aren't supported in the Scheduling YAML: `nodeSelector`, `topologySpreadConstraints`, `matchFields`, pod affinity, and pod anti-affinity.

### Configure dedicated nodes

The following example applies to both managed node pools and Karpenter NodePools. Configure the compute pool to add:

* Custom label: `node-type=automq`
* Taint: `dedicated=automq:NoSchedule`

Enter this Scheduling YAML when you create the AutoMQ instance:

```yaml theme={null}
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node-type
              operator: In
              values:
                - automq
tolerations:
  - key: dedicated
    operator: Equal
    value: automq
    effect: NoSchedule
```

For a Karpenter NodePool, add the label and taint under `spec.template`, and ensure that `spec.template.spec.requirements` permits the instance types and Availability Zones selected in AutoMQ Console.

Instead of adding a custom label, you can match a node pool label provided by your cloud provider's managed Kubernetes service or node pool implementation. Label keys vary across cloud providers and implementations. Replace the key and value in `matchExpressions` above with the actual node pool label on the target nodes.

### System-managed scheduling conditions

Don't add these keys to `matchExpressions`:

* `node.kubernetes.io/instance-type`
* `topology.kubernetes.io/zone`

AutoMQ derives these conditions from the **Instance type** and **Availability Zone** fields in the creation form. A request containing either key in the Scheduling YAML is rejected.

The load balancer subnets are also selected separately in the creation form. Select subnets that cover the workload Availability Zones; don't place subnet IDs in the Scheduling YAML.

### Troubleshoot pending pods

If AutoMQ pods remain pending, check:

* The target nodes contain the label used by node affinity.
* Every `NoSchedule` taint on the target nodes has a matching toleration.
* The managed node pool or Karpenter NodePool permits the selected instance type and Availability Zones.
* The compute pool hasn't reached its resource limits and the cloud provider has capacity.

Use these commands to inspect the scheduling state:

```bash theme={null}
kubectl get nodes \
  -L node-type,node.kubernetes.io/instance-type,topology.kubernetes.io/zone
kubectl describe pod <pod-name> --namespace <automq-namespace>
```

For Karpenter, also inspect NodePool and NodeClaim status:

```bash theme={null}
kubectl get nodepools,nodeclaims
```


# Client SDK Guide
Source: https://docs.automq.com/automq-cloud/getting-started/client-sdk-guide

AutoMQ provides full Apache Kafka protocol compatibility for scalable, cloud-native messaging. This guide covers recommended client SDKs for various languages to ensure optimal performance and stability.

AutoMQ is fully compatible with the Apache Kafka protocol, allowing you to connect and exchange messages using standard Kafka client SDKs. This guide outlines the recommended clients for various programming languages and highlights important considerations for their use with AutoMQ.

## Compatibility Notes

AutoMQ is compatible with the Apache Kafka protocol, supporting clients from Kafka versions 0.10 to 3.9. This compatibility ensures that any standard Kafka client for these versions can connect to AutoMQ without modification.

<Tip>
  We strongly recommend using the latest stable version of your chosen client SDK to leverage recent performance enhancements, features, and critical bug fixes.
</Tip>

## Recommended Client SDK List

The Apache Kafka community maintains a list of client SDKs for various programming languages, which you can find in the [official documentation](https://cwiki.apache.org/confluence/display/KAFKA/Clients). Based on extensive testing and production use, we recommend the following clients for optimal performance and stability with AutoMQ. To help you get started, we provide code samples for each recommended client in our [AutoMQ Labs](https://github.com/AutoMQ/automq-labs) repository.

| **Programming Language**<br /> | **Client SDK**<br />                                              | **Recommended Version**<br /> | Sample<br />                                                                                         |
| ------------------------------ | ----------------------------------------------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------- |
| Java<br />                     | [Apache Kafka Java Client](https://github.com/apache/kafka)<br /> | >= 3.2.0<br />                | [Demo](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/clients/java)<br />       |
| C/C++<br />                    | [librdkafka](https://github.com/edenhill/librdkafka)<br />        | >= 2.8.0<br />                | [Demo](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/clients/cpp)<br />        |
| Go<br />                       | [franz-go](https://github.com/twmb/franz-go)<br />                | >= 1.17.1<br />               | [Demo](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/clients/go)<br />         |
| Python<br />                   | [kafka-python](https://pypi.org/project/kafka-python/)<br />      | >= 2.2.3<br />                | [Demo](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/clients/python)<br />     |
| NodeJS<br />                   | [KafkaJS](https://kafka.js.org/)<br />                            | >= 2.2.4<br />                | [Demo](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/clients/javascript)<br /> |

<Tip>
  If the SDK you are using is not among the recommended ones listed above, as long as it is compatible with the Apache Kafka protocol, you can continue to access AutoMQ. We suggest reviewing the known issues list below and adjusting parameters accordingly.
</Tip>

## Client Parameter Tuning

When using the SDK to send and receive messages, in addition to selecting the recommended SDK version, you need to tune the key parameters of the Producer to achieve optimal performance. Below are the recommended Producer configurations for each SDK and Kafka Output.

### Apache Kafka Java Client > 2.1

```properties theme={null}
# https://kafka.apache.org/documentation/#producerconfigs
# Batch size, up to 1MiB of data can be accumulated at a time.
batch.size=1048576
# Batch linger time. If the batch limit is not reached within this time, sending is triggered anyway. This represents the maximum send delay.
linger.ms=1000
```

### Apache Kafka Java Client \<= 2.1

```properties theme={null}
# https://kafka.apache.org/21/documentation.html#producerconfigs
# Maximum retry count
retries=2147483647

# Initial backoff delay time
retry.backoff.ms=100
# Maximum backoff delay time
retry.backoff.max.ms=1000
# RPC timeout for each send request
request.timeout.ms=30000
# Total timeout for the entire send call. No further retries after this time; an exception is returned to the caller.
delivery.timeout.ms=120000
# Batch size, up to 1MiB of data can be accumulated at a time.
batch.size=1048576
# Batch linger time. If the batch limit is not reached within this time, sending is triggered anyway. This represents the maximum send delay.
linger.ms=1000
```

### franz-go

```go theme={null}
package main

import (
	"time"

	"github.com/twmb/franz-go/pkg/kgo"
)

func main() {
	client, err := kgo.NewClient(
		kgo.SeedBrokers("localhost:9092"),
		// Equivalent to the linger.ms configuration in the Java client, set to 1 second.
		kgo.ProducerLinger(1*time.Second),
	)
	if err != nil {
		panic(err)
	}
	defer client.Close()
}
```

### kafka-go

```go theme={null}
// https://pkg.go.dev/github.com/segmentio/kafka-go#Writer
// Configure kafka.Writer fields; values align with Apache Kafka Java producer defaults.
writer := &kafka.Writer{
	// Maximum number of attempts to deliver a message.
	// Aligned with the Apache Kafka Java Client retries parameter (Integer.MAX_VALUE).
	MaxAttempts: 2147483647,
	// Maximum number of messages to trigger a flush.
	BatchSize: 4096,
	// Number of acknowledges from partition replicas required before responding.
	RequiredAcks: kafka.RequireAll,
}
```

### Sarama

```go theme={null}
package main

import (
	"time"

	"github.com/IBM/sarama"
)

func main() {
	// https://pkg.go.dev/github.com/IBM/sarama#Config
	config := sarama.NewConfig()

	// Byte threshold to trigger a flush.
	config.Producer.Flush.Bytes = 1048576
	// Message count threshold to trigger a flush.
	config.Producer.Flush.Messages = 4096
	// Maximum flush wait time, equivalent to the Kafka Java producer's linger.ms.
	config.Producer.Flush.Frequency = 1 * time.Second

	// Maximum retry count (default 3), equivalent to the Kafka Java producer's retries.
	config.Producer.Retry.Max = 2147483647
	// Backoff wait time between retries (default 100ms), equivalent to the Kafka Java producer's retry.backoff.ms.
	config.Producer.Retry.Backoff = 1 * time.Second

	// TODO: use config to create a producer, e.g. sarama.NewSyncProducer(brokers, config)
	_ = config
}
```

### Logstash Kafka Output

```ruby theme={null}
# https://www.elastic.co/docs/reference/logstash/plugins/plugins-outputs-kafka
output {
  kafka {
    # other connection configs ...
    # Batch size in bytes, up to 1MiB of data can be accumulated at a time.
    batch_size => 1048576
    # Batch linger time in ms. If the batch limit is not reached within this time, sending is triggered anyway.
    linger_ms => 1000
    # The number of acknowledgments the producer requires the leader to have received.
    acks => "all"
    # Compression type for all data generated by the producer.
    compression_type => "lz4"
    # The amount of time to wait before attempting to retry a failed produce request.
    retry_backoff_ms => 1000
  }
}
```

### Filebeat Kafka Output

```yaml theme={null}
# https://www.elastic.co/docs/reference/beats/filebeat/kafka-output
# Retry backoff wait time during leader elections.
retry.backoff: 3s
# Wait time before sending bulk Kafka requests.
bulk_flush_frequency: 1s
```

### Fluentd Kafka Output

```xml theme={null}
# https://github.com/fluent/fluent-plugin-kafka
<match **>
  @type kafka2
  # other connection configs ...

  <buffer>
    # use default configs
  </buffer>
</match>
```

### Fluent Bit Kafka Output

```yaml theme={null}
# https://docs.fluentbit.io/manual/pipeline/outputs/kafka
pipeline:
  outputs:
    - name: kafka
      match: '*'
      # other connection configs ...
      # Batch linger time in ms.
      rdkafka.queue.buffering.max.ms: 1000
      # Compression type.
      rdkafka.compression.type: lz4
```

### Vector Kafka Sink

```toml theme={null}
# https://vector.dev/docs/reference/configuration/sinks/kafka/
[sinks.my_kafka]
type = "kafka"
inputs = ["my-source-or-transform-id"]
# other connection configs ...
# Compression type for all data generated by the producer.
compression = "lz4"

# Batch linger time is controlled via librdkafka options.
[sinks.my_kafka.librdkafka_options]
"queue.buffering.max.ms" = "1000"
```

## Appendix: Other Known Issues

### [Sarama Go SDK](https://github.com/IBM/sarama)

| **Known Defects**<br />                                            | **Defect Information**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | **Resolution Methods**<br />                    |
| ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| Async Producer has no memory usage limit during send retries<br /> | <ul><li>Defect Version: V1.43.3</li><li>Community Issue: [https://github.com/IBM/sarama/issues/1358](https://github.com/IBM/sarama/issues/1358)</li><li>Community PR: [https://github.com/IBM/sarama/pull/3026](https://github.com/IBM/sarama/pull/3026)</li><li>Phenomenon: When using async\_producer to send messages and the corresponding Topic undergoes partition reassignment, the Producer's memory usage can increase abnormally, potentially leading to OOM.</li><li>Cause: When async\_producer receives server-side retryable errors (like NOT\_LEADER\_OR\_FOLLOWER due to partition reassignment), it temporarily stores the messages in memory and retries according to the policy. However, this retry cache has no size limit, and if substantial and continuous retries occur, memory usage may unexpectedly increase.</li></ul> | <ul><li>Upgrade to version >= V1.44.0</li></ul> |

### [Kafka-Go SDK](https://github.com/segmentio/kafka-go)

| **Known Defects**<br />                                      | **Defect Information**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                           | **Resolution Methods**<br />                                                                                                                                                        |
| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Consumers Unable to Identify Group Coordinator Changes<br /> | <ul><li>Defect Version: V0.4.47</li><li>Issue: After partition reassignment of the \_\_consumer\_offsets topic, some groups experience consumption interruptions, with errors logged.</li><li>Cause: When the \_\_consumer\_offsets partitions are reassigned, the coordinator for some groups changes. Kafka-go does not correctly handle this situation, as it continues to send requests to the old coordinator following a change, resulting in NOT\_COORDINATOR errors.</li></ul> | <ul><li>Temporarily restart the Consumer to recover.</li><li>Switch SDKs, and it is recommended to use the Franz-Go SDK as per the recommended versions in this document.</li></ul> |

### [Kafka JS SDK](https://github.com/tulios/kafkajs)

| **Known Defects**<br />                                        | **Defect Information**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | **Resolution Methods**<br />                                                                                                                                             |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Inaccurate Consumer Configuration of `heartbeatInterval`<br /> | <ul><li>Defect Version: V2.2.4</li><li>Community Notes: [https://github.com/tulios/kafkajs/issues/130#issuecomment-422024849](https://github.com/tulios/kafkajs/issues/130#issuecomment-422024849)</li><li>Issue: When the Consumer's configured `heartbeatInterval` is too close to `sessionTimeout`, the Consumer may periodically Leave & Rejoin the Group (though it generally does not affect consumption), logging errors.</li><li>Cause: In kafkajs, the configuration of `heartbeatInterval` only guarantees the "minimum interval between two heartbeats." This means that in certain scenarios (e.g., the Consumer reaches the end of the topic with no new messages to consume), the interval between the Consumer sending heartbeats to the Coordinator may exceed the `heartbeatInterval`; if set too large, it may exceed `sessionTimeout`, causing the Coordinator to evict the Consumer.</li></ul> | <ul><li>Lower the `heartbeatInterval` or increase the `sessionTimeout` (it is advised that heartbeat.interval.ms should not exceed 1/3 of session.timeout.ms).</li></ul> |


# Experience AutoMQ
Source: https://docs.automq.com/automq-cloud/getting-started/experience-automq

Create an AutoMQ instance and verify message production and consumption.

This guide shows you how to create an AutoMQ instance and verify that it can produce and consume Kafka messages.

<Info>
  In this document, product service provider, service provider, and AutoMQ refer to AutoMQ HK Limited.
</Info>

## Prerequisites

Complete the installation and operations authorization for an AutoMQ BYOC (Bring Your Own Cloud) environment. If the environment is not ready, use the guide for your cloud provider:

* [AWS](/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws)
* [Google Cloud](/automq-cloud/getting-started/install-byoc-environment/google-cloud/install-automq-on-gcp)
* [Azure](/automq-cloud/getting-started/install-byoc-environment/install-env-on-azure)

## Create an Instance

1. Sign in to the AutoMQ Console for your environment.
2. In the navigation pane, choose **Instances**, and then choose **New Instance**.
3. If prompted, select the Deployment Profile for the instance.
4. Follow the wizard to configure the instance settings, capacity, additional settings, and IAM permissions. The wizard displays only the settings supported by the selected environment and AutoMQ version.
5. Review the configuration and choose **Create Instance**.
6. Wait until the instance status is **Running** before connecting a client.

## Send and receive messages

1. Open the instance details page and choose **Topics**.
2. Choose **Create Topic**, configure the Topic, and submit the request.
3. Open the Topic details page and choose the message production function.
4. Enter a test record and send it.
5. Use the message consumption function or a Kafka consumer to verify that the record can be consumed.

## Next steps

Use a supported Kafka client SDK to integrate your application. See the [Client SDK development guide](/automq-cloud/getting-started/client-sdk-guide).

<Tip>
  Delete test instances and other cloud resources when they are no longer required to avoid additional charges.
</Tip>


# Install AutoMQ on AWS
Source: https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws

This article describes how to install the AutoMQ console on AWS and complete the initial configuration to prepare for creating and managing AutoMQ clusters.

## Overview

To use the AutoMQ Cloud BYOC environment on AWS, you first need to install the AutoMQ console. The console is a web application deployed in your account that you will use to create, manage, and monitor your AutoMQ clusters.

## Prerequisites

The AutoMQ Console can be deployed directly to Amazon EC2 or Amazon EKS. Prepare the resources required by the deployment option that you select.

| Deployment option | Required resources                                                                                                  | Preparation guide                                                                                                                                                               |
| ----------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Amazon EC2        | A VPC for the AutoMQ Console and AutoMQ instances. Use the VPC that contains your Kafka applications when possible. | [Prepare a VPC](/automq-cloud/getting-started/install-byoc-environment/aws/prepare-vpc)                                                                                         |
| Amazon EKS        | A VPC and an existing EKS cluster with the required compute, networking, storage, and identity configuration.       | [Prepare a VPC](/automq-cloud/getting-started/install-byoc-environment/aws/prepare-vpc) and [Deploy to Amazon EKS](/automq-cloud/deploy-automq-on-kubernetes/deploy-to-aws-eks) |

## Register Your AutoMQ Environment

First, you need to register your deployment environment on the AutoMQ official website to obtain the metadata for environment installation.

1. Visit [AutoMQ Cloud](https://account.automq.cloud) and log in to your account.
2. On the console page, click **Create Environment**.
3. Follow the wizard to enter the following information:
   * **Environment Name**: Set a custom name for your environment.
   * **Description**: Enter a description for the environment.
   * **Cloud Provider**: Select **AWS**.
   * **Region**: Select the AWS region where you plan to deploy AutoMQ.
4. After creation, you will receive the environment metadata for deployment.

## Deploy AutoMQ Console on EC2

### Prepare and Deploy the Console EC2 Instance

The AutoMQ console is the core component for managing the AutoMQ Cloud BYOC environment, and it is deployed on a standalone EC2 instance. Therefore, you need to prepare this EC2 instance before installation.

#### Basic Requirements

To ensure the stable operation of the console, the EC2 instance you prepare needs to meet the following basic requirements:

1. **Instance Specifications**:
   * **Configuration**: At least a **2-core CPU** and **8 GB of memory**.
   * **System Volume**: At least **20 GiB** of storage space.
   * **Data Volume**: An additional data volume of at least **20 GiB** must be mounted to store metadata.

<Tip>
  **Important Note**: To prevent metadata loss from the console due to EC2 instance migration or failure, it is crucial to mount an additional data volume to the EC2 instance for the AutoMQ console and store the metadata on this volume.
</Tip>

2. **Network Location**:
   * The EC2 instance **must** be located in the same VPC as the AutoMQ cluster that will be deployed later.
   * Typically, this VPC should also be the same as the one where your application services are located to ensure network connectivity.

3. **Public Access**:
   * The EC2 instance must have internet access to download installation packages and synchronize necessary metadata. You can achieve this by associating an Elastic IP (EIP) or by configuring a NAT gateway.

#### Creation and Deployment Steps

1. Log in to the AWS [EC2 console](https://console.aws.amazon.com/ec2/home) and click **Launch instances**.

2. **Operating System (AMI)**: Choose a Linux distribution you are familiar with, such as Amazon Linux 2023 or Ubuntu.

3. **Instance Type**: Select an instance type that meets the above specifications (e.g., `t3.large` or higher).

4. **Network Settings**:
   * Select the target VPC and a public subnet (if using an EIP) or a private subnet (if using a NAT gateway).
   * Ensure the security group opens necessary ports, such as SSH (22) for management access and the console's service port (default 8080). Restrict the source to trusted IP ranges.

5. **Storage (Volumes)**:
   * Configure the root volume size to at least 20 GiB.
   * Click **Add new volume** to add an additional data volume, ensuring its size is not less than 20 GiB.

6. Complete the remaining configurations and launch the instance.

7. **Mount the Data Volume**:
   * After the instance starts, log in to it.
   * Mount the additional data volume to a specified directory (e.g., `/data`). This path will be used as the data mount path when starting the Docker container.
   * Ensure the mount point has the correct permissions.

8. **Install and Start Docker Service**:

   * Install Docker on the EC2 instance. The following are reference commands for Amazon Linux 2023:

   ```bash theme={null}
   # Update packages
   sudo dnf update -y
   # Install Docker
   sudo dnf install docker -y
   # Start the Docker service
   sudo systemctl start docker
   # Add the current user to the docker group to run docker commands without sudo
   sudo usermod -aG docker ec2-user
   # Enable the Docker service to start on boot
   sudo systemctl enable docker
   # Verify Docker installation
   docker version
   ```

   * **Note**: If you are using another Linux distribution (like Amazon Linux 2), refer to the official Docker documentation or use the appropriate package manager for installation.

9. **Deploy the AutoMQ Console**
   * Once the EC2 instance is ready, you can copy the installation command from the installation wizard to start the AutoMQ console.

### Configure Console Permissions and Complete Initialization

After the AutoMQ console starts successfully, you need to log in to the console. On your first visit, you will need to grant it the necessary permissions to operate AWS cloud resources.

1. **Log in to the Console**
   * Access `http://<Your EC2 Public IP>:8080` in your browser.
   * Log in with the initial username and password.

2. **Grant IAM Permissions to the Console**
   After logging in, the interface will guide you to complete IAM authorization for the AutoMQ console. This is to allow the console to have permission to create and manage cluster resources in your AWS account. The process is as follows:

   1. **Create an IAM Policy**:
      * Go to the AWS IAM console and create a new customer-managed policy based on the policy JSON provided in the console interface.

   2. **Create an IAM Role**:
      * Create a new IAM Role.
      * When selecting the trusted entity, choose **AWS service** and **EC2**.
      * Attach the IAM Policy created in the previous step to this Role.

   3. **Associate the IAM Role with the EC2 Instance**:
      * Go to the AWS EC2 console and find the EC2 instance you are using to deploy the console.
      * In **Actions** -> **Security** -> **Modify IAM role**, associate the IAM Role created in the previous step with this instance.

3. **Complete Initialization**
   * Return to the AutoMQ console page and click **Confirm Authorization** or **Next**.
   * The console will automatically verify if the permissions are configured correctly. Once verified, you can officially enter the AutoMQ console and start creating your first cluster.

## Deploy AutoMQ Console on EKS

This section deploys AutoMQ Console as a Helm release in Amazon EKS. Before you begin, [prepare the EKS cluster and compute resources](/automq-cloud/deploy-automq-on-kubernetes/deploy-to-aws-eks#prepare-eks-cluster-and-compute-resources) and keep the deployment configuration generated by the environment installation wizard available.

The following commands and manifests use `<...>` placeholders. Replace them with the corresponding resource identifiers from your environment.

### Prepare the ServiceAccount and runtime configuration

Before installing the Chart, ensure that the target namespace and ServiceAccount exist. You can reuse existing resources or prepare them through your own resource management process. The Chart references `<console-service-account>` in `<console-namespace>` but does not create or modify it.

The environment installation wizard provides the same base64-encoded deployment configuration used by the EC2 Docker deployment.

Prepare a Secret whose lowercase `config` key contains the complete base64 string generated by the installation wizard. Replace `<config-secret-name>` with the Secret name, `<console-namespace>` with the namespace where the console runs, and `<base64-encoded-runtime-config>` with the configuration generated by the installation wizard:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: <config-secret-name>
  namespace: <console-namespace>
stringData:
  config: "<base64-encoded-runtime-config>"
```

Save the manifest to a local file and replace `<runtime-config-secret-file>` with its path:

```bash theme={null}
kubectl apply -f <runtime-config-secret-file>
```

### Configure the workload identity

Bind the IAM role to the ServiceAccount with EKS Pod Identity or IRSA. EKS Pod Identity is recommended for new EKS deployments.

<Tabs>
  <Tab title="EKS Pod Identity">
    1. Confirm that the EKS Pod Identity Agent is installed. EKS Auto Mode includes the agent.
    2. Prepare or select an IAM role with this trust policy:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "pods.eks.amazonaws.com"
          },
          "Action": [
            "sts:AssumeRole",
            "sts:TagSession"
          ]
        }
      ]
    }
    ```

    3. Ensure that the role is associated with the ServiceAccount. If it is not already associated, run the following command. `<eks-cluster-name>` is the target EKS cluster, `<console-namespace>` and `<console-service-account>` identify the ServiceAccount used by the console, `<aws-account-id>` and `<console-role-name>` identify the IAM role, and `<aws-region>` is the region of the EKS cluster:

    ```bash theme={null}
    aws eks create-pod-identity-association \
      --cluster-name <eks-cluster-name> \
      --namespace <console-namespace> \
      --service-account <console-service-account> \
      --role-arn "arn:aws:iam::<aws-account-id>:role/<console-role-name>" \
      --region <aws-region>
    ```

    For more information, see [Learn how EKS Pod Identity grants pods access to AWS services](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).
  </Tab>

  <Tab title="IRSA">
    1. Confirm that the EKS cluster has an IAM OIDC provider.
    2. Prepare or select an IAM role whose trust policy allows `sts:AssumeRoleWithWebIdentity` for this ServiceAccount.
    3. Ensure that the ServiceAccount has the role ARN annotation. If it is not already configured, run the following command. `<console-service-account>` and `<console-namespace>` identify the ServiceAccount used by the console, while `<aws-account-id>` and `<console-role-name>` identify the IAM role:

    ```bash theme={null}
    kubectl annotate serviceaccount <console-service-account> \
      --namespace <console-namespace> \
      "eks.amazonaws.com/role-arn=arn:aws:iam::<aws-account-id>:role/<console-role-name>"
    ```

    For the required trust policy, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
  </Tab>
</Tabs>

### Grant access to the EKS cluster

AutoMQ Console uses the same IAM role to manage Kubernetes resources. Ensure that the role has a standard EKS access entry with the cluster-scoped `AmazonEKSClusterAdminPolicy`. If it is not already configured, run the following commands. `<eks-cluster-name>` is the target EKS cluster, `<aws-account-id>` and `<console-role-name>` identify the IAM role associated with the ServiceAccount, and `<aws-region>` is the region of the EKS cluster:

```bash theme={null}
aws eks create-access-entry \
  --cluster-name <eks-cluster-name> \
  --principal-arn "arn:aws:iam::<aws-account-id>:role/<console-role-name>" \
  --type STANDARD \
  --region <aws-region>

aws eks associate-access-policy \
  --cluster-name <eks-cluster-name> \
  --principal-arn "arn:aws:iam::<aws-account-id>:role/<console-role-name>" \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster \
  --region <aws-region>
```

### Configure the Helm values

Set the following values in your Helm values file:

* `<config-secret-name>`: The runtime configuration Secret prepared above.
* `<aws-account-id>`: The AWS account where the console is deployed.
* `<vpc-id>`: The VPC that contains the EKS cluster.
* `<alb-subnet-id-1>` and `<alb-subnet-id-2>`: Subnets in different Availability Zones where the ALB will be created.
* `<console-service-account>`: The ServiceAccount already associated with the IAM role.
* `<storage-class-name>`: An existing StorageClass in the target EKS cluster that supports dynamic `ReadWriteOnce` provisioning, typically provided by the Amazon EBS CSI driver. Run `kubectl get storageclass` to list available StorageClasses.
* `<console-domain-name>`: The domain name used to access the console.
* `<acm-certificate-arn>`: The ARN of an ACM certificate that matches the console domain.

```yaml theme={null}
automqByocConsole:
  config:
    existingSecret: <config-secret-name>

aws:
  account: "<aws-account-id>"
  network:
    vpcId: <vpc-id>
    subnetIds:
      - <alb-subnet-id-1>
      - <alb-subnet-id-2>

serviceAccount:
  name: <console-service-account>

persistence:
  storageClass: <storage-class-name>

ingress:
  enabled: true
  host: <console-domain-name>
  alb:
    scheme: internet-facing
    protocol: HTTPS
    certificateArn: <acm-certificate-arn>
```

The ALB subnets expose AutoMQ Console and are independent of the NLB subnets selected when you create an AutoMQ instance. To keep the endpoint private, set `ingress.alb.scheme` to `internal` and use private subnets.

The Chart does not constrain pod placement by default. If the EKS cluster has dedicated nodes for AutoMQ Console, add their label as a required affinity. Replace `<node-label-key>` and `<node-label-value>` with a label key and value that already exist on the target nodes:

```yaml theme={null}
nodeAffinities:
  - key: <node-label-key>
    values:
      - <node-label-value>
```

### Install AutoMQ Console

Install the Chart from the AutoMQ OCI registry. The latest recommended stable Chart version is `8.3.7`. Replace `<release-name>` with the Helm release name, `<console-namespace>` with the namespace where the console runs, and `<values-file>` with the path to the values file prepared above:

```bash theme={null}
helm upgrade --install <release-name> \
  oci://automq.azurecr.io/helm/automq-byoc-console-chart \
  --version 8.3.7 \
  --namespace <console-namespace> \
  --values <values-file> \
  --wait
```

Wait for the pod and Ingress to become ready. Replace `<console-namespace>` with the namespace where the console runs:

```bash theme={null}
kubectl get pods,ingress \
  --namespace <console-namespace>
```

Create the DNS record for the hostname in `ingress.host` and point it to the ALB hostname shown in the Ingress status.

### Complete initialization

Open `https://<console-domain-name>` and sign in with the temporary bootstrap credentials:

```text theme={null}
Username: admin
Password: admin
```

<Warning>
  These credentials are public bootstrap credentials. Restrict access to the ALB and change the password immediately after the first sign-in.
</Warning>

The initialization page displays the IAM policy required by AutoMQ Console. Create a customer-managed policy from that JSON and attach it to the same IAM role associated with the ServiceAccount. Return to AutoMQ Console and confirm the authorization.

After the permission check succeeds, you can create and manage AutoMQ instances.

## Next Steps

* **Use AutoMQ via WebUI:** Continue to create instances through the environment console and experience the product features. [Experience AutoMQ▸](/automq-cloud/getting-started/experience-automq)
* **Use AutoMQ via Terraform:** After installing the console, users can manage and use AutoMQ through the AutoMQ Terraform Provider. For using AutoMQ via Terraform, [refer to the documentation](https://registry.terraform.io/providers/AutoMQ/automq/latest/docs).


# Prepare VPC
Source: https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/prepare-vpc

Ensure you have the necessary AWS resources, including VPC, EKS, and S3, ready before installing AutoMQ. This article details the steps and considerations for preparing various cloud resources on AWS.

All components of the AutoMQ Cloud BYOC environment are deployed within your AWS account to ensure data privacy and security. Before installing AutoMQ, you need to prepare various AWS resources in advance. This article describes how to prepare the necessary cloud resources for AutoMQ on the AWS public cloud.

<Info>
  In this document, "AutoMQ," "AutoMQ service provider," and "the AutoMQ service" all refer to AutoMQ HK Limited and its affiliates.
</Info>

## Prepare VPC

You can choose to use an existing VPC when installing AutoMQ, or create a new one for it. For production environments, we recommend checking and adjusting your existing VPC configuration to meet AutoMQ's requirements. For POC testing, creating a new VPC is a quicker option.

### VPC Requirements

To deploy AutoMQ BYOC environment on AWS, your VPC must meet the following conditions:

| **Resource**               | **Requirement Description**                                                                                                                                                                                                                                                                                                                   |
| :------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Control Plane Subnet**   | **1**. Used to deploy the AutoMQ BYOC environment console, which needs to connect to the public network via NAT or other methods.                                                                                                                                                                                                             |
| **Data Plane Subnet**      | **1 or 3**. Used to deploy the AutoMQ data plane cluster. The number of subnets depends on whether you choose a single-AZ or three-AZ deployment model.                                                                                                                                                                                       |
| **S3 Gateway Endpoint**    | **1**. Ensures AutoMQ components access S3 via the AWS internal network, avoiding public network traffic costs. The endpoint's route table must cover all private subnets.                                                                                                                                                                    |
| **EC2 Interface Endpoint** | **1**. Ensures AutoMQ components can call EC2 and Auto Scaling APIs via the AWS internal network.                                                                                                                                                                                                                                             |
| **DNS Configuration**      | **Enabled**. The VPC must have **DNS hostnames** and **DNS resolution** enabled, as AutoMQ relies on Route 53 for DNS resolution within the VPC.                                                                                                                                                                                              |
| **NAT Gateway**            | **1**. If you plan to deploy the AutoMQ data plane cluster on EKS, you must configure a NAT gateway and the correct public outbound routes for the private subnets where the EKS worker nodes are located.<br /><Danger> **Note**: If the NAT gateway is not configured correctly, the AutoMQ cluster installation on EKS will fail.</Danger> |

### Scenario 1: Create a New VPC

If you need to create a new VPC for AutoMQ, we recommend using the **VPC and more** feature in the AWS VPC console.

1. Log in to the [AWS VPC console](https://console.aws.amazon.com/vpc/home).
2. In the left navigation pane, choose **Your VPCs**, then click **Create VPC**.
3. In **VPC settings**, select **VPC and more**.
4. In the **Name tag auto-generation** section, set a clear name for your VPC.
5. Configure the **IPv4 CIDR block**, **Number of Availability Zones**, **Number of public subnets**, and **Number of private subnets**.
6. Depending on your deployment needs, choose whether to create **NAT gateway(s)** and **VPC endpoints**.
   * **NAT Gateway**: If you plan to deploy on EKS, select at least **1 per AZ**.
   * **VPC Endpoints**: Select **S3 Gateway**, which will automatically create a gateway-type S3 endpoint.
7. Ensure that both **Enable DNS hostnames** and **Enable DNS resolution** under **DNS options** are checked.
8. Click **Create VPC**.

After the VPC is created, you also need to manually create an EC2 interface endpoint for it.

1. In the [AWS VPC console](https://console.aws.amazon.com/vpc/home), choose **Endpoints** from the left navigation pane.
2. Click **Create endpoint**.
3. **Name tag**: Name the endpoint.
4. **Service category**: Select **AWS services**.
5. **Services**: Search for and select the `com.amazonaws.<region>.ec2` interface service.
6. **VPC**: Select the VPC you just created.
7. **Subnets**: Select all the private subnets intended for AutoMQ deployment.
8. **Security groups**: Select or create a security group and add an inbound rule to allow HTTPS (443) traffic from your VPC's CIDR.
9. Click **Create endpoint**.

### Scenario 2: Configure an Existing VPC

If you are deploying AutoMQ in an existing VPC, verify that the following configurations meet the requirements.

#### 1. Check DNS Configuration

1. In the [AWS VPC console](https://console.aws.amazon.com/vpc/home), choose **Your VPCs** from the left navigation pane.
2. Select your target VPC and view the **Details** tab.
3. Confirm that the values for **DNS hostnames** and **DNS resolution** are both **Enabled**. If not, click **Actions** -> **Edit VPC settings** to modify them.

#### 2. Check S3 Endpoint

1. In the [AWS VPC console](https://console.aws.amazon.com/vpc/home), choose **Endpoints** from the left navigation pane.
2. Search for and check if there is an S3 endpoint of type **Gateway** (service name `com.amazonaws.<region>.s3`) associated with your VPC.
3. If it doesn't exist, click **Create endpoint** to create one.
   * When creating it, ensure its route table (**Route tables**) is associated with all private subnets that will be used for deploying AutoMQ.

#### 3. Check EC2 Endpoint

1. On the **Endpoints** page, continue to search for and check if there is an EC2 endpoint of type **Interface** (service name `com.amazonaws.<region>.ec2`) associated with your VPC.
2. If it doesn't exist, click **Create endpoint** to create one.
   * **Subnets**: Ensure you select all private subnets that will be used for deploying AutoMQ.
   * **Security groups**: Select or create a security group and add an inbound rule to allow HTTPS (443) traffic from your VPC's CIDR.
   * **Enable DNS name**: Make sure to check **Enable DNS name** to enable Private DNS resolution, ensuring that AutoMQ components can correctly access the EC2 API via private DNS names.

## Prepare Private DNS (Optional)

The AutoMQ cluster exposes service addresses to applications through Private DNS, which relies on private domain name resolution within the VPC. Therefore, a Private DNS must be configured to provide reliable domain name resolution services. You can choose to have the AutoMQ console create the Private DNS Zone automatically or prepare it yourself by following the tutorial below.

### Private DNS Zone Requirements

When configuring Private DNS, the following two key conditions must be met:

| **Condition**<br />                       | **Description**<br />                                                                                                                        | **Importance**<br /> |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| **Associated VPC**<br />                  | The Private DNS must be associated with the VPC where AutoMQ is deployed.                                                                    | **Required**         |
| **Domain Name and TLS Consistency**<br /> | If you need to use the TLS feature later, the domain name of the Private DNS must be consistent with the domain name in the TLS certificate. | **As needed**        |

### Private DNS Configuration Steps

#### Step 1: Check Existing Private DNS

1. Log in to the AWS console and visit the [Route53 console](https://console.aws.amazon.com/route53/).
2. In the left navigation bar, select **Hosted zones**.
3. Check if a Private DNS associated with the target VPC already exists.

#### Step 2: Create or Configure Private DNS

**If a Private DNS already exists:**

1. Select the target Private DNS and click **Details**.
2. Confirm that the **Type** is **Private hosted zone**.
3. Check the **VPCs** section to ensure it includes the VPC where AutoMQ is deployed.
4. Record the domain name information for subsequent TLS feature configuration.

**If you need to create a new Private DNS:**

1. Click **Create hosted zone**.
2. Configure the basic information:
   * **Domain name**: Enter a private domain name (e.g., `internal.example.com`).
   * **Type**: Select **Private hosted zone**.
   * **VPC**: Select the VPC where AutoMQ is deployed and the corresponding region.
3. Click **Create hosted zone**.

#### Step 3: Verify the Configuration

1. On the Private DNS details page, confirm the VPC association status.
2. Check if the domain name configuration meets the planning requirements for the TLS feature.
3. Record the ID and domain name of the Private DNS, which will be provided to the AutoMQ components during subsequent installation.

## Prepare Cluster Security Group (Optional)

During the cluster deployment process, you can opt to use a pre-configured security group for precise network access control. If you choose this method, you must create a security group in advance and configure its inbound rules according to the table below. If you prefer to simplify network configuration, you can choose to have AutoMQ automatically create a new security group with all the necessary rules during cluster creation, in which case you can skip this section.

| Port | Purpose                                                                                        | Required Access Source                                                             | Enablement Condition                                |
| :--- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------- | :-------------------------------------------------- |
| 9093 | Exposes the PLAINTEXT protocol for internal communication among Controller nodes.              | <ul><li>Internal cluster components</li></ul>                                      | Enabled by default                                  |
| 9103 | Exposes the SASL\_PLAINTEXT protocol for internal communication among Controller nodes.        | <ul><li>Internal cluster components</li></ul>                                      | Enabled by default                                  |
| 9092 | Exposes the Kafka service to clients via the PLAINTEXT protocol.                               | <ul><li>Console</li><li>Internal cluster components</li><li>Applications</li></ul> | Enabled by default                                  |
| 9102 | Exposes the Kafka service to clients via the SASL\_PLAINTEXT protocol.                         | <ul><li>Console</li><li>Internal cluster components</li><li>Applications</li></ul> | Enabled by default                                  |
| 9112 | Exposes the Kafka service to clients via the SASL\_SSL protocol.                               | <ul><li>Console</li><li>Internal cluster components</li><li>Applications</li></ul> | Optional                                            |
| 9122 | Exposes the Kafka service to clients via the SSL protocol.                                     | <ul><li>Console</li><li>Internal cluster components</li><li>Applications</li></ul> | Optional                                            |
| 9090 | Provides an endpoint for Prometheus Metrics Exporter to read metrics.                          | <ul><li>Console</li><li>Applications (Optional)</li></ul>                          | Enabled by default                                  |
| 22   | Default SSH port for cluster nodes, exposed to the console node for emergency troubleshooting. | <ul><li>Console</li></ul>                                                          | Enabled by default                                  |
| 8081 | Service port for the managed Schema Registry.                                                  | <ul><li>Internal cluster components</li><li>Applications</li></ul>                 | Optional, required only when TableTopic is enabled. |

<Tip>
  Notes on the access sources mentioned in the table above:

  * **Console**: It is recommended to configure this as the Subnet CIDR where the AutoMQ console is located.
  * **Internal cluster components**: Select the Subnet CIDR chosen when creating the cluster instances.
  * **Applications**: Refers to the source from which applications access the Kafka service. Configure this based on your actual business requirements.
</Tip>

## Configure cross-VPC access

An AutoMQ instance that uses a Route 53 private hosted zone can be resolved only from VPCs associated with that hosted zone. To access the instance from an application in another VPC, complete both of the following tasks:

1. Establish network connectivity between the AutoMQ VPC and the application VPC, for example by using VPC peering or AWS Transit Gateway. Update the relevant route tables and security groups to allow Kafka traffic.
2. Associate the Route 53 private hosted zone used by the AutoMQ instance with the application VPC.

### Find the private hosted zone

Copy the instance endpoint from the AutoMQ Console and identify its domain suffix. In the [Route 53 console](https://console.aws.amazon.com/route53/), choose **Hosted zones**, open the matching private hosted zone, and record its hosted zone ID.

<img alt="Find the Route 53 private hosted zone ID used by the AutoMQ instance." />

### Associate a VPC in the same AWS account

On the private hosted zone details page, edit the VPC associations and add the application VPC and its AWS Region.

<img alt="Associate a VPC in the same AWS account with the private hosted zone." />

### Associate a VPC in a different AWS account

For a cross-account association, first use credentials for the account that owns the private hosted zone to authorize the application VPC:

```bash theme={null}
aws route53 create-vpc-association-authorization \
  --hosted-zone-id <hosted-zone-id> \
  --vpc VPCRegion=<application-vpc-region>,VPCId=<application-vpc-id>
```

Then use credentials for the account that owns the application VPC to create the association:

```bash theme={null}
aws route53 associate-vpc-with-hosted-zone \
  --hosted-zone-id <hosted-zone-id> \
  --vpc VPCRegion=<application-vpc-region>,VPCId=<application-vpc-id>
```

Allow several minutes for the association and DNS records to propagate. Then verify that a host in the application VPC can resolve the AutoMQ endpoint and connect to the configured Kafka listener.

## Prepare FSx File System Security Group (Optional)

If you plan to use AutoMQ in an AWS environment and leverage FSx for NetApp ONTAP as the storage backend for the WAL (Write-Ahead Log), you need to configure a dedicated security group for the FSx file system. This section details the required inbound rules for that security group. If you prefer AutoMQ to automatically create and configure this security group during deployment, you can skip this section.

| Port | Access Source                                                 | Purpose                                                                         | Enablement Condition |
| :--- | :------------------------------------------------------------ | :------------------------------------------------------------------------------ | :------------------- |
| 443  | <ul><li>Console</li><li>Internal cluster components</li></ul> | Allows the AutoMQ cluster and console to call the FSx for ONTAP management API. | Enabled by default   |
| 22   | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |
| 111  | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |
| 635  | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |
| 2049 | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |
| 4045 | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |
| 4046 | <ul><li>Internal cluster components</li></ul>                 | Allows nodes within the AutoMQ cluster to access the FSx service.               | Enabled by default   |

<Tip>
  Notes on the access sources mentioned in the table above:

  * **Console**: It is recommended to configure this as the Subnet CIDR where the AutoMQ Console is located.
  * **Internal cluster components**: Select the Subnet CIDR chosen when creating the cluster instances.
</Tip>

## Prepare EFS File System Security Group (Optional)

If you plan to use AutoMQ in an AWS environment and leverage EFS (Elastic File System) as the storage backend for NFS WAL, you need to configure a dedicated security group for the EFS file system. This section details the required inbound rules for that security group. If you prefer AutoMQ to automatically create and configure this security group during deployment, you can skip this section.

| Port | Access Source                                 | Purpose                                                                                    | Enablement Condition |
| :--- | :-------------------------------------------- | :----------------------------------------------------------------------------------------- | :------------------- |
| 2049 | <ul><li>Internal cluster components</li></ul> | Allows nodes within the AutoMQ cluster to access the EFS file system via the NFS protocol. | Enabled by default   |

<Tip>
  Notes on the access sources mentioned in the table above:

  * **Internal cluster components**: Select the Subnet CIDR chosen when creating the cluster instances.
</Tip>


# Install AutoMQ on Google Cloud
Source: https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/google-cloud/install-automq-on-gcp

This article describes how to install the AutoMQ console on Google Cloud with Docker and complete the initial permission configuration before deploying to GKE.

## Overview

To use an AutoMQ BYOC environment on Google Cloud, you first need to install the AutoMQ console. The console is a web application deployed in your Google Cloud project. You use it to create, manage, and monitor AutoMQ clusters.

Starting from 8.x, the AutoMQ console is distributed and started as a Docker image. New environments should use the Docker installation method.

## Prerequisites

Before you begin installing the AutoMQ console, prepare the required Google Cloud network resources.

* **VPC environment**: The AutoMQ console and GKE cluster need to run in your VPC, usually the same VPC as your Kafka client applications or a VPC that can reach them.

## Steps

### Step 1: Register your environment on the AutoMQ website

First, register your deployment environment on the AutoMQ website to obtain the environment metadata and Docker startup command.

1. Visit [AutoMQ Cloud](https://account.automq.cloud) and log in to your account.
2. On the console page, click **Create Environment**.
3. Follow the wizard to enter the environment name, cloud provider, region, and related information.
4. After creation, copy the Docker startup command generated by the installation wizard.

### Step 2: Prepare and deploy the console VM

The AutoMQ console needs to run on a standalone Google Cloud VM. The VM should be in the VPC used by the GKE cluster that you will deploy later.

On GCP, the AutoMQ console running on the VM calls Google Cloud APIs through the Google Service Account bound to that VM. To avoid using the default Compute Engine service account, create a dedicated service account in the service project where the console runs before you create the VM. This document refers to it as the console Google Service Account.

The console Google Service Account is the control plane identity for the AutoMQ console. During permission initialization, you grant this service account the permissions needed to manage GKE, GCS, Cloud DNS, IAM bindings, and other cloud resources required by AutoMQ. At this point, you only need to create the service account and bind it to the VM when you create the VM. You don't need to grant the full AutoMQ permissions in advance. After the console starts, the initialization page shows the permissions that need to be granted to this service account and guides you through authorization and verification.

If you use `gcloud` to create the console Google Service Account, use the following example:

```bash theme={null}
PROJECT_ID="<service-project-id>"
CONSOLE_SA_ID="automq-console"

gcloud iam service-accounts create "${CONSOLE_SA_ID}" \
  --project="${PROJECT_ID}" \
  --display-name="AutoMQ console"
```

#### Basic requirements

For stable console operation, we recommend the following VM configuration:

1. **Instance specifications**
   * At least **2 vCPUs** and **8 GB memory**.
   * At least **20 GiB** system disk.
   * Use a separate data disk or a stable data directory for console data.

2. **Network location**

   * The console VM should be in a VPC and subnet reachable from the GKE cluster.
   * If you use Shared VPC, confirm the permission boundary between the service project where the console runs and the host project where the VPC resides.
   * The console service listens on port `8080` by default. Allow your operations access source to reach port `8080` on the console VM, and restrict the source CIDR ranges to your office network, bastion host, or another approved access path.

   If you use `gcloud` to create the firewall rule, use the following command as a reference. It narrows the target to VMs bound to the console Google Service Account by using `--target-service-accounts`. `NETWORK_PROJECT_ID` is the project that owns the VPC and firewall rules. In non-Shared VPC environments, it is usually the service project.

   ```bash theme={null}
   NETWORK_PROJECT_ID="<network-project-id>"
   VPC_NAME="<vpc-name>"
   CONSOLE_SA_EMAIL="<console-service-account-email>"
   SOURCE_CIDR="<operator-source-cidr>"

   gcloud compute firewall-rules create "automq-console-allow-8080" \
     --project="${NETWORK_PROJECT_ID}" \
     --network="${VPC_NAME}" \
     --direction=INGRESS \
     --action=ALLOW \
     --source-ranges="${SOURCE_CIDR}" \
     --target-service-accounts="${CONSOLE_SA_EMAIL}" \
     --rules=tcp:8080
   ```

3. **Internet access or egress capability**
   * The console VM needs to pull Docker images and synchronize required metadata. Use an external IP, Cloud NAT, or another egress path.
   * If the console VM doesn't use an external IP and doesn't have Cloud NAT or another egress path, enable Private Google Access on the subnet that hosts the console VM so it can reach Cloud Storage, GKE, IAM, Cloud DNS, and other Google APIs. This configuration only covers Google API access. If the VM needs to reach external image registries or other non-Google endpoints, it still needs an external IP, Cloud NAT, or another egress path. For details, see the Google Cloud documentation [Configure Private Google Access](https://docs.cloud.google.com/vpc/docs/configure-private-google-access).

4. **VM service account**
   * When you create the VM, bind the console Google Service Account prepared earlier to the VM. This is the same service account you created at the beginning of this step, and the later permission initialization also applies to this service account.
   * The operator who creates the VM needs the `iam.serviceAccounts.actAs` permission on this service account to bind it to the VM.
   * Set the VM access scope to allow full access to all Cloud APIs (`https://www.googleapis.com/auth/cloud-platform`). Google Cloud's default access scopes only include limited permissions such as read-only Cloud Storage, Logging, and Monitoring.
   * Even if the console Google Service Account has the required IAM roles, a restrictive VM access scope can still block calls to GKE, IAM, Cloud DNS, or write operations to GCS. For details, see the Google Cloud documentation [Create a VM that uses a user-managed service account](https://docs.cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances).

#### Install and start Docker

Log in to the console VM, then install and start Docker. The following commands use Debian / Ubuntu as an example:

```bash theme={null}
sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo docker version
```

#### Start the AutoMQ console

After the VM is ready, copy the Docker startup command generated by the installation wizard in Step 1 and start the AutoMQ console.

<Tip>
  The command generated by the installation wizard already contains the environment ID, cloud provider, region, authentication parameters, Docker image address, and related information. Do not assemble these parameters manually.
</Tip>

After you run the startup command, check the container status and logs:

```bash theme={null}
sudo docker ps
sudo docker logs -f "<container-name-or-id>"
```

After the Docker container runs correctly and port 8080 is reachable, you can access the AutoMQ console in your browser.

### Step 3: Configure console permissions and complete initialization

After the AutoMQ console starts successfully, the first login may ask you to initialize permissions. Follow the GCP permission instructions on the page and grant the required permissions to the console Google Service Account.

Permission initialization allows the AutoMQ console to create and manage cloud resources. In GCP, the console VM calls Google Cloud APIs through the bound console Google Service Account. Follow the page instructions and grant the listed permissions to this service account. The console verifies those permissions during initialization.

GCP permissions can be granted at different locations depending on the capability. In non-Shared VPC environments, the service project and host project are usually the same project. In Shared VPC environments, GKE, the console VM, buckets, and Google Service Accounts usually reside in the service project, while VPC, subnet, and firewall resources reside in the host project. Some bucket and Cloud DNS permissions may also need to be granted on the target resource. The following table helps you understand the bindings shown on the initialization page. Grant permissions based on the content generated on that page.

| Capability                                 | Recommended binding location                                  | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------------------ | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Console system data management             | Bucket shown on the initialization page                       | The console reads and writes system logs, metrics, and other runtime data, and maintains the bucket IAM policy. Recommended role: `roles/storage.admin`.                                                                                                                                                                                                                                                                                                                            |
| GKE and network discovery                  | Service project and host project                              | The console reads GKE Standard clusters, node pools, VPCs, and subnets. In Shared VPC environments, grant VPC and subnet read permissions in the host project. Grant `roles/container.admin` in the service project, and `roles/compute.networkViewer` in the project that owns the network.                                                                                                                                                                                        |
| Data bucket management                     | Service project                                               | The console creates managed data buckets based on instance configuration, or discovers and validates existing data buckets. Recommended role for managed bucket mode: `roles/storage.admin`; existing bucket mode usually requires `roles/storage.bucketViewer`.                                                                                                                                                                                                                    |
| Cloud DNS management                       | Service project; also host project in Shared VPC environments | The console creates or reads Cloud DNS private managed zones, and maintains zone IAM policies in managed DNS mode. Managed DNS mode requires `roles/dns.admin` and creates a custom role named `automqConsoleDnsZoneIamPolicyAdmin`, which contains `dns.managedZones.getIamPolicy` and `dns.managedZones.setIamPolicy`. In Shared VPC environments, the host project also needs `roles/dns.admin` for private zone binding. Existing DNS mode usually requires `roles/dns.reader`. |
| Instance Google Service Account management | Service project                                               | The console creates the instance Google Service Account and configures instance-level IAM bindings in managed identity mode. In existing identity mode, the console reads the target Google Service Account. Managed identity mode requires `roles/iam.serviceAccountAdmin`, `roles/iam.roleAdmin`, and `roles/resourcemanager.projectIamAdmin`; existing identity mode usually requires `roles/iam.serviceAccountViewer`.                                                          |

After granting the permissions, return to the AutoMQ console and click the verification or initialization action. After verification succeeds, you can enter the AutoMQ console and continue configuring GKE.

## Next steps

* **Deploy to GKE**: After installing the console and completing permission initialization, see [Deploy to Google Cloud GKE](/automq-cloud/deploy-automq-on-kubernetes/deploy-to-google-cloud-gke).
* **Use AutoMQ via WebUI**: Continue to create instances through the environment console and experience the product features. [Experience AutoMQ▸](/automq-cloud/getting-started/experience-automq)
* **Use AutoMQ via Terraform**: After installing the console, you can manage and use AutoMQ through the AutoMQ Terraform Provider. For Terraform usage, see the [Terraform Provider documentation](https://registry.terraform.io/providers/AutoMQ/automq/latest/docs).


# Azure
Source: https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/install-env-on-azure

Install AutoMQ on Azure with seamless Kafka compatibility. Enjoy cloud-native scalability and enhanced data privacy with BYOC support for cost-efficient operations.

Refer to [Overview▸](/automq-cloud/getting-started/overview); before using AutoMQ Cloud, you need to set up the environment first. This article outlines how to install the BYOC environment from Azure.

If you're already familiar with the requirements and steps for installing AutoMQ on Azure, you can click <a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAutoMQ%2Fautomq-labs%2Frefs%2Fheads%2Fmain%2Fbyoc-examples%2Fsetup%2F%2Fazure%2Farm%2Fmain.json/uiFormDefinitionUri/https%3A%2F%2Fraw.githubusercontent.com%2FAutoMQ%2Fautomq-labs%2Frefs%2Fheads%2Fmain%2Fbyoc-examples%2Fsetup%2F%2Fazure%2Farm%2FuiFormDefinition.json">Install AutoMQ From Marketplace</a>.

<Info>
  In this document, references to the AutoMQ product service provider, AutoMQ service provider, and AutoMQ all pertain to AutoMQ HK Limited and its subsidiaries.
</Info>

## Operational Flow

### Step 1: Prepare an Azure virtual network (VNet)

The AutoMQ BYOC environment is deployed within the user's Virtual Network to ensure data privacy and security. When installing the AutoMQ environment, users must specify the Virtual Network and subnet information, and ensure that the Virtual Network meets the following conditions:

* **Subnet CIDR requirements:** The subnet must provide at least 512 IP addresses. A subnet CIDR block of `/23` or larger is recommended so that the AutoMQ data plane has sufficient addresses for deployment and scaling.

* **Configure Public NAT Gateway for Access to Public Services:** AutoMQ requires access to public services like image repositories. Therefore, the subnet where AutoMQ resides needs to be configured with a public NAT gateway.

* **Configure Storage Service Endpoint:** AutoMQ requires access to the Azure Storage service, which necessitates configuration of the Service Endpoint. This allows access to object storage services through a private network, thereby avoiding unnecessary public traffic costs. Please refer to the appendix at the end of this document for configuration methods.

<Tip>
  Users can refer to the appendix section of this document for guidance on configuring Virtual Networks and Subnets.
</Tip>

### Step 2: Install AutoMQ Using ARM Templates.

1. **Visit** [AutoMQ Installation Link](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAutoMQ%2Fautomq-labs%2Frefs%2Fheads%2Fmain%2Fbyoc-examples%2Fsetup%2F%2Fazure%2Farm%2Fmain.json/uiFormDefinitionUri/https%3A%2F%2Fraw.githubusercontent.com%2FAutoMQ%2Fautomq-labs%2Frefs%2Fheads%2Fmain%2Fbyoc-examples%2Fsetup%2F%2Fazure%2Farm%2FuiFormDefinition.json)**,** to open the installation interface. The AutoMQ Cloud BYOC environment is installed on Azure by default through an ARM template.

2. **Refer to the parameter description below to set the appropriate Basics parameter values.**

| **Set Parameter**<br />  | **Value Explanation**<br />                                                                                                                                                                                                         |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Subscription**<br />   | <ul><li>Description: Select an Azure subscription, all cloud resources used by AutoMQ will be billed using this subscription.</li><li>Recommendation: It is advised to keep it consistent with the application.</li></ul>           |
| **Resource group**<br /> | <ul><li>Description: Choose the resource group assigned to AutoMQ. It is recommended to place all subsequent cloud resources associated with AutoMQ in the same resource group.</li></ul>                                           |
| **Region**<br />         | <ul><li>Description: Select the target region for the AutoMQ deployment, which must be consistent with the region used by your AutoMQ Producer and Consumer applications. Cross-region access to AutoMQ is not supported.</li></ul> |

<img />

3. **Refer to the parameter descriptions below and set appropriate values for the Virtual Machine Configuration parameters.**

| **Set Parameter**<br />         | **Value Explanation**<br />                                                                                                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **VM Size**<br />               | <ul><li>Description: Select the VM instance size used by the AutoMQ console. It requires at least 2vCPU and 8GB of memory. It is recommended to keep the default setting.</li></ul> |
| **Admin Username**<br />        | <ul><li>Description: Set the host username for subsequent SSH logins to the console VM.</li></ul>                                                                                   |
| **SSH public key source**<br /> | <ul><li>Description: Set the SSH Key-Pair used for subsequent SSH login to console VM. Supports creating a new Key or using an existing one.</li></ul>                              |

<img />

4. **Refer to the parameter descriptions below to appropriately configure Network Configuration parameter values.**

| **Set Parameter**<br />     | **Value Explanation**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Public IP Address**<br /> | <ul><li>Description: Set the public IP used for providing services externally via the AutoMQ console. Users can access the AutoMQ console through this public IP later on.</li><li>Optional parameter values:</li><ul><li>**None:** No need to assign a public IP; subsequent access to the console will directly use the private IP address. This mode is suitable for enterprises with VPN, dedicated line, etc., connecting to the Virtual Network.</li><li>**(New) xxx:** Create a new public IP; the name should be set manually.</li><li>**Select an existing public IP:** Choose a public IP that has already been created and is available.</li></ul></ul> |
| **Virtual Network**<br />   | <ul><li>Description: Set the target Virtual Network where the AutoMQ console will be deployed. It needs to be consistent with Producer and Consumer applications.</li><li>Optional parameter values:</li><ul><li>**Select an existing Virtual Network:** Choose the private network where the application resides. This mode is suitable for production scenarios.</li></ul></ul>                                                                                                                                                                                                                                                                                  |
| **Subnet**<br />            | <ul><li>Description: Set the target subnet for AutoMQ console that is about to be deployed.</li><li>Optional parameter values:</li><ul><li>Subnet must be set to select an existing Subnet.</li></ul></ul><Tip><br />Refer to the appendix section of this document, the subnet needs to configure a public NAT gateway, as the AutoMQ console requires access to public networks.<br /></Tip><br />                                                                                                                                                                                                                                                               |

<img />

5. **Refer to the parameter descriptions below to set suitable Storage Configuration parameter values.**

| **Set Parameter**<br />   | **Value Explanation**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Storage Account**<br /> | <ul><li>Description: Set up the Storage Account used by AutoMQ Ops Bucket. Subsequently, AutoMQ will collect system logs, Metrics, and other data to upload to this Storage Account for alerts and analysis.</li><li>Optional parameter values:</li><ul><li>**(New) xxx:** Create a new Storage Account.</li><li>**Select existing value**: Choose an already created, available Storage Account.</li></ul></ul><Tip><br />For the working principle of the Ops Bucket, refer to [Overview▸](/automq-cloud/manage-environments/manage-environment-ops-authing/overview).<br /></Tip><br /> |

<img />

6. **Preview the configuration and create it. View the initial account passwords and console address output on the console.**

| **Output**<br />                                | **Output Description**<br />                                                                                                                                               |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Automq Byoc Endpoint**<br />                  | <ul><li>The URL address of the AutoMQ BYOC environment console, which users can access via a browser or through APIs and Terraform to interact with the service.</li></ul> |
| **Automq Byoc Initial Username**<br />          | <ul><li>The initial username for the environment console.</li></ul>                                                                                                        |
| **Automq Byoc Initial Password**<br />          | <ul><li>The initial password for the environment console. Users are required to change it immediately upon first login.</li></ul>                                          |
| **Automq Byoc Managed Identity ClientID**<br /> | <ul><li>The Managed Identity ClientID bound to the AutoMQ console, which the console uses to authenticate and access cloud resources.</li></ul>                            |

<img />

7. Use the information above to sign into the AutoMQ console.

<img />

### Step 3: **Complete Operations Authorization for BYOC Environment**

The BYOC environment is deployed in your virtual network (VNet), which isolates application data within your network. The environment also generates operational logs, metrics, and diagnostic data. After installation, follow [Operations authorization overview](/automq-cloud/manage-environments/manage-environment-ops-authing/overview) to grant the permissions required for AutoMQ to monitor service health and perform managed recovery operations.

## Next Steps

AutoMQ only supports deployment on an AKS cluster in Azure. Therefore, once the environment installation is complete, users must first prepare the AKS cluster and then configure it in the AutoMQ console to deploy the AutoMQ cluster to AKS.

Prepare AKS and complete the subsequent deployment documentation. Please refer to [Deploy to Azure AKS▸](/automq-cloud/deploy-automq-on-kubernetes/deploy-to-azure-aks).

## Appendix

### Configure Virtual Network and Subnet

#### Configure the Subnet, Ensuring There Are Enough IP Addresses Available.

When you create the subnet in the Azure portal, configure a CIDR block that provides at least 512 IP addresses. A `/23` or larger subnet is recommended.

<img />

#### Configuring a Subnet to Enable NAT Gateway for Accessing Public Services

Users can enable the NAT Gateway during the creation or editing of a Subnet in the Virtual Network console.

<img />

#### Configure a Subnet to Enable a Storage Service Endpoint for Accessing the Blob Storage Service.

AutoMQ requires access to Azure Storage Service; therefore, all Subnets deploying AutoMQ must have the Storage Service Endpoint enabled. Refer to the diagram below and select `Microsoft.Storage`.

<img />


# OCI
Source: https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/oci-install-automq



# Install AutoMQ on OCI

This document will guide you through the process of deploying AutoMQ for Kafka on Oracle Cloud Infrastructure (OCI). We will cover environment preparation and installation steps to ensure a smooth deployment.

## 1. Prepare a Virtual Cloud Network (VCN)

First, we need a correctly configured Virtual Cloud Network (VCN) to host the Kubernetes cluster and AutoMQ instances.

Please ensure your VCN meets the following requirements:

* **Same Region**: The VCN must be in the same OCI region where you plan to deploy the OKE cluster and your business applications.
* **Subnets**:
  * The VCN must contain at least two subnets, ideally one Public Subnet and at least one Private Subnet.
  * **Public Subnet**: Used for deploying resources that require public internet access, such as Load Balancers.
  * **Private Subnet**: Used for deploying OKE worker nodes and AutoMQ instances to enhance security.
  * Ensure each subnet has sufficient available IP addresses to meet the scaling needs of your OKE cluster and applications.
  * Reference: [Managing VCNs and Subnets](https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingVCNs.htm)
* **Internet Gateway**: Attach an Internet Gateway to your Public Subnet and configure appropriate routing rules to allow traffic from the internet.
  * Reference: [Managing Internet Gateways](https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingIGs.htm)
* **NAT Gateway**: Attach a NAT Gateway to your Private Subnet and configure appropriate routing rules. This allows resources within the Private Subnet (e.g., OKE nodes) to access the internet for downloading images and dependencies, without allowing direct internet access to these resources.
  * Reference: [Managing NAT Gateways](https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/manage-nat.htm)
* **Service Gateway**: Enable a Service Gateway for your VCN and configure routing rules to allow resources within the private network to access OCI Object Storage via OCI's backbone network. This ensures secure and high-performance data transfer.
  * Reference: [Access to Oracle Services: Service Gateway](https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/servicegateway.htm)

## 2. Prepare Object Storage

AutoMQ offloads streaming data to cloud provider object storage for disaggregated storage and compute, achieving significant cost savings. In an OCI environment, each AutoMQ cluster requires two separate object storage buckets:

* **Data Bucket**: Used for storing Kafka message data. Recommended naming convention: `automq-data-<cluster-name>`.
* **Ops Bucket**: Used for storing operational data such as cluster logs and metrics. Recommended naming convention: `automq-ops-<cluster-name>`.

Please create these two buckets in the same OCI region as your OKE cluster.

* Reference: [Managing Buckets in Object Storage](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingbuckets.htm)

## 3. Prepare the Kubernetes Cluster (OKE)

AutoMQ is a cloud-native messaging and streaming system designed to run entirely on Kubernetes. Therefore, you need to prepare an Oracle Kubernetes Engine (OKE) cluster.

<Tip>
  **Note**: You can refer to [this Terraform script](https://github.com/AutoMQ/automq-labs/tree/main/byoc-examples/setup/kubernetes/oci/terraform/oke) to create a compliant OKE cluster with one click.
</Tip>

### 3.1. Create OKE Cluster

When creating your OKE cluster, pay attention to the following key parameter configurations:

* **Kubernetes Version**: It is recommended to choose a recent stable version, for example, `v1.33.x`. Please refer to the official OCI documentation for the latest supported versions.
* **Network Mode**:
  * Select **"VCN-Native Pod Networking"** (NPN) network mode.
  * In the network configuration, select the VCN you prepared in the first step and configure the Kubernetes API Endpoint and Worker Nodes subnets to the correct private subnets.
  * Configure Network Security Groups (NSGs) to allow necessary traffic into and out of the cluster.
* Reference: [Creating a Kubernetes Cluster](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingclusterusingoke.htm)

### 3.2. Create Node Pool

For optimal performance and resource isolation, it is recommended to create a dedicated node pool for AutoMQ.

* **Instance Type (Shape)**: We recommend `VM.Standard.E5.Flex` shapes, which offer flexible CPU and memory ratios.
* **Resource Configuration**:
  * **OCPUs**: 2
  * **Memory**: 16 GB
* **Node Count (Size)**: Select at least 3 nodes to ensure high availability.

## 4. Deploy AutoMQ with Helm Chart

Once all the above preparatory work is complete, you can proceed to deploy AutoMQ BYOC using Helm Chart. For detailed installation steps, parameter configurations (such as choosing between S3WAL and EBSWAL modes), and advanced configurations, refer to our standard Helm Chart deployment documentation.

**[Deploy AutoMQ BYOC with Helm Chart▸](/automq-cloud/appendix/deploy-automq-enterprise-via-helm-chart)**


# Overview
Source: https://docs.automq.com/automq-cloud/getting-started/overview

Register an account, install an AutoMQ BYOC environment, and create an AutoMQ instance.

This guide provides the workflow for using AutoMQ Cloud to install an AutoMQ BYOC (Bring Your Own Cloud) environment and create an Apache Kafka-compatible instance in your cloud account.

<Info>
  Throughout this document, "AutoMQ" refers to AutoMQ HK Limited and its affiliates.
</Info>

## Prerequisites

Before you begin, ensure your cloud environment meets the following requirements:

* **Cloud Infrastructure:** Prepare the provider-specific cloud network used by your applications, such as an AWS VPC, a Google Cloud VPC network, or an Azure VNet. The installation also requires access to the object storage and compute services specified by the cloud provider guide.

* **Cloud Account:** Use an active public cloud account whose identity has the permissions required to create and manage the documented network, storage, compute, and identity resources.

## Getting Started Workflow

The process of getting started with AutoMQ Cloud involves three steps:

1. **Register an AutoMQ Cloud account:** Go to the [AutoMQ Cloud sign-up page](https://account.automq.cloud) and create an account.

2. **Create and install an Environment:** Create the logical management boundary for your cloud infrastructure, network, and AutoMQ resources, and then install the AutoMQ Console in your cloud account. For more information, see [Environments overview](/automq-cloud/manage-environments/overview).

3. **Create an Instance:** After the Environment is installed and authorized, create an AutoMQ Instance. An Instance corresponds to an Apache Kafka® cluster. After the Instance is running, use its bootstrap server endpoint to connect Kafka clients and produce and consume messages.


# Manage Environment License
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-license

Obtain and activate a prepaid license for an AutoMQ BYOC environment.

<Tip>
  This document applies only to the prepaid billing model. If you are using an AWS environment, we recommend [Usage-Based Billing](/automq-cloud/subscriptions-and-billings/byoc-env-billings/usage-based-billing), which does not require License management.
</Tip>

Use this document to manage the environment license when you select prepaid billing for AutoMQ BYOC (Bring Your Own Cloud).

<Info>
  In this document, AutoMQ product service provider, AutoMQ service provider and AutoMQ each specifically refer to AutoMQ HK Limited.
</Info>

## Obtaining a License

When a new user installs an AutoMQ BYOC environment for the first time, the environment includes a free trial license by default.

<Tip>
  The default free trial license is valid for 14 days and provides 24 AKUs. For the definition of AKU, see [Billing Instructions for BYOC](/automq-cloud/subscriptions-and-billings/byoc-env-billings/billing-instructions-for-byoc).
</Tip>

Formal production environment users can purchase a license through the following channels:

<Tabs>
  <Tab title="Offline Contract Procurement License">
    Users can contact the AutoMQ technical team to purchase a license through [Obtaining Services▸](/automq-cloud/support/obtaining-services).
  </Tab>

  <Tab title="AWS Marketplace Purchase License">
    The AutoMQ BYOC subscription license is available on **AWS Marketplace** as [AutoMQ for Kafka (BYOC Monthly Subscription License)](https://aws.amazon.com/marketplace/pp/prodview-5jgdxi4mhl4rg). Use the following procedure:

    1. AWS users can visit Marketplace, search for AutoMQ or Kafka, locate [AutoMQ for Kafka (BYOC Monthly Subscription License)](https://aws.amazon.com/marketplace/pp/prodview-5jgdxi4mhl4rg), select the appropriate specification version and subscription period, then activate the subscription.

    <img />

    2. Choose the desired subscription quantity and duration.

    <img />

    3. After completing the purchase, click **Setup Product** to be redirected to the AutoMQ Contact page. Complete the License information and reach out to AutoMQ technical support to obtain the product key.

    <img />
  </Tab>

  <Tab title="Azure Marketplace Purchase License">
    Azure Marketplace provides the AutoMQ BYOC prepaid license as a SaaS offer. Azure Marketplace is the purchasing and billing channel; the AutoMQ control plane and data plane remain in your cloud environment.

    1. Sign in to Azure and verify that your account can manage the applicable Billing Profile. For role details, see [Understand Microsoft Customer Agreement administrative roles in Azure](https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/understand-mca-roles).
    2. Open the [AutoMQ BYOC offer in Azure Marketplace](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/automq.automq-byoc-license/selectionMode~/false/resourceGroupId//resourceGroupLocation//dontDiscardJourney~/false/selectedMenuId/home/launchingContext~/%7B%22galleryItemId%22%3A%22automq.automq-byoc-licenseaku%22%2C%22source%22%3A%5B%22GalleryFeaturedMenuItemPart%22%2C%22VirtualizedTileDetails%22%5D%2C%22menuItemId%22%3A%22home%22%2C%22subMenuItemId%22%3A%22Search%20results%22%2C%22telemetryId%22%3A%22de8e32d0-4735-4d9d-bc96-23813e277828%22%7D/searchTelemetryId/faa4a435-a76d-4e3b-ac22-138f431b9a50).
    3. Choose **Subscribe**, select the Resource Group, enter a subscription name, and configure renewal according to your procurement policy.
    4. After Azure creates the subscription, choose **Set up product account** and complete the requested account synchronization.
    5. Contact [AutoMQ Support](/automq-cloud/support/obtaining-services) with the subscription information and Environment Installation ID to obtain the product key.

    For negotiated pricing or discounts, request an Azure Marketplace Private Offer from AutoMQ before subscribing.
  </Tab>
</Tabs>

## Using the License to Activate the Environment

After installing the user BYOC environment and purchasing the License, refer to the documentation below for environment activation.

### Operational Steps

1. Log in to the BYOC environment console and navigate to the **Settings** page.

<img />

2. Gather the environment **Installation ID** and configure the license information purchased in the previous step. Contact AutoMQ technical personnel to obtain the activation **product key**.

The product key for environment activation will be in a format similar to the following:

```text theme={null}

eyJpbnN0YWxsSWQiOiJmMGQzZDRlNmJmMjhjNTBjNGM3M2RjNzdkZTUzYTZiOSIsImVudmlyb25txxxxxxxdXRvTVEgZm9yIEthZmthICggQllPQyBNb250aGx5IFN1YnNjcmlwdGlvxxxxaWNlbnNlICkifX19

```

3. Return to the **Settings** page and click **Update product license**. Enter the product key to complete environment activation. You can then proceed to create instances normally.

<img />


# Overview
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/overview

Ensure seamless operations and stability of your AutoMQ Cloud BYOC environment by authorizing read and write permissions for system logs and updates. Discover the steps for different cloud providers.

# Manage Environment Ops Authing

When using the AutoMQ Cloud BYOC product, system logs, metrics, and other data are generated at the underlying level of the BYOC environment. The AutoMQ service provider requires the user to provide corresponding operational authorization for system stability monitoring and self-healing maintenance operations.

<Info>
  In this article, the terms AutoMQ product service provider, AutoMQ service provider, and AutoMQ specifically refer to AutoMQ HK Limited.
</Info>

## Authorization Principle

AutoMQ's operational authorization includes three scenarios:

1. **Scenario 1: System monitoring and alerts, requiring read permissions.** During the operation of the AutoMQ Cloud BYOC environment, system logs, inspection logs, and system metrics data are generated by the environment console and the AutoMQ data plane cluster. This data is automatically uploaded to the operational bucket specified when the user creates the environment. The AutoMQ service provider needs the user’s cloud account to authorize read permissions for this operational bucket. This allows the AutoMQ operational platform to monitor the stability and perform self-healing of the user’s cluster.

2. **Scenario 2: Subscription license dynamic updates and new version updates, requiring write permissions.** When the subscription license of the AutoMQ BYOC environment triggers renewal, scaling, or new version releases, the control components within the environment need to perceive the new subscription information and the metadata of the new version. At this point, the AutoMQ service provider writes the new information to the operational bucket, and the environment console dynamically loads it.

3. **Scenario 3: Emergency Trouble Shooting scenario requiring remote access to the BYOC environment.** The AutoMQ BYOC environment supports Ops Tunnel, which allows users to create secure, trusted, and auditable operational channels. This grants AutoMQ technical staff access to the BYOC environment within a limited scope for locating and analyzing complex issues.

<img />

## Ops Bucket Authorization

The authorization operation is completed through the bucket authorization policy provided by the cloud provider’s object storage. The user, as the bucket owner, grants the AutoMQ service provider's cloud account the permission to read the specified operational bucket.

**The details of the required read permissions are as follows:**

| **Cloud Providers**<br /> | **AutoMQ Service Provider Cloud Account ID**<br />                                                                                              | **Granted Permissions List**<br />                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Alibaba Cloud<br />       | 1431115939942888<br />                                                                                                                          | <ul><li>Authorized Resource: Maintenance Bucket for the specified environment</li><li>Authorized Operations: Read and write permissions for the bucket.</li></ul>                                                                                                                                                                                                                                                     |
| AWS<br />                 | 381492316447<br />                                                                                                                              | <ul><li>Authorized Resource: Maintenance Bucket for the specified environment</li><li>Authorized Operations:</li><ul><li>s3:GetObject</li><li>s3:GetObjectVersion</li><li>s3:GetObjectAcl</li><li>s3:GetObjectTagging</li><li>s3:GetObjectVersionAcl</li><li>s3:GetObjectVersionTagging</li><li>s3:ListBucket</li><li>s3:ListBucketVersions</li><li>s3:ListBucketMultipartUploads</li><li>s3:PutObject</li></ul></ul> |
| GCP<br />                 | [automq-public-ops-authing@automq-public.iam.gserviceaccount.com](mailto:automq-public-ops-authing@automq-public.iam.gserviceaccount.com)<br /> | <ul><li>Storage Object User role</li></ul>                                                                                                                                                                                                                                                                                                                                                                            |
| Azure<br />               | 123d6e96-84f4-4569-82ec-c25edb2dc696 (App ID)<br />                                                                                             | <ul><li>Authorized Resource: Ops Bucket Blob Container for the specified environment</li><li>Authorized Role: Storage Blob Data Contributor</li></ul>                                                                                                                                                                                                                                                                 |

<Tip>
  Environmental Maintenance Authorization Statement:

  The maintenance bucket must be isolated from the data bucket and other application buckets used, storing only system logs and metrics data. It does not contain user business messages or other data, ensuring no data security risk.
</Tip>

### Configuration Guide

Since the control panels of object storage products vary among different cloud providers, the steps for granting operation authorization also differ. Please refer to the documentation listed below for detailed steps:

* [Set Ops Authing from AWS▸](/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-aws)

* [Set Ops Authing from Google Cloud▸](/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-google-cloud)

* [Set Ops Authing from Azure▸](/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-azure)


# Set Ops Authing from AWS
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-aws

Configure delegated ops authorization for AutoMQ cloud service on AWS S3 with detailed steps and policies. Enjoy cloud-native scalability and Kafka compatibility.

Refer to [Manage Environment Ops Authing▸](/automq-cloud/manage-environments/manage-environment-ops-authing/overview), when using the AutoMQ Cloud BYOC product, the AutoMQ service provider needs the user to provide the appropriate ops authorization. This document explains how to configure delegated ops authorization in AWS S3.

<Tip>
  When installing the AutoMQ console in an AWS environment, operational authorization is usually completed automatically by default.

  * **If your version is >= 8.0**: manual authorization is usually not required.
  * **If your version is \< 8.0**: manual authorization must be performed according to the guidelines in this document.
</Tip>

### AWS Environment Authorization

The AWS S3 console provides the Bucket Policy authorization feature. The operation steps are referenced in the document below.

#### Console Operation Steps

Log in to the cloud account that owns the Ops Bucket or a sub-account with authorization capabilities, and go to the [S3 Console](https://ap-southeast-1.console.aws.amazon.com/s3/buckets?region=ap-southeast-1):

1. **Go to the Bucket list page and find the target Bucket.** The user visits the S3 console, clicks on **Bucket List** on the left navigation bar, and enters the list page. Search for the Ops Bucket configured for the current environment and click on the details.

<img />

2. **Go to the Permissions menu and find the Bucket Policy.** Click **Edit** to open the authorization form.

<img />

3. **Fill in Authorization Information and Create a Bucket Authorization Policy.** Follow the instructions below to fill in the necessary authorization information, click OK, and complete the creation of the authorization policy.

   * Authorized Resource: Select the specified Ops Bucket.

   * Authorized User: Refer to [Overview▸](/automq-cloud/manage-environments/manage-environment-ops-authing/overview), and fill in the AutoMQ service provider cloud account ID "381492316447".

   * Authorized Operation: Refer to the policy below\..

```json theme={null}
{
        "Version": "2012-10-17",
        "Statement": [
                {
                        "Effect": "Allow",
                        "Principal": {
                                "AWS": "arn:aws:iam::381492316447:root"
                        },
                        "Action": [
                                "s3:GetObject",
                                "s3:GetObjectVersion",
                                "s3:GetObjectAcl",
                                "s3:GetObjectTagging",
                                "s3:GetObjectVersionAcl",
                                "s3:GetObjectVersionTagging",
                                "s3:ListBucket",
                                "s3:ListBucketVersions",
                                "s3:ListBucketMultipartUploads",
                                "s3:PutObject"
                        ],
                        "Resource": [
                                "arn:aws:s3:::{Ops Bucket Name}",
                                "arn:aws:s3:::{Ops Bucket Name}/*"
                        ]
                }
        ]
}
```

<img />


# Set Ops Authing from Azure
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-azure

Configure cross-tenant operational authorization for AutoMQ on Azure by creating a service principal and granting Storage Blob Data Contributor access to the Ops Bucket.

As described in [Manage Environment Ops Authing▸](/automq-cloud/manage-environments/manage-environment-ops-authing/overview), the AutoMQ operational platform requires read and write access to the Ops Bucket in your Azure environment. This access enables system stability monitoring, self-healing, subscription license updates, and version upgrades for your BYOC environment.

In Azure, cross-tenant authorization is achieved by creating a service principal from AutoMQ's multitenant application in your tenant, then assigning the appropriate role to that service principal. For more details on this approach, see [Create a service principal for a cross-tenant application](https://learn.microsoft.com/entra/identity/enterprise-apps/create-service-principal-cross-tenant?pivots=azure-cli) and [Authorize cross-tenant access](https://learn.microsoft.com/en-us/azure/azure-signalr/signalr-howto-authorize-cross-tenant).

<Info>
  In this document, references to the AutoMQ product service provider, AutoMQ service provider, and AutoMQ all specifically mean AutoMQ HK Limited.
</Info>

## Procedure

### Step 1: Sign in to Azure CLI with your tenant administrator

Sign in to Azure CLI using your tenant administrator identity. Replace `<Your Tenant ID>` with your Microsoft Entra tenant ID.

```shell theme={null}
az login --tenant <Your Tenant ID>
```

### Step 2: Collect your Ops Bucket information

Before proceeding, gather the following information about the Ops Bucket created when you set up the AutoMQ BYOC environment. You can retrieve these values from the Azure portal or Azure CLI.

| Parameter       | Description                                                    |
| --------------- | -------------------------------------------------------------- |
| Subscription ID | The Azure subscription that contains the Ops Bucket resources. |
| Resource Group  | The resource group where the Storage Account is located.       |
| Storage Account | The Storage Account that hosts the Ops Bucket Blob Container.  |
| Blob Container  | The name of the Blob Container used as the Ops Bucket.         |

To list your Storage Accounts and Blob Containers via Azure CLI:

```shell theme={null}
# List Storage Accounts in a resource group
az storage account list --resource-group <Your Resource Group> --query "[].name" -o table

# List Blob Containers in a Storage Account
az storage container list --account-name <Your Storage Account> --query "[].name" -o table
```

### Step 3: Create a service principal for the AutoMQ operational platform

Create a service principal in your tenant from AutoMQ's multitenant application. This allows the AutoMQ operational platform to authenticate in your Azure environment.

Run the following command:

```shell theme={null}
az ad sp create --id 123d6e96-84f4-4569-82ec-c25edb2dc696
```

<Warning>
  The `--id` value `123d6e96-84f4-4569-82ec-c25edb2dc696` is the official App ID of the AutoMQ operational platform. Do not modify this value.
</Warning>

After the command succeeds, verify the service principal:

```shell theme={null}
az ad sp show --id 123d6e96-84f4-4569-82ec-c25edb2dc696
```

### Step 4: Assign the Storage Blob Data Contributor role

Grant the AutoMQ service principal the **Storage Blob Data Contributor** role, scoped to the Ops Bucket Blob Container. AutoMQ only receives this role on the specific Blob Container used as the Ops Bucket — no other resources are affected.

Replace the placeholders with the values collected in Step 2:

```shell theme={null}
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee 123d6e96-84f4-4569-82ec-c25edb2dc696 \
  --scope "/subscriptions/{Your Subscription ID}/resourceGroups/{Your Resource Group}/providers/Microsoft.Storage/storageAccounts/{Your Storage Account}/blobServices/default/containers/{Your Ops Bucket Blob Container}"
```

After the command completes successfully, the Ops Bucket authorization is finished. You can proceed with cluster creation and management on the AutoMQ platform.


# Set Ops Authing from Google Cloud
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-google-cloud

Enable AutoMQ to provide 24/7 monitoring by configuring Google Cloud GCS operational authorization for cross-account access, ensuring seamless data management.

Referencing [Manage Environment Ops Authing▸](/automq-cloud/manage-environments/manage-environment-ops-authing), when using the AutoMQ Cloud BYOC product, the AutoMQ service provider requires users to provide the appropriate operational authorization. Users can grant cross-account access to the Ops Bucket, where AutoMQ stores logs and monitoring data, enabling AutoMQ to deliver 24/7 monitoring and governance for the customer's production environment. This document explains how to configure managed operational authorization on Google Cloud GCS.

<Info>
  In this document, references to the AutoMQ product service provider, AutoMQ service provider, and AutoMQ all specifically mean AutoMQ HK Limited.
</Info>

### Google Cloud Environment Authorization

GCS on Google Cloud offers a web interface and simplified policy authorization.

#### Console Operational Steps

Log in to the cloud account that owns the Ops Bucket and switch to the corresponding Project. Then navigate to the [GCS Console](https://console.cloud.google.com/storage/browser):

1. **Navigate to the Bucket List page to find the target OpsBucket.** Access the GCS Console and click the **Bucket List** in the left navigation bar to enter the list page. Search for the operations Bucket configured in the current environment, and click on details.

<img />

2. **Go to Permission Management > Add Authorization.** Click the add authorization button to open the form for new authorization.

<img />

3. **Add the Bucket Authorization Policy according to the template information.** Refer to the following instructions to fill in the respective authorization information, click confirm, and complete the authorization policy creation.

   * **Authorized main account:** Refer to the **Authorization Principle** section of this document and select the AutoMQ service-side **service account "[automq-public-ops-authing@automq-public.iam.gserviceaccount.com](mailto:automq-public-ops-authing@automq-public.iam.gserviceaccount.com)".**

   * **Bound Role:** Choose the **Storage Object User** role.

<img />


# Set Ops Authing from OCI
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-authing-from-oci



This document describes how to configure operational authorization for AutoMQ on OCI.

## Background Information

When creating a BYOC cluster, you need to authorize the AutoMQ operational platform to access your OCI Bucket for cluster maintenance operations. This document will guide you through the operational authorization process on OCI.

## Procedure

You need to create a Policy in your OCI tenancy that allows the Group used by the AutoMQ operational platform to access the corresponding operational Bucket.

Please refer to the following commands and replace `<ops-bucket-name>` with your OPS Bucket name.

```shell theme={null}
Define group OpsAccessGroup as ocid1.group.oc1..aaaaaaaakauhv6pn2cmx4vrrbzysk4oeqmp2lqabjistbkgapzawnl5m74ia

Define tenancy SourceTenancy as ocid1.tenancy.oc1..aaaaaaaaeufibuie3dttnycwg6h2jnfpqot6w2guvdjz2oiqseggpdpmfl7a

Admit group OpsAccessGroup of tenancy SourceTenancy to read,write objects in tenancy where target.bucket.name='<ops-bucket-name>'
```

After completing the above steps, you can proceed with subsequent cluster creation and management operations on the AutoMQ platform.


# Set Ops Tunnel for Troubleshooting
Source: https://docs.automq.com/automq-cloud/manage-environments/manage-environment-ops-authing/set-ops-tunnel-for-troubleshooting

AutoMQ's BYOC Ops Tunnel provides secure temporary access for troubleshooting, ensuring cloud-native scalability, Kafka compatibility, and cost efficiency. Learn setup and usage in detail.

Refer to [Overview▸](/automq-cloud/manage-environments/manage-environment-ops-authing/overview). The AutoMQ BYOC environment supports Ops Tunnel, enabling environment administrators to initiate and establish a secure and trusted Ops Tunnel for temporarily granting access to AutoMQ technicians in the BYOC environment. This article explains how to use Ops Tunnel.

## Prerequisites

1. OpsTunnel utilizes the Ngrok secure proxy tunnel to provide a secure and trusted external WebSocket channel.
2. It supports temporary access to the BYOC environment for AutoMQ technical staff.
3. Using OpsTunnel only requires allowing the AutoMQ environment console to access the secure Ngrok service.

For more information about Ngrok, please refer to the [Ngrok Product Introduction](https://ngrok.com/).

## Usage Process

### Enable or Disable Ops Tunnel

To facilitate timely responses and resolution of potential issues by our technical support team, AutoMQ enables Ops Tunnel by default during standard deployment. This tunnel establishes a secure, encrypted connection, allowing authorized AutoMQ support engineers to access your BYOC environment for diagnostics and troubleshooting when necessary.

While we recommend keeping this tunnel active to receive prompt support, you may need to temporarily or permanently disable this feature in certain scenarios. You can manually disable Ops Tunnel by following these steps:

1. **Log in to the Console**: Log in to the AutoMQ console with your environment administrator account.
2. **Navigate to Ops Tunnels**: In the main menu, locate and open the "Ops Tunnels" management page.
3. **Close the Tunnel**: On the Ops Tunnels detail page, find and click the "Close Tunnel" button. The system will prompt you for confirmation.
4. **Confirm Closure**: In the confirmation dialog, click confirm to complete the closure process.

<Danger>
  **Important Note:**

  Closing the Ops Tunnel will immediately terminate all active connections and reject any new connection requests. The Ops Tunnel can be re-enabled or temporarily enabled at any time, which does not affect existing service operations.
</Danger>

### Inspecting Operation Audits

AutoMQ environment administrators can review historical sessions and operation logs of Ops Tunnel through the operation audit function.

<img />


# Overview
Source: https://docs.automq.com/automq-cloud/manage-environments/overview

Discover the power of AutoMQ with cloud-native scalability, 100% Kafka compatibility, and cost-efficient management in both SaaS and BYOC environments.

The environment is a crucial operational resource for AutoMQ Cloud. This article introduces the definition, resource model, internal attributes, classification, and usage recommendations of the AutoMQ Cloud environment.

<Info>
  In this article, references to AutoMQ product service providers, AutoMQ service providers, and AutoMQ specifically refer to AutoMQ HK Limited.
</Info>

<Info>
  Terms such as cloud providers and Public Cloud providers mentioned in this article refer to the cloud service providers that AutoMQ Cloud relies on, as specified by the user, including AWS, Google Cloud, Azure, OCI, and Alibaba Cloud.
</Info>

## Definition

The environment is a logical abstract resource provided by AutoMQ Cloud, used to determine the foundational configuration information that needs to be shared at the runtime level of Kafka and to manage the actual usage of Kafka instances (clusters) in a unified manner. The environment defines information such as infrastructure type, network environment, and permission policies, and all subsequent Kafka instances are created and managed under the environment.

The main roles of the environment are elaborated as follows:

* It stipulates the underlying infrastructure of the Kafka service, clarifying the basic information such as the type of cloud provider, region, and permission policies of the infrastructure.

* It stipulates the basic information such as the network topology, availability zone topology, and external service exposure methods of the entire control system.

* Common foundational configurations across instances, such as metrics integration configurations.

In a BYOC environment, all physical resources belong to the user, and the user authorizes the AutoMQ service provider to offer maintenance and management.

## Model Relationships

In the domain model of AutoMQ Cloud, the environment is the top-level resource for managing product instances, as roughly illustrated below:

<img />

## Internal Attributes

### Attribute: Infrastructure Configuration

Infrastructure configuration is a necessary setup for deploying AutoMQ Cloud environments, specifying the type of infrastructure, network topology, permission policies, and other configuration information. The environment console will use this configuration to deploy and manage actual Kafka clusters.

### Attribute: Integration Configuration

Integration configuration abstracts configuration at the environment level for external data integration across clusters. For example, the Metrics integration configuration can support pushing Metrics data from multiple clusters within the environment to a user-defined Prometheus cluster for custom monitoring and analysis.

### Attribute: Environment Version

Technically, the environment itself is a software foundation and, as such, will define multiple environment versions with the continuous iteration of the software. Upgrading the environment version generally does not affect the upper-layer product services. Users are advised to update to the latest environment version regularly to ensure defect fixes and feature optimizations.

### Attribute: Environment Status

The environment status shows the various service states in the environment lifecycle. The complete lifecycle of an environment will include the following processes and states:

<img />

* **Creating:** The initial environment creation operation is an asynchronous process. When the creation operation is completed, the backend will asynchronously deploy the full software foundation. During this process, the user cannot make any changes until the creation is complete.

* **Running:** The current environment is in an active running state, allowing normal operational use such as creating, deleting, updating, and querying internal product instances.

* **Changing:** The current environment is performing operational changes, version upgrades, or other operations. Since these operations are asynchronous, any changes to the environment are prohibited until the process is complete.

* **Service Exception:** The current environment is unable to provide normal services due to the execution of creation, maintenance changes, or underlying infrastructure anomalies.

## BYOC Environment

### Environment Architecture

As mentioned above, the BYOC environment is maintained by the AutoMQ Cloud agent. The underlying resources belong to the user's custom cloud account under the VPC, allowing for complete data isolation and autonomous control. In this environment type, the data plane system and control plane system are as shown in the following figure:

<img />

* **Resource Location:** The control plane system (environment console) and the data plane system (Kafka service cluster) are all deployed in the user-defined network environment.

* **Network Exposure:** The environment console can be accessed via the public network by default based on user choice, or it can be restricted to private network access only. The data plane system supports only private network access.

* **Maintenance Authorization:** The BYOC environment is a fully privately isolated private environment that requires user authorization for AutoMQ to perform environment maintenance (issue troubleshooting and version upgrades).

* **System Observability:** The BYOC environment collects system metrics and log data through the maintenance bucket provided by the cloud providers. Users authorize AutoMQ service providers with read-only access to achieve system operation monitoring and alerting.


# Upgrade Environment Version
Source: https://docs.automq.com/automq-cloud/manage-environments/upgrade-environment-version

Upgrade your AutoMQ BYOC environment version on Alibaba Cloud or AWS for improved functionality and bug fixes. Ensure seamless updates with our step-by-step guide.

When using AutoMQ BYOC, users can refer to this document to upgrade the BYOC environment console version as needed to receive updates and fix bugs.

<Info>
  The terms AutoMQ product service provider, AutoMQ service provider, and AutoMQ mentioned in this document specifically refer to AutoMQ HK Limited.
</Info>

AutoMQ BYOC environments will produce updated versions as functionalities evolve. Users can refer to the [Release Notes▸](/automq-cloud/release-notes) to view the corresponding versions and choose to upgrade the environment version.

## Upgrade Operation

### AWS

For upgrading the environment version in AWS, this document is divided into two sub-sections, applicable for versions **>= 8.0** and versions **\< 8.0**.

#### Version >= 8.0

Starting from version 8.0, the AutoMQ environment console is installed and managed via Docker. Therefore, upgrading the version only requires updating the Docker image and restarting the container.

**Steps**

1. **Check the version number**
   Go to the [Release Notes](/automq-cloud/release-notes) to confirm the target version number you wish to upgrade to.

2. **Get the installation command**
   Log in to the AutoMQ official website, find the metadata for your current environment, and copy the `docker run` command used for the initial installation.

3. **Update and restart the container**
   On the host machine where the console is located, stop and remove the old container, then re-run the installation command with the new version number.

For example, if your initial installation command was:

```bash theme={null}
# Initial installation command example
sudo docker run -d -v /home/admin:/root \
  -p 8080:8080 \
  -e CLOUD_PROVIDER=aws \
  -e REGION=ap-northeast-1 \
  -e OPS_BUCKET=automq-ops-xxxx \
  -e CONSOLE_INITIAL_USER=admin \
  -e CONSOLE_INITIAL_PASSWORD=admin \
  -e CLIENT_ID=env-xxxx \
  -e CLIENT_SECRET=xxx \
  xxx/automq/automq_byoc_console:8.0.0
```

To upgrade to version `8.0.1`, you can perform the following actions:

```bash theme={null}
# 1. Stop and remove the old container
sudo docker stop automq_byoc_console
sudo docker rm automq_byoc_console

# 2. Start the new container with the new version number
sudo docker run -d -v /home/admin:/root \
  -p 8080:8080 \
  -e CLOUD_PROVIDER=aws \
  -e REGION=ap-northeast-1 \
  -e OPS_BUCKET=automq-ops-xxxx \
  -e CONSOLE_INITIAL_USER=admin \
  -e CONSOLE_INITIAL_PASSWORD=admin \
  -e CLIENT_ID=env-xxxx \
  -e CLIENT_SECRET=xxx \
  xxx/automq/automq_byoc_console:8.0.1
```

After the upgrade is complete, you can log back into the AutoMQ environment console to verify that the new version is active.

#### Version \< 8.0

If your AutoMQ BYOC environment is older than version 8.0 and was installed via Marketplace CloudFormation on AWS, then upgrading to an updated 7.x version also needs to be done via CloudFormation.

<Tip>
  If you want to upgrade from a 7.x version to an 8.x version, please refer to the [BYOC Console Upgrade to 8.x Guide](/automq-cloud/appendix/byoc-console-upgrade-to-8.x-guide).
</Tip>

**Steps**

Log in to the cloud account associated with the Console or a sub-account with authorization capabilities, and navigate to the Marketplace Console.

1. **Go to the Marketplace to check the latest version**. Visit the [AWS Marketplace product page](https://aws.amazon.com/marketplace/pp/prodview-suwr5pyxwakrk), select the target version you wish to upgrade to; it is recommended to upgrade to the most recent 'Latest' version.

<img />

2. **Select the desired upgrade version.** Click "Launch with CloudFormation".

<img />

3. **Copy the new version's CloudFormation Template URL.**

<img />

4. **Go to the Stack list page and find the Stack that deploys the AutoMQ BYOC environment.** Access the CloudFormation console, click on **Stacks** in the left navigation bar to enter the list page. Search for the Stack corresponding to the current environment and click **Stack Actions**.

<img />

5. Find the operations supported by Stack, then click **Replace existing template, create ChangeSet.**

<img />

6. Update the ImageId to the target version. For example, substitute **7.0.3 with 7.1.0**.

<img />

<Danger>
  **Warning:**

  When creating the ChangeSet, aside from the new version Template update, the following parameters cannot be changed; otherwise, the environment may become unstable:

  ExistingVPCId: After the environment is set up, the VPC cannot be altered.

  ExistingPublicSubnetId: Once the environment installation is complete, you cannot modify the subnet where the environment console resides.

  MsgBucket: Once the environment installation is complete, the MsgBucket cannot be changed.

  OpsBucket: After the installation of the environment is finalized, the OpsBucket is not alterable.
</Danger>

<img />

7. **Submit changes step-by-step for a preview.** After confirming the submission, you can review the scope of resources affected by the current version upgrade.

<img />

8. **Confirm the execution of the ChangeSet.** Wait for the execution to complete and verify whether the upgrade results align with expectations.

### Google Cloud

In the AutoMQ BYOC environment on GCP via Marketplace installation, since Marketplace does not offer a console page for upgrades, it is advised to upgrade versions using the gcloud CLI.

#### Steps

1. Locally, use the gcloud CLI to log into the cloud account linked with the environment console, or a sub-account with authorized operational capabilities.

```bash theme={null}

gcloud auth login

```

2. **Create New Version System Disk**. Check the AutoMQ version logs to determine the environment version that needs an upgrade. Use the following command as a guideline to create a system disk for the new version of the console VM.

```bash theme={null}
gcloud compute disks create $disk_name  \
--image projects/mpi-automq-public/global/images/automq-control-center-prod-$version-x86-64 \
--zone $zone

```

The parameter descriptions are as follows:

* disk\_name: You can customize the name of the system disk.

* image: The name of the image used by the system disk. Here, you need to input the corresponding value based on the version number, such as “projects/automq-public/global/images/automq-control-center-prod-**7-3-11**-x86-64”, replacing **7-3-11** with the appropriate **X.Y.Z** version number.

* Zone: Enter the name of the zone where the AutoMQ console is located.

3. **Stop the AutoMQ Console Virtual Machine**. Go to the Compute Engine console and find the virtual machine named AutoMQ Console. Run the following command to power it off.

```bash theme={null}
gcloud compute instances stop $instance_name --zone $INSTANCE_ZONE

```

The parameter descriptions are as follows:

* instance\_name: Enter the instance name in the AutoMQ console.

* Zone: Enter the name of the zone where the AutoMQ console is located.

4. **Unmount the Old System Disk**.

```bash theme={null}
gcloud compute instances detach-disk ${Your VM Instance} \
    --disk=${Your Original System Disk} --zone $INSTANCE_ZONE

```

The parameter descriptions are as follows:

* instance\_name: Enter the instance name in the AutoMQ console.

* disk: Enter the current system disk name from the AutoMQ Console.

5. **Mount the new system disk.**

```bash theme={null}
gcloud compute instances attach-disk $INSTANCE_NAME \
    --disk $NEW_DISK_NAME \
    --zone $INSTANCE_ZONE \
    --boot

```

The parameter descriptions are as follows:

* instance\_name: Enter the instance name in the AutoMQ console.

* disk: Enter the new system disk name from the AutoMQ Console. This is the name of the system disk created in step 1.

* Zone: Enter the name of the zone where the AutoMQ console is located.

6. **Restart the AutoMQ Console.**

```bash theme={null}
gcloud compute instances start $INSTANCE_NAME --zone $INSTANCE_ZONE

```

The parameter descriptions are as follows:

* instance\_name: Enter the instance name in the AutoMQ console.

* Zone: Enter the name of the zone where the AutoMQ console is located.

7. Once the upgrade is complete and verified to meet expectations, you can **delete the system disk of the previous version**. After deletion, rolling back to the previous version will not be possible.

```bash theme={null}
gcloud compute disks delete $DISK_NAME --zone $INSTANCE_ZONE

```

The parameter descriptions are as follows:

* DISK\_NAME: Enter the system disk name of the previous version in the AutoMQ console.

* INSTANCE\_ZONE: Enter the name of the zone where the AutoMQ console is located.


# Identity Providers and SSO Authentication
Source: https://docs.automq.com/automq-cloud/manage-identities-and-access/identity-providers-and-sso-authentication

AutoMQ integrates with enterprise identity providers using SAML for SSO authentication. Manage all users in one place and allow access with existing credentials.

AutoMQ supports configuring enterprise identity providers using the SAML protocol, allowing you to manage and authenticate AutoMQ environment members through your existing identity provider (hereinafter referred to as Identity Provider, IdP) without needing to use AutoMQ local usernames and passwords.

After enabling SSO in AutoMQ, you can manage all users in one place and allow users to log in to AutoMQ using their existing SSO credentials.

## Prerequisites

The AutoMQ console by default provides an initial admin member and local member login method. To enable SSO login, the following conditions must be met:

* The enterprise's IdP must support the **SAML 2.0 protocol**. AutoMQ does not support the OIDC protocol.

* Enterprise IdP needs to clearly identify each unique user.

* The username provided by the enterprise IdP must not conflict with member names of the AutoMQ local type. Refer to [Member Accounts▸](/automq-cloud/manage-identities-and-access/member-accounts).

## Usage Limitations

When using Enterprise IdP for SSO login in the AutoMQ Console, the following limitations apply:

* Each AutoMQ Console only supports the configuration of a single Enterprise IdP and does not support multiple identity sources.

* After configuring the IdP, SSO is enabled by default, but the local user login method is still supported. Environment administrators can manage SSO through local login.

* If the IdP is deleted, SSO is disabled by default, and existing SSO members will not be deleted and need to be manually removed.

* AutoMQ does not support converting between SSO and local member types.

* AutoMQ does not support the SAML Session logout protocol.

## Configuring Enterprise Identity Providers and SSO.

### Step 1: Configure AutoMQ Console Domain Name

Before configuring IdP and enabling SSO login, the environment admin needs to set up the domain name for the AutoMQ console. This ensures that enterprise users can access the AutoMQ console via their browsers using the specified domain name. The configuration steps are as follows:

1. The enterprise admin clicks the **Settings** menu.

2. On the settings page, click **Add Console Domain Name**.

3. Enter the domain name, port, and protocol that the console will use to provide services to users.

<img alt="Settings page showing the Add Console Domain Name configuration form" />

<Tip>
  Enterprise users who require the use of the HTTPS protocol are recommended to mount the console domain onto a front load balancing service such as ALB to complete the TLS protocol offloading.
</Tip>

4. Once setup is complete, conduct a connectivity test.

<img alt="Connectivity test result for the configured AutoMQ Console domain" />

### Step 2: Obtain SP Information from the AutoMQ Console

An AutoMQ environment administrator (a member authorized with the environment administrator role) should view the AutoMQ SP information and provide it to the enterprise IdP administrator. Follow these steps:

1. Click on the **Access and Control** section in the navigation bar. Then click **Add IdP**.

2. **Select Console External Domain**.

3. View the identity information of the AutoMQ console as a Service Provider (SP). Submit the SP information to the enterprise IdP administrator for entry. The administrator can manually copy the information or download the metadata file directly.

   1. **SP EntityID:** A unique identifier for the AutoMQ console.

   2. **Assertion Consumer Service URL:** The ACS URL is the unique address where AutoMQ, as an SP, receives SAML Responses, and it needs to be configured on the IdP.

   3. **AutoMQ SP Certificate:** The certificate used to declare AutoMQ's signed requests.

<img alt="AutoMQ Console SP information panel showing SP EntityID, ACS URL, and SP certificate" />

### Step 3: Configure AutoMQ Service in Enterprise IdP

After obtaining the SP information in Step 2, the enterprise IdP administrator needs to configure the SP information of the AutoMQ console into the IdP service. The operations for this step vary depending on the IdP service. Below are common configuration methods for IdP.

<Tabs>
  <Tab title="Auth0">
    1. Log in to the [Auth0 account](https://manage.auth0.com/).

    2. Select **Applications**.

    3. Click **Create Application**.

    4. Enter the application name.

    5. Select **Regular Web Applications** and click **Create.**

    6. Once the application is created, navigate to the Addons settings page to enable SAML2.0 configuration.

    7. Click on the **SAML 2 WEB APP** option to open the settings page, and configure the following parameters:

       1. In the **Application Callback URL** parameter box, set the **ACS URL** obtained from the AutoMQ console.

    8. Click **Enable** and save.

    9. On the **Usage Tab** page, record the **Identity Provider Login URL**, **Issuer URN**, and **Identity Provider Certificate** information for use in Step 4 for IdP information entry.
  </Tab>

  <Tab title="OKTA">
    1. In the Okta admin console, navigate to **Applications**.

    2. Click **Add Application** to start creating a custom application.

    3. Select the **SAML 2.0** application type.

    4. Enter **App Name**, and click Next.

    5. Configure the SP-related parameters according to the instructions below, and click Next.

       1. Single sign-on URL: Enter the ACS URL provided by the SP from the second step.

       2. Audience URI (SP Entity ID): Enter the SP Entity ID provided by the SP from the second step.

       3. Name ID format: Fill in the unique identifier field according to your organization's standards, with the email address being the recommended option.

       4. Application username: According to the enterprise standard, fill in the unique identifier field. It is recommended to use the OKTA username.

    <img alt="Okta SAML application configuration showing single sign-on URL and Audience URI fields" />

    6. Click "Finish" to complete the SP entry. Click to download the metadata configuration file, which will be used to complete the IdP information in Step 4.

    <Tip>
      **Note:**

      After creating a new application in OKTA, users who have not been assigned to this OKTA application by default cannot use the AutoMQ application. Therefore, it is recommended to allocate the application to relevant users based on requirements.
    </Tip>
  </Tab>
</Tabs>

### Step 4: Complete IdP Entry in the AutoMQ Console

After configuring the enterprise IdP, the IdP information needs to be recorded in the AutoMQ console to complete the connection. The AutoMQ console supports manual entry and direct upload of metadata files. The required information is as follows:

* **IdP Alias:** This is used to distinguish the information of the IdP. It supports English and Chinese letters, numbers, hyphens, and underscores, with a length limit of 3-64 characters.

* **IdP Entity ID:** This is the unique identifier used to recognize the IdP.

* **IdP SSO URL:** This is the unique login address assigned by the IdP to the AutoMQ console.

* **IdP Certificate:** This is the certificate information used by the IdP to sign and encrypt SAML responses.

* **UserID Mapping (Optional):** AutoMQ extracts the unique user ID attribute from the SAML response. If not set, the default value will be used.

* **UserName Mapping (Optional):** AutoMQ extracts the displayed environment member name attribute from the SAML response. If not set, it will be the same as UserID.

* **Session Expiration Time Mapping (Optional):** AutoMQ extracts the session expiration time attribute from the SAML response. If not set, it will default to 6 hours.

<img alt="IdP entry form in AutoMQ Console for configuring IdP alias, entity ID, SSO URL, and certificate" />

Advanced attribute field settings are as follows:

<img alt="Advanced attribute mapping settings for UserID, UserName, and session expiration time" />

### Step Five: SSO User Login

Once IdP input and SSO configuration are complete, the AutoMQ console can be accessed using SSO login.

<img alt="AutoMQ Console login screen with SSO login option" />

<Tip>
  **Note:**

  When a new user logs in directly using SSO, they will, by default, have no permissions to operate within the environment and will require authorization from an environment administrator to access specific resources.

  Environment administrators can also pre-create SSO members and assign predefined roles to them, allowing these SSO members to operate the console normally upon subsequent logins.
</Tip>


# Member Accounts
Source: https://docs.automq.com/automq-cloud/manage-identities-and-access/member-accounts

Manage environment members in AutoMQ Cloud. Each environment has an isolated account system to prevent permission leaks across environments.

Refer to [Overview▸](/automq-cloud/manage-identities-and-access/overview), users need to log into the environment console to use the product. This article describes how to create and manage environment members in the AutoMQ Cloud edition.

## Environment Members

Each AutoMQ environment can be independently accessed, with a separate account system within each environment to manage the resources.

* Environment members are completely isolated between different environments to prevent confusion and permission leaks.

* Environment members are not associated with cloud provider accounts and are mutually independent.

### Initial Environment Members

Refer to [Overview▸](/automq-cloud/manage-identities-and-access/overview), when an environment is created for the first time, the system automatically initializes and creates an Admin role member for the current environment.

<Info>
  The initial Admin member's username is admin, and the default password varies depending on the cloud provider environment. It is recommended to change the initial default password after logging in.
</Info>

The method to obtain the initial password varies depending on the cloud provider environment. Refer to the documentation below.

<Tabs>
  <Tab title="Deploy using Docker">
    In an AWS environment, starting from version 8.0, deploying the BYOC environment using a Docker installation package is supported. The initial username and password are freely specified by the installer at startup.
  </Tab>
</Tabs>

## Login Environment

The environment administrator records the domain name of the environment console and distributes it to all environment members. Environment members can log in directly by entering their account information.

1. Open the environment console address in a browser. This address is provided by the environment administrator from the deployment system.

2. Enter the environment member's username and password to log in.

3. For the first login, if using a system-generated password, it is recommended to change it to a custom password as soon as possible.

<img alt="AutoMQ Console login screen with username and password fields" />

## Create Environment Members

Using the AutoMQ Cloud environment console, environment members with the Admin role can create new member accounts to share the environment.

The steps to create environment members are as follows:

1. Admin role environment members enter the environment.

2. Click on the "Members" button in the left navigation bar to access the member list.

3. Click "Add Member," select the type of member account, enter the member's username, choose the role to assign, and click "Add."

   1. **Local Type:** The local type is the default member account type, using a static username and password for login authentication.

   2. **SSO Type:** The SSO type needs to cooperate with an enterprise identity service provider, using SSO for login, and does not support username and password login.

4. The system will return the new member's username and default password.

<img alt="Add Member dialog in AutoMQ Console showing member type, username, and role selection" />

<Tip>
  To create an SSO environment member, you must first configure the enterprise identity provider (IDP) and ensure the SSO integration is completed.
</Tip>

## Modifying Environment Members

Admin role environment members can manage environment members at any time through the environment console, including modifying member permission roles and deleting environment members. The steps are similar to adding members.


# Overview
Source: https://docs.automq.com/automq-cloud/manage-identities-and-access/overview

AutoMQ Cloud offers scalable, Kafka-compatible message queues with identity recognition and RBAC permissions, ensuring secure account access for both members and services.

AutoMQ Cloud supports identity recognition and RBAC (Role-Based Access Control) permissions. This article introduces the basic concepts of accounts in the AutoMQ Cloud product system.

## Account Types

AutoMQ Cloud offers two types of accounts: **Member Account** and **Service Account**. Their definitions and differences are as follows:

| **Account Type**<br />    | **Roles and Differences**<br />                                                                                                                                                                                                                                                                                                                    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Member Account**<br />  | <ul><li>**Scenario:** A Member Account corresponds to an individual, typically held and used by a company employee.</li><li>**Access:** Access AutoMQ Cloud through the environment console WebUI.</li><li>**Identity Recognition:** Username and password login authentication.</li></ul>                                                         |
| **Service Account**<br /> | <ul><li>**Scenario:** A Service Account is used only for applications and API integration, generally configured within application code.</li><li>**Access:** Service Accounts typically access AutoMQ Cloud via APIs, Terraform, etc.</li><li>**Identity Recognition:** Access Key Id and Secret Access Key, verified through signature.</li></ul> |

## Member Account

### Definition

Member accounts are system-generated by default, manually created by existing environment members, or automatically created through enterprise SSO login, serving as credentials for environment-level operations.

Member accounts support multiple roles based on the required permission scope, including Admin, Operator, and Viewer roles.

### Creation Method

* **Local Type:** The initial Admin member of each environment is automatically created by the system upon environment creation. Subsequent members can be manually created by Admin members.

* **SSO Type:** AutoMQ supports configuring enterprise Identity Provider (IDP) services. Environment members of the SSO type are created through SSO login.

### Relationship Between Environment and Environment Members

When a new environment is created, the system will automatically initialize and create an Admin role member for the current environment. Subsequent members are then created by the initial Admin member.

<img alt="Diagram showing the relationship between AutoMQ Cloud environments and environment members" />

## Service Account

### Definition

Service accounts are provided by AutoMQ Cloud for external systems to access AutoMQ via APIs and application integration. Service accounts do not have login passwords and cannot be operated through the WebUI.

### Creation Method

Service accounts can be created by member accounts in the AutoMQ Console or via API.

## RBAC Permission Control

AutoMQ Cloud includes both **member accounts** and **service accounts**, and both support RBAC (Role-Based Access Control). The system comes with several predefined permission roles, each offering different scopes of operational permissions. An Admin role account conducts authorization operations, assigning roles to other accounts.

For information on RBAC control, please refer to the document [Role-Based Access Control▸](/automq-cloud/manage-identities-and-access/role-based-access-control).


# Role-Based Access Control
Source: https://docs.automq.com/automq-cloud/manage-identities-and-access/role-based-access-control

AutoMQ's Role-Based Access Control (RBAC) system ensures fine-grained, least-privilege permissions for secure, stable resource management in a cloud-native Kafka-compatible environment.

AutoMQ supports Role-Based Access Control (RBAC), providing fine-grained permission management to ensure that member accounts and service accounts can access necessary resources with the principle of least privilege. Through RBAC, environment administrators can effectively manage and control access to AutoMQ resources, ensuring system security and stability.

### RBAC Principles

The core principle of RBAC is to bind predefined roles to member accounts and service accounts. Each role contains a set of predefined permissions that determine the actions an account can perform. This approach allows administrators to flexibly assign and manage permissions, ensuring users can only access and operate on the resources they are authorized to.

### Appendix: Predefined System Roles

AutoMQ provides a series of built-in roles, each corresponding to different levels of permissions to meet the needs of various scenarios. Below is a list of the built-in roles supported by AutoMQ along with their operation lists:

<table>
  <thead>
    <tr>
      <th>
        Role

        <br />
      </th>

      <th>
        Action

        <br />
      </th>

      <th>
        Permission Level and Applicable Scenarios

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        InstanceViewer

        <br />
      </td>

      <td>
        Instance:ListInstances

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Instance Viewer Role</li>

          <li>Permission Scope: Read-only access to specified instances within the environment (excluding message viewing).</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstanceMetadata

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLPolicies

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLUsers

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:ListConsumerGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:GetConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:GetTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:ListTopics

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:ListKafkaLinks

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:GetKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        InstanceDeveloper

        <br />
      </td>

      <td>
        Instance:GetInstance

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Instance Developer</li>

          <li>Permission Scope: Access to specified instances within the environment and management of resources like Topic and Group within the instance. Instance developers cannot perform instance configuration changes, upgrades, etc.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstances

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLPolicies

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstanceMetadata

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLUsers

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:ListConsumerGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:CreateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:GetConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:UpdateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:DeleteConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:ListTopics

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:GetTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:DeleteTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:UpdateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        InstanceAdmin

        <br />
      </td>

      <td>
        Instance:GetInstance

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Instance Admin Role</li>

          <li>Permission Scope: Access to view, modify, and delete specified instances within the environment.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstances

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:UpdateInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstanceMetadata

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:UpdateInstanceIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLUsers

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLPolicies

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:ListConsumerGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:CreateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:GetConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:UpdateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:DeleteConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:GetTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:DeleteTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:UpdateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:ListTopics

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrationTypes

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:CreateKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:ListKafkaLinks

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:GetKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:DeleteKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        IntegrationAdmin

        <br />
      </td>

      <td>
        Integration:UpdateInstanceIntegration

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Integration Administrator</li>

          <li>Permission Scope: Access and edit specified integrations within the environment.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:GetIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:UpdateIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:DeleteIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrationTypes

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        EnvironmentViewer

        <br />
      </td>

      <td>
        Instance:GetInstance

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Environment Read-Only Member Role</li>

          <li>Permission Scope: Read-only access to instance resources within the environment; cannot manage the environment or other members (excluding message viewing).</li>

          <ul>
            <li>Instance Viewing</li>

            <li>Integration Viewing</li>
          </ul>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstances

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLPolicies

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstanceMetadata

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLUsers

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:ListConsumerGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:GetConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:ListTopics

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:GetTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:GetIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrationTypes

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:ListMigrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:GetMigration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListProductVersions

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListProviders

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListRegions

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListZones

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListSubnets

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListNodeGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetNodeGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetEnvironment

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:ListKafkaLinks

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:GetKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        EnvironmentOperator

        <br />
      </td>

      <td>
        Instance:GetInstance

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Environment Operator Member Role</li>

          <li>Permission Scope: Write access to instance resources within the environment; cannot manage the environment or other members:</li>

          <ul>
            <li>Integration Management</li>

            <li>Instance Management</li>
          </ul>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstances

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLPolicies

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLPolicy

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:UpdateInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstance

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:GetInstanceMetadata

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:UpdateInstanceIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:ListInstanceACLUsers

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:CreateInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance:DeleteInstanceACLUser

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:ListTopics

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:GetTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:DeleteTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:UpdateTopic

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic:CreateMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:ListConsumerGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:CreateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:GetConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:UpdateConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        ConsumerGroup:DeleteConsumerGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:CreateIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:GetIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:UpdateIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:DeleteIntegration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Integration:ListIntegrationTypes

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:CreateMigration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:UpdateMigration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:ListMigrations

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:GetMigration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Migration:DeleteMigration

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:ListDeployProfiles

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:GetDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:CreateDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:UpdateDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Profile:DeleteDeployProfile

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:CreateProductVersion

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetDeploymentOrder

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListProductVersions

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListProviders

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListRegions

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListZones

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListSubnets

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:ListNodeGroups

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetNodeGroup

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:GetMessage

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:DeleteEndPoint

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Environment:CreateEndPoint

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:CreateKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:ListKafkaLinks

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:GetKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        KafkaLinking:DeleteKafkaLink

        <br />
      </td>
    </tr>

    <tr>
      <td>
        EnvironmentAdmin

        <br />
      </td>

      <td>
        All

        <br />
      </td>

      <td>
        <ul>
          <li>Role Description: Environment Administrator</li>

          <li>Permissions: Has full operational access to all resources within the environment, including but not limited to:</li>

          <ul>
            <li>Manage Members</li>

            <li>Manage Integrations</li>

            <li>Manage Instances</li>
          </ul>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

By utilizing built-in roles and their corresponding operations, administrators can flexibly manage permissions in the AutoMQ console, ensuring both system security and efficiency.


# Service Accounts
Source: https://docs.automq.com/automq-cloud/manage-identities-and-access/service-accounts

Create and manage service accounts in AutoMQ Cloud for API and Terraform access. Service accounts are fully isolated between environments and independent of cloud provider accounts.

Refer to [Overview▸](/automq-cloud/manage-identities-and-access/overview), to manage environment resources through APIs, Terraform, or other methods, you must first create a service account. This article provides instructions for creating and managing service accounts in AutoMQ Cloud.

## Service Account

Each AutoMQ environment is managed independently, with a separate account system within each environment to manage resources specific to that environment.

* Service accounts are completely isolated between different environments to prevent confusion and permission leaks.

* There is no association between service accounts and cloud vendor accounts; they are entirely independent.

## Create a Service Account

Using the AutoMQ Cloud environment console, members with an Admin role can create service accounts.

The steps to create a service account are as follows:

1. An environment member with an Admin role accesses the environment.

2. Click the **Service Accounts** button in the left navigation bar to enter the **Service Accounts List**.

3. Click **Create Service Account**, enter the service account alias, choose the assigned permission role, and then click **Create**.

4. The system will provide the **AccessKey** information for the service account, which users need to download and save securely.

<Tip>
  The Access Key information for the service account is only available for download at the time of creation and cannot be modified or viewed later. Users should keep it safe to prevent leaks and loss.
</Tip>

## Modifying a Service Account

Members with Admin role privileges can access the environment console at any time to manage service accounts, including modifying the authorized role of the service account or deleting the service account. The steps for these actions are similar to those for creating an account.


# Manage Consumer Groups
Source: https://docs.automq.com/automq-cloud/managing-automq/manage-consumer-groups

Manage Consumer Groups in AutoMQ: view, modify, reset consumption offsets, and delete consumer groups in a Kafka-compatible, cloud-native environment.

This article introduces the basic concepts of Consumer Groups, relevant constraints, and operations related to viewing, modifying, and deleting Consumer Groups.

## Consumer Group

Referencing the domain model of Apache Kafka®, a Consumer Group is a load balancing grouping in the Kafka system that handles multiple consumers with consistent consumption behavior.

Unlike a Consumer, a Consumer Group is not a running entity but a logical resource. In Kafka, horizontal scalability of consumption performance and high availability disaster recovery are achieved by initializing multiple consumers within a consumer group.

## Viewing Consumer Group Details

### Procedure

1. Log in to the environment console, click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to enter the instance's internal functions.

2. In the left navigation bar, click on **Consumer Groups**, then find the target Consumer Group on the **Consumer Groups** page.

3. Click on the target Consumer Group to view summary information, client connection information, subscription relationships, consumption progress, and related Metrics data.

## Reset Consumption Offset

You can reset the consumption offset to clear accumulated or unwanted messages before resuming consumption. Alternatively, you can jump to a specific time point to consume messages from that point onwards, regardless of whether messages before that time point have been consumed.

<Info>
  When resetting the consumption offset, the corresponding Consumer Group must be in an Empty state, meaning the Consumer must be offline.
</Info>

### Procedure

1. Locate the Consumer Group whose consumption offset needs to be reset, and click on **Reset Offset** on its **Details Page**.

2. Choose from the following options based on actual needs:

   1. Reset based on relative time: last 5 minutes, last 1 hour, last 24 hours

   2. Reset progress based on a specific time point: custom timestamp

   3. Reset by relative position: latest offset, earliest offset

   4. Reset by specified offset: custom offset

## Delete Consumer Group

After deleting a Consumer Group, all consumers under the Consumer Group may be unable to consume messages, and the previous state data cannot be restored. Therefore, you must first confirm that the business no longer uses it and ensure that the Consumer Group is in a Dead or Empty state.

### Procedure

1. Log in to the environment console, click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to enter the instance's internal functions.

2. In the left navigation pane, click **Consumer Group**, and then find the target Consumer Group on the **Consumer Group** list page.

3. Click the action button on the right, click delete, and complete the secondary confirmation to delete.


# Manage Instances
Source: https://docs.automq.com/automq-cloud/managing-automq/manage-instances

Create, view, scale, upgrade, and delete AutoMQ BYOC (Bring Your Own Cloud) instances.

An AutoMQ instance is the top-level data plane resource and corresponds to an independent Apache Kafka® cluster.

## Instance lifecycle

<img />

* **Creating:** AutoMQ is provisioning cloud resources and deploying the instance. Configuration changes are unavailable until creation completes.
* **Running:** The instance can serve Kafka client requests and management APIs.
* **Modifying:** AutoMQ is applying a configuration change or version upgrade. Additional changes are unavailable until the operation completes.
* **Service Exception:** The instance cannot provide normal service because instance deployment, a change operation, or underlying infrastructure failed.
* **Deleting:** AutoMQ is deleting the instance and releasing its managed resources. The instance no longer accepts client traffic.
* **Deleted:** The instance and its managed resources have been released. The instance cannot be recovered.

## Create an Instance

1. Sign in to the AutoMQ Console.
2. In the navigation pane, choose **Instances**, and then choose **Create Instance**.
3. If prompted, select the Deployment Profile for the instance.
4. Follow the wizard to configure the instance settings, capacity, additional settings, and IAM permissions. The wizard displays only the settings supported by the selected environment and AutoMQ version.
5. Review the configuration and choose **Create**.
6. Wait until the instance status is **Running**.

## View an Instance

On the **Instances** page, choose the instance name. The details page displays status, endpoints, capacity, network, security, configuration, metrics, and related resources.

<img />

## Scale an Instance

1. Open the instance details page.
2. Choose **Change Configuration**.
3. Update the capacity settings and confirm the change.
4. Wait until the instance returns to **Running**.

<img />

## Upgrade an Instance

Review [Release Notes](/automq-cloud/release-notes) before upgrading. The AutoMQ Console must support the target instance version; if required, first follow [Upgrade Environment Version](/automq-cloud/manage-environments/upgrade-environment-version).

1. Open the instance details page.
2. Choose **Upgrade** when a target version is available.
3. Review the target version and confirm the rolling upgrade.

<Tip>
  AutoMQ performs instance upgrades as rolling operations. Monitor application error rates and latency throughout the change.
</Tip>

## Delete an Instance

On the **Instances** page, open the actions menu for the instance, choose **Delete**, and confirm the operation.

<Warning>
  Deleting an instance cannot be undone. Verify the instance name before you confirm the operation.
</Warning>


# Manage Topics
Source: https://docs.automq.com/automq-cloud/managing-automq/manage-topics

Learn how to create, view, modify, and delete Kafka-compatible topics in AutoMQ. Covers topic creation, message queries, and safe deletion practices.

This article introduces the basic concepts of Topics, related constraints, and operations for creating, viewing, modifying, and deleting Topics.

## Creating a Topic

### Operation Steps

1. Log in to the environment console and click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to access the instance's internal functions.

2. Click **Topic** in the left navigation bar, then click **Create New Topic** on the **Topic** list page.

3. On the **Create New Topic** form page, enter the name, number of partitions, and message cleanup policy, then click **OK** to complete the creation.

<Info>
  If the instance has automatic Topic creation enabled, Topic creation will be automatically triggered when producing or subscribing to messages. It is recommended to use the console or the Kafka Admin API for Topic management in a production environment to avoid the risk of uncontrolled automatic creation.
</Info>

## Viewing Topic Details

### Operation Steps

1. Log in to the environment console and click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to access the instance's internal functions.

2. Click **Topics** in the left navigation bar, then locate the target Topic on the **Topic** list page.

3. Click the target Topic to enter its details and view overview information, subscription information, and related metrics data.

## Message Query

AutoMQ supports querying messages directly from the console. The query conditions and parameters are as follows:

* By Relative Time: When multiple partitions are selected, it supports querying messages by relative time.

* By Relative Offset: When a single partition is selected, you can specify the offset to query messages.

### Operation Steps

1. Log in to the environment console and click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to access the instance's internal functions.

2. Click **Topics** in the left navigation bar, then locate the target Topic on the **Topic** list page.

3. Click on the target Topic to enter the Topic details, then click on Message Query and input the relevant filter conditions to query the message list.

## Delete Topic

After deleting the Topic, all messages and configurations within the Topic will be cleared and cannot be recovered. Therefore, it is crucial to confirm that the business no longer requires it.

<Info>
  If the instance has automatic Topic creation enabled, the Topic will be re-created automatically when producing or subscribing to messages. For production environments, it is recommended to use the console or Kafka Admin API for Topic management to avoid the risk of uncontrolled automatic creation.
</Info>

### Operation Steps

1. Log in to the environment console and click **Instance List** in the left navigation bar. Then, in the instance list, click the target instance name to access the instance's internal functions.

2. Click **Topics** in the left navigation bar, then locate the target Topic on the **Topic** list page.

3. Click the action button on the right, select delete, and confirm the deletion to complete.


# Overview
Source: https://docs.automq.com/automq-cloud/migrate-to-automq/overview

Migrate to AutoMQ Cloud with full Kafka compatibility. Adaptive elastic scaling, batch migration, and cost-efficient cloud-native infrastructure for reliable data synchronization.

AutoMQ is a next-generation Kafka distribution redesigned based on cloud-native principles and compatible with Apache Kafka®. This article explains the Kafka Linking feature provided by AutoMQ and how to use Kafka Linking to migrate business from Apache Kafka® or other Kafka distributions to AutoMQ.

## Migration scenarios

AutoMQ is compatible with Apache Kafka®, enabling Kafka Linking to migrate Kafka distributions provided by other vendors. Details are listed below:

<table>
  <thead>
    <tr>
      <th>
        Source Cluster

        <br />
      </th>

      <th>
        Target Cluster

        <br />
      </th>

      <th>
        Migration Support

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Apache Kafka

        <br />
      </td>

      <td>
        AutoMQ

        <br />
      </td>

      <td>
        Support

        <br />

        <Tip>
          Kafka Linking is compatible with Apache Kafka® versions >= 0.11.
        </Tip>

        <br />
      </td>
    </tr>

    <tr>
      <td>
        AWS MSK

        <br />
      </td>

      <td>
        Support

        <br />

        <Tip>
          Kafka Linking does not support MSK IAM Role authentication mode and only supports the native ACL authentication mode.
        </Tip>

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Confluent Platform

        <br />
      </td>

      <td>
        Supported

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Alibaba Cloud Kafka

        <br />
      </td>

      <td>
        Supported

        <br />
      </td>
    </tr>
  </tbody>
</table>

<Tip>
  If the current Kafka release version you are using is not listed above, please contact us immediately for support and assistance.
</Tip>

## Overview of Kafka Linking

AutoMQ Cloud provides the Kafka migration tool Kafka Linking, which supports migration from Apache Kafka® or other Kafka distributions to AutoMQ without application changes.

The Kafka Linking migration tool has the following advantages:

* **Non-Stop Migration**: Unlike MirrorMaker2, which requires client downtime for switching, Kafka Linking is equipped with a built-in traffic proxy component that ensures the switching process for Producers does not require downtime.

* **Byte-to-byte replication:** Unlike MirrorMaker2, which changes offsets during replication, Kafka Linking preserves partition offsets when it replicates source data to the target. Applications such as Flink jobs that restore offsets from checkpoints, savepoints, or external stores must still verify that every saved offset is readable on the target. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#validate-the-consumers-actual-offset-source) for the validation procedure.

* **Synchronized Consumption Progress:** Kafka Linking supports synchronized source cluster consumption progress, allowing consumers to resume consumption from the target cluster post-migration with consistent position information.

* **Elastic Scaling on Demand:** Kafka Linking requires no separate deployment and scales with the AutoMQ cluster on demand, reducing operational complexity during migration.

* **High Kafka Compatibility:** Kafka Linking supports most historical versions of Apache Kafka (version >=0.11) as well as other commercial editions, allowing transitions to AutoMQ.

## Kafka Linking terminology and concepts

Kafka Linking defines a series of terms, listed as follows:

### Kafka Link

A Kafka Link is a data synchronization and proxy task for a set of Topics from an external Apache Kafka cluster to an AutoMQ Kafka instance, including the configuration required for data synchronization, real-time operational status, and traffic proxy control.

### Source Cluster

The Source Cluster is an abstract configuration defined in Kafka Linking for the migration source cluster. The Source Cluster is described by AutoMQ's console-provided Kafka integration, containing configuration information such as the source cluster's access point, ACL rules, alias, etc.

### Sink Cluster

The Sink Cluster is an abstract configuration defined in Kafka Linking for the migration target cluster. The Sink Cluster specifically refers to an AutoMQ instance (Kafka Linking supports only AutoMQ instances as Sink Clusters), containing configuration information such as the target cluster's access point, ACL rules, alias, etc.

### Mirror Topic

Mirror Topic is the basic unit used in Kafka Linking to synchronize and migrate a Topic from the source cluster to the target cluster. Each Mirror Topic corresponds to a specific Topic in the source cluster and serves as the unique representation of that Topic in the target cluster.

### Source Topic

Source Topic is the expression of the Topic in the source cluster for Kafka Linking. In each Kafka Link, the Source Topic and Mirror Topic correspond one-to-one.

### Consumer Group

Kafka Linking supports copying consumption progress information from the source cluster's Consumer Group and synchronizing it to the target cluster. Users can specify the Source Cluster's Consumer Group and configure it to the Sink Cluster's designated Consumer Group.

## Kafka Linking architecture

Unlike MirrorMaker2, which implements data synchronization based on Connectors, the Kafka Linking migration tool reconstructs two paths: Byte-to-Byte replication path and Producer proxy path:

* **Byte-to-Byte Replication Path:** This is used for synchronizing data from the Source Cluster to the Sink Cluster, ensuring data integrity and position consistency.

* **Producer Proxy Path:** Traditional MirrorMaker2 requires Producers to be completely shut down before switching to the target cluster to avoid data disorder caused by simultaneous writing to both clusters. AutoMQ Kafka Linking provides a Producer proxy path that supports traffic proxying back to the source cluster during the switching process and subsequent unified switching, enabling zero-downtime switching.

### Byte-to-Byte replication path

<img alt="The data synchronization link of Kafka Linking can achieve the following effects:" />

Kafka Linking's data synchronization links can achieve the following effects:

* Topic partition numbers are completely consistent.

* Message full synchronization, with offsets exactly aligned with the source cluster.

* Consumer offsets are synchronized; the specific partitions' consumer offsets for specific Consumer Groups in the source cluster are fully synchronized to the target cluster, maintaining consistency.

### Producer proxy link

<img alt="Producer proxy link provided by Kafka Linking has the following capabilities:" />

Producer proxy link provided by Kafka Linking has the following capabilities:

* Supports Producer rolling-switch access point migration, with no downtime during the switch, ensuring no disruption to writing processes.

* Supports normal read/write operations on the source cluster during migration, with no impact, allowing applications to switch as needed.

## Next steps

To understand the background and functionality of Kafka Linking, it is recommended to follow this sequence:

* **Prerequisites sorting:** Identify the characteristics of the source cluster and business relationships that need migration, plan the migration batches, and check prerequisite resources. [Prerequisites▸](/automq-cloud/migrate-to-automq/prerequisites)

* **Review migration best practices:** Create a runbook for each batch based on endpoints, client types, offset sources, and rollback requirements. [Kafka Linking best practices▸](/automq-cloud/migrate-to-automq/kafka-linking-best-practices)

* **Execute the migration plan:** After completing the prerequisite review, follow the migration plan step by step. [Executing migration▸](/automq-cloud/migrate-to-automq/executing-migration)


# Prerequisites
Source: https://docs.automq.com/automq-cloud/migrate-to-automq/prerequisites

Ensure you verify all prerequisites before migrating to AutoMQ to ensure no impact on production business operations. AutoMQ provides cloud-native scalability and 100% API compatibility with Kafka.

Before migrating to AutoMQ, make sure to verify the following conditions and constraints to ensure that your production environment remains unaffected.

## Function compatibility assessment

Due to version support differences, AutoMQ provides only Kafka Server services and does not offer managed services for other ecosystem components. Therefore, before migrating to AutoMQ, you need to confirm whether the following functionalities are in use and refer to the following handling guidelines:

* **Kafka Server:** AutoMQ offers a more cost-effective and flexible Kafka Server, **compatible with Apache Kafka versions 0.11.x \~ 3.9.x**. If your current Apache Kafka Server version falls outside this supported range, please contact us for support and updates.

* **Kafka Client:** AutoMQ is compatible with the original Apache Kafka Client SDK. Ensure that your existing SDK version is **between 0.10.x \~ 3.9.x** to maintain compatibility.

In addition to the differences in service components, before migrating to AutoMQ, you need to confirm whether the following functionalities are in use and handle them according to the guidance below.

* **Access Point Protocols:** AutoMQ supports protocols such as PLAINTEXT, SASL\_PLAINTEXT, SASL\_SSL, and mTLS for internal network access. If your application uses other protocols to connect to the cluster, it is recommended to consult with AutoMQ technical staff to evaluate the risks associated with modifications.

## Business scope sorting

For migrating from Apache Kafka to AutoMQ in a production environment, it is generally recommended to perform the migration in batches. This involves gradually selecting Topics and Consumer Groups from the source cluster and migrating them in batches based on business chains and business scope.

Therefore, it is recommended to perform business scope sorting according to the following process:

1. **Inventory of Source Cluster Resources:** It is advisable to use the Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster and classify and organize them by business system. Clean up invalid or unclaimed Topics and Consumer Groups in advance.

<Tip>
  It is recommended to check the Topics in the source cluster and clean up Topics and Consumer Groups that have had no writes or subscriptions for a long time in advance to reduce the scope of the migration.
</Tip>

2. **Batch Division by Business System:** In a production environment, it is recommended to plan migration batches by business system. Start by migrating offline and non-core business systems, and gradually expand the migration scope to avoid unexpected risks.

3. **Coordinate with Business Personnel for Migration:** During the process of migrating Apache Kafka® to AutoMQ, it is necessary for business personnel to cooperate in changing access points and restarting applications. Therefore, before executing the migration plan, it is important to coordinate with the business personnel for their cooperation.

<Info>
  Apache Kafka Client does not disconnect from the source cluster after establishing a server connection. Therefore, even if domain names are used to access the server, it is still necessary to restart the client application to connect to the new AutoMQ cluster.
</Info>

## Cluster environment preparation

### Create an AutoMQ instance and ensure capacity meets your requirements

Before executing the migration tasks, an AutoMQ instance needs to be created in advance, and it must be ensured that the AutoMQ instance is in a running state. When creating an AutoMQ instance, please pay attention to the following considerations:

* **Network Connectivity Check:** When creating an AutoMQ instance, it is necessary to select a VPC and subnet. Please ensure that the VPC subnet where the AutoMQ instance is located can communicate with the current Apache Kafka. Otherwise, data migration and synchronization cannot be executed subsequently.

<Tip>
  To utilize Kafka Linking, ensure that your AutoMQ instance version is >= 5.0. If your version is below this requirement, please upgrade the instance version first.
</Tip>

* **Instance Capacity Specification Check:** When creating an AutoMQ instance, it is crucial to confirm the throughput scale of the source cluster. Based on the **peak read/write throughput** of the source cluster, select the appropriate AutoMQ instance specifications to avoid system failures due to insufficient capacity during the later stages of migration.

<Tip>
  When using Kafka Linking, it is recommended to reserve **an additional 50% capacity** on top of the peak throughput specification of the source cluster for the target AutoMQ instance to account for the consumption of cluster capacity by Kafka Linking.

  *Example:*

  *If the peak throughput of the source Kafka cluster is 100MBps, and you are migrating to AutoMQ using Kafka Linking, it is advisable to reserve 100\*150% = 150MBps during migration. After the migration is completed, the capacity can be reduced to match the specifications of the source cluster.*
</Tip>

For detailed instructions on creating an instance, please refer to [Experience AutoMQ▸](/automq-cloud/getting-started/experience-automq).

### Retrieve source cluster configuration

The principle of migrating from Apache Kafka® to AutoMQ is to use migration tools to synchronize existing data and configurations from the source cluster to the new cluster, then switch client configurations at the appropriate time to enable access to the new cluster.

In the process described above, synchronizing data and configurations from the source cluster requires obtaining the source cluster's configuration. Therefore, it is necessary to confirm in advance whether the following configuration information is prepared:

* **Source Cluster Access Points:** Configure the source cluster's bootstrap endpoint list. Use stable DNS names or the formal endpoint provided by the Kafka service. Do not use Kubernetes Pod IPs or node IP and NodePort combinations. The AutoMQ data plane must also reach every broker address returned through `advertised.listeners`. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices) for the full validation checklist.

* **Source Cluster Access Protocol:** The migration tool needs to access the source cluster, and you must confirm the access protocol used by the source cluster. Currently, Kafka Linking supports PLAINTEXT, SASL\_PLAINTEXT, SASL\_SSL, and mTLS protocols.

* **Source Cluster Username and Password (Optional):** If the source cluster requires access using the SASL\_PLAINTEXT or SASL\_SSL protocol, you need to configure the username and password and ensure that the **username has permission to subscribe to all topics**.

* **Source Cluster Network Firewall Policy (Optional):** If the source cluster is configured with network firewall or security group policies, the subnet of AutoMQ needs to be allowed.

## Next steps

Once the prerequisites are sorted out and met, you can proceed with the subsequent migration process. It is recommended to handle it in the following order:

* **Review migration best practices:** Create a batch runbook based on client types, offset sources, and rollback requirements. [Kafka Linking best practices▸](/automq-cloud/migrate-to-automq/kafka-linking-best-practices)

* **Implement the migration plan:** After completing the prerequisite review, follow the migration plan step by step. [Executing migration▸](/automq-cloud/migrate-to-automq/executing-migration)


# Glossary
Source: https://docs.automq.com/automq-cloud/overview/glossary

Definitions of the primary resources and concepts used in AutoMQ Cloud.

<Info>
  In this documentation, AutoMQ product service provider, AutoMQ service provider, and AutoMQ refer to AutoMQ HK Limited.
</Info>

## Terminology Details

### Environment

An Environment is the management boundary for the infrastructure, region, network, and authorization settings used by AutoMQ Cloud. AutoMQ resources are created and managed within an Environment.

### Environment Status

Environment Status indicates the current lifecycle or operational state of an Environment, such as creating, running, updating, or service disruption.

### Instance

An Instance is an AutoMQ service resource that corresponds to an Apache Kafka® cluster. Its configuration defines the cluster lifecycle, capacity, network, security, and feature settings.

### Member

A Member is an account identity that can access AutoMQ Cloud resources. AutoMQ Cloud provides the Admin, Operator, and Viewer roles. For more information, see [Identity and access management overview](/automq-cloud/manage-identities-and-access/overview).

### ServiceAccount

A ServiceAccount is a non-human AutoMQ Cloud account identity used for programmatic access. It has a name, description, status, and source, and is authorized through AutoMQ identity and access management.

### Connect Cluster

A Connect Cluster is a managed Kafka Connect worker cluster. It defines the runtime version, plugins, compute capacity, deployment location, worker configuration, and identity used to run Connectors.

### Connector

A Connector is a source or sink workload that runs on a Connect Cluster. It uses a connector plugin and configuration to transfer data between Kafka and an external system.

### OpsTunnel

OpsTunnel is a time-bound managed operations channel between an AutoMQ BYOC (Bring Your Own Cloud) environment and the AutoMQ operations service. It is used for authorized troubleshooting access and can be closed by the environment administrator.

### Data Bucket

A Data Bucket is an object storage bucket used by an AutoMQ instance to store Kafka data and metadata.

### Ops Bucket

An Ops Bucket is an object storage bucket used to store logs, metrics, and diagnostic data for managed operations. Access is controlled separately from the Data Bucket.

### Payment

A Payment identifies the payment method associated with an AutoMQ subscription. It records the payment type, display name, and, when applicable, the customer identifier assigned by the payment provider.

### Kafka Links

Kafka Links is an AutoMQ capability for continuously replicating Kafka topics between source and destination instances. It can be used for data migration, disaster recovery, and cross-region data distribution.

### EBS (Elastic Block Store) WAL (Write-Ahead Log)

EBS WAL is a WAL mode that stores data on Amazon EBS volumes attached to AutoMQ broker nodes.

### NFS WAL

NFS WAL is a write-ahead log mode that stores WAL data on a shared Network File System that is accessible to AutoMQ broker nodes.

### S3 WAL

S3 WAL is a write-ahead log mode that stores WAL data in S3-compatible object storage.


# Restrictions
Source: https://docs.automq.com/automq-cloud/overview/restrictions

AutoMQ Cloud enforces standardized parameters and quotas across its products. Learn about parameter limits, resource quotas, and deployment region restrictions.

## Overview

To prevent various edge cases in production caused by unclear behavior definitions, AutoMQ Cloud enforces and standardizes parameters, quotas, and behaviors across its products. Users should ensure they do not violate these constraints while using the system. If these constraints do not align with actual usage scenarios, please promptly submit a support ticket for assistance, as referenced in [Obtaining Services▸](/automq-cloud/support/obtaining-services).

## Limitations on Parameters

The names and descriptions of resources within the following parameters have restricted values that cannot be adjusted. Please adhere strictly to the specifications to prevent system processing anomalies caused by special characters or excessive length.

### Public Parameter Limitations

<table>
  <thead>
    <tr>
      <th>
        Parameter Items

        <br />
      </th>

      <th>
        **Limit Value**

        <br />
      </th>

      <th>
        **Description**

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Instance Name

        <br />
      </td>

      <td>
        <ul>
          <li>Character Restrictions: Supports only Chinese characters, letters a-z or A-Z, numbers 0-9, underscores (\_), and hyphens (-).</li>

          <li>Length Restrictions: 3-64 characters.</li>
        </ul>
      </td>

      <td>
        Names and notes should preferably use simple, common phrases and abbreviations. Avoid using special characters that may cause display issues.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Instance Description

        <br />
      </td>

      <td>
        <ul>
          <li>Character Restrictions: UTF-8 visible characters.</li>

          <li>Length limit: 0 to 255 characters.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Environment Member Name

        <br />
      </td>

      <td>
        <ul>
          <li>Character Restrictions: Only supports letters a-z or A-Z, digits 0-9, underscores (\_), and hyphens (-).</li>

          <li>Length Restrictions: 3-64 characters.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Environment Member Password

        <br />
      </td>

      <td>
        <ul>
          <li>Character Restrictions: Only supports visible characters.</li>

          <li>Length restriction: 8 to 16 characters.</li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

### Instance-Level Configuration

<table>
  <thead>
    <tr>
      <th>
        Parameter Items

        <br />
      </th>

      <th>
        **Limit Value**

        <br />
      </th>

      <th>
        **Description**

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_log.retention.ms">log.retention.ms</a>

        <br />

        Message Retention Period

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: 7 days</li>

          <li>Range: 1 hour \~ 60 days</li>

          <li>Can be customized and modified</li>
        </ul>
      </td>

      <td>
        The message retention period affects storage space consumption and cost. It is advisable for businesses to assess this carefully. If unsatisfactory, please
        <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
        Seek assistance.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_message.max.bytes">message.max.bytes</a>

        <br />

        Maximum message size

        <br />
      </td>

      <td>
        <ul>
          <li>Default: 20 MiB</li>

          <li>Constraint Value: 0 B \~ 50 MiB</li>

          <li>Custom modification supported</li>
        </ul>
      </td>

      <td>
        If the current parameter limit does not meet your needs, please
        <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
        seek assistance.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_offsets.retention.minutes">offsets.retention.minutes</a>

        <br />

        Consumption Progress Retention Duration

        <br />
      </td>

      <td>
        <ul>
          <li>Limit: 7 days</li>

          <li>Customization not supported</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_fetch.max.bytes">fetch.max.bytes</a>

        <br />

        Single Data Fetch Limit

        <br />
      </td>

      <td>
        <ul>
          <li>Limit: 55 MB</li>

          <li>Custom modifications are not supported.</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Single partition write throughput limit

        <br />
      </td>

      <td>
        <ul>
          <li>Limit: 4 MBps</li>

          <li>Custom modifications are not supported.</li>
        </ul>
      </td>

      <td>
        There are limitations on single partition read/write throughput. It is recommended that applications use different message keys to distribute the read/write load across different partitions to avoid partition hotspots. Please
        <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
        Seeking Assistance.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Single Partition Read Throughput Limit

        <br />
      </td>

      <td>
        <ul>
          <li>Limit: 4 MBps</li>

          <li>Customization not supported</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_auto.create.topics.enable">auto.create.topics.enable</a>

        <br />

        Auto-create Topic Switch

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: true</li>

          <li>Constraint range: true/false</li>

          <li>Can be customized and modified</li>
        </ul>
      </td>

      <td>
        It is recommended to disable the automatic creation of topics and instead manage all topics through the control system and Admin API to prevent them from becoming unmanageable.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#brokerconfigs_num.partitions">num.partitions</a>

        <br />

        Default number of partitions for created topics

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: 1</li>

          * Limit range: 1-1024

          <li>Can be customized and modified</li>
        </ul>
      </td>

      <td>
        The default number of partitions to create for a topic when using automatic topic creation should be set reasonably at the cluster level to avoid consuming too many of the partition quota with automatically created topics.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        automq.zone.cidr.blocks

        <br />

        Client source CIDR availability zone identification

        <br />
      </td>

      <td>
        <ul>
          * Default value: None

          * Limit range: None

          <li>Can be customized and modified</li>
        </ul>
      </td>

      <td>
        The server matches the CIDR set in the current rule with the client's source IP to determine the client's corresponding availability zone. This configuration is used to optimize and eliminate cross-availability zone traffic for clients. The format specification for CIDR is \{zoneId}@\{cidr}\<>\{zoneId}@\{cidr}. Different availability zones are separated by "\<>", while multiple CIDR segments under the same availability zone are separated by ",".

        <br />

        Example:

        <br />

        <ul>
          <li>Configuring multiple availability zones: az-1\@10.0.0.1/30\<>az-2\@10.0.0.2/30.</li>

          <li>Multiple CIDR segments under the same availability zone: az-1\@10.0.0.1/30,10.0.0.2/30</li>
        </ul>

        <Tip>
          In the AWS environment, the availability zone identifier should be set as ZoneID rather than ZoneName. Regarding the difference between the two
          [Reference Documentation](https://docs.aws.amazon.com/global-infrastructure/latest/regions/az-ids.html)
          .
        </Tip>

        <br />
      </td>
    </tr>
  </tbody>
</table>

### Topic-Level Configuration

<table>
  <thead>
    <tr>
      <th>
        Parameter Items

        <br />
      </th>

      <th>
        **Limit Value**

        <br />
      </th>

      <th>
        **Description**

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_compression.type">compression.type</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: producer</li>

          <li>Constraint Value:</li>

          <ul>
            <li>Zstd</li>

            <li>lz4</li>

            <li>Producer</li>

            <li>Uncompressed</li>

            <li>Snappy</li>

            <li>gzip</li>

            <ul>
              <li>Custom modification supported</li>
            </ul>
          </ul>
        </ul>
      </td>

      <td>
        Set the final compression type for the Topic. This configuration supports ('gzip', 'snappy', 'lz4', 'zstd'); in addition to these options, it also supports 'uncompressed', meaning no compression is applied. The 'producer' configuration value refers to retaining the compression type set by the producer.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_cleanup.policy">cleanup.policy</a>

        <br />

        Message cleanup policy

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: delete</li>

          <li>Range limitation:</li>

          <ul>
            <li>delete: Periodically remove the oldest messages based on the retention time.</li>

            <li>compact: Periodically merge messages based on their keys, retaining only the latest version.</li>

            <ul>
              <li>Can be customized and modified</li>
            </ul>
          </ul>
        </ul>
      </td>

      <td>
        In most business scenarios, it is recommended to use the delete type, and only use the compact type in scenarios where preserving state data is required.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_retention.ms">retention.ms</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default Value: Uses current cluster configuration</li>

          <li>Limit value: 1 hour to 60 days.</li>

          <li>Custom modification supported</li>
        </ul>
      </td>

      <td>
        Configure a custom retention period for messages in a topic, applicable only to topics with a deletion clean-up policy. This configuration will override the default cluster settings.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_max.message.bytes">max.message.bytes</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default Value: Uses current cluster configuration</li>

          <li>Constraint Value: 0 B \~ 50 MiB</li>

          <li>Custom modification supported</li>
        </ul>
      </td>

      <td>
        Set the maximum allowable record batch size for the current topic, and any messages exceeding this limit will be rejected.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_message.timestamp.type">message.timestamp.type</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: CreateTime</li>

          <li>Constraint Value:</li>

          <ul>
            <li>LogAppendTime</li>

            <li>CreateTime</li>

            <ul>
              <li>Can be customized and modified</li>
            </ul>
          </ul>
        </ul>
      </td>

      <td>
        Define whether the timestamp in the message corresponds to the message's creation time or the log append time. This value should be either 'create time' or 'log append time'.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_retention.bytes">retention.bytes</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: -1</li>

          <li>Custom modification supported</li>
        </ul>
      </td>

      <td>
        This setting applies to topics with a delete cleanup policy and controls the maximum retention space for each partition. By default, there is no size limit, only a time limit. This restriction is at the partition level, so multiply it by the number of partitions to compute the topic-level retention space. This configuration overrides the cluster's default settings.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        <a href="https://kafka.apache.org/34/documentation.html#topicconfigs_delete.retention.ms">delete.retention.ms</a>

        <br />
      </td>

      <td>
        <ul>
          <li>Default value: 1 day</li>

          <li>Limit value: 0 hours \~ 60 days</li>

          <li>Custom modification supported</li>
        </ul>
      </td>

      <td>
        Set the retention time for deletion tombstones for Compact-type Topics, applicable only to Compact-type Topics. This configuration requires consumers to read messages within this timeframe to obtain the last valid snapshot; otherwise, they may read incomplete data.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic Naming

        <br />
      </td>

      <td>
        <ul>
          <li>Character Restrictions: Only supports letters a\~z or A\~Z, numbers 0\~9, underscores (\_), hyphens (-), and dots (.).</li>

          <li>Length Limit: 1\~250 characters.</li>

          <li>Other restrictions: Cannot start with an underscore (\_).</li>
        </ul>
      </td>

      <td>
        None.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic Notes

        <br />
      </td>

      <td>
        <ul>
          <li>Character restrictions: Only supports Chinese characters, letters a\~z or A\~Z, numbers 0\~9, and underscore (\_).</li>

          <li>Length restrictions: 1\~256 characters.</li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

## Resource Quota Limits

AutoMQ Cloud has established initial constraints on certain performance metrics and parameters based on its operational experience in large-scale production environments. In most cases, the default constraint range will meet your needs. However, if there are special circumstances where these do not suffice, please promptly submit a support ticket for assistance.

### Environmental Public Resource Quota Limits

<table>
  <thead>
    <tr>
      <th>
        **Limit Item**

        <br />
      </th>

      <th>
        **Limit Value**

        <br />
      </th>

      <th>
        **Description**

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Total number of product instances allowed within a single environment

        <br />
      </td>

      <td>
        Refer to subscription plan version constraints

        <br />
      </td>

      <td>
        If current constraints are not met, seek help through a support ticket.
        <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
        .

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Number of member accounts in a single environment:

        <br />
      </td>

      <td>
        100

        <br />
      </td>
    </tr>
  </tbody>
</table>

### Instance Quota Limitations:

AutoMQ Cloud has established initial constraints on certain performance metrics and parameters based on its operational experience in large-scale production environments. In most cases, the default constraint range will meet your needs. However, if there are special circumstances where these do not suffice, please promptly submit a support ticket for assistance.

<table>
  <thead>
    <tr>
      <th>
        **Limit Item**

        <br />
      </th>

      <th>
        **Limit Value**

        <br />
      </th>

      <th>
        **Description**

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Single Instance (Cluster) Computing Specification

        <br />
      </td>

      <td>
        3AKU to 500AKU, offering the following capabilities:

        <br />

        <ul>
          <li>Write Throughput: 60MBps\~10000MBps</li>

          <li>Read Throughput: 60MBps\~10000MBps</li>
        </ul>
      </td>

      <td>
        The instance's compute specification denotes the maximum throughput capacity for sending and receiving messages on a single instance (cluster).

        <br />

        If there's a need for larger-scale clusters, please
        <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
        apply to raise the specification limits.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Limits on the Number of Topics per Instance

        <br />
      </td>

      <td>
        Defined by the compute specifications of the instance; see specification limits for further details.
        <a href="/automq-cloud/subscriptions-and-billings/byoc-env-billings/billing-instructions-for-byoc">Billing Instructions for BYOC▸</a>
        .

        <br />
      </td>

      <td>
        For security and stability purposes, it's advisable to separate different business operations into distinct instances to avoid consolidating all operations into one instance.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Maximum Number of Partitions per Single Instance (Cluster)

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Maximum Request QPS per Single Instance (Cluster)

        <br />
      </td>
    </tr>
  </tbody>
</table>

## AutoMQ BYOC Deployment Region Restrictions

<Tabs>
  <Tab title="AWS">
    | **Code**<br />       | **Name**<br />                  |
    | -------------------- | ------------------------------- |
    | us-east-1<br />      | US East (N. Virginia)<br />     |
    | us-east-2<br />      | US East (Ohio)<br />            |
    | us-west-1<br />      | US West (N. California)<br />   |
    | us-west-2<br />      | US West (Oregon)<br />          |
    | af-south-1<br />     | Africa (Cape Town)<br />        |
    | ap-east-1<br />      | Asia Pacific (Hong Kong)<br />  |
    | ap-south-2<br />     | Asia Pacific (Hyderabad)<br />  |
    | ap-southeast-3<br /> | Asia Pacific (Jakarta)<br />    |
    | ap-southeast-5<br /> | Asia Pacific (Malaysia)<br />   |
    | ap-southeast-4<br /> | Asia Pacific (Melbourne)<br />  |
    | ap-south-1<br />     | Asia Pacific (Mumbai)<br />     |
    | ap-northeast-3<br /> | Asia Pacific (Osaka)<br />      |
    | ap-northeast-2<br /> | Asia Pacific (Seoul)<br />      |
    | ap-southeast-1<br /> | Asia Pacific (Singapore)<br />  |
    | ap-southeast-2<br /> | Asia Pacific (Sydney)<br />     |
    | ap-northeast-1<br /> | Asia Pacific (Tokyo)<br />      |
    | ca-central-1<br />   | Canada (Central)<br />          |
    | ca-west-1<br />      | Canada West (Calgary)<br />     |
    | eu-central-1<br />   | Europe (Frankfurt)<br />        |
    | eu-west-1<br />      | Europe (Ireland)<br />          |
    | eu-west-2<br />      | Europe (London)<br />           |
    | eu-south-1<br />     | Europe (Milan)<br />            |
    | eu-west-3<br />      | Europe (Paris)<br />            |
    | eu-south-2<br />     | Europe (Spain)<br />            |
    | eu-north-1<br />     | Europe (Stockholm)<br />        |
    | eu-central-2<br />   | Europe (Zurich)<br />           |
    | il-central-1<br />   | Israel (Tel Aviv)<br />         |
    | me-south-1<br />     | Middle East (Bahrain)<br />     |
    | me-central-1<br />   | Middle East (UAE)<br />         |
    | sa-east-1<br />      | South America (São Paulo)<br /> |
  </Tab>

  <Tab title="GCP">
    | Region ID<br />               | Name<br />            |
    | ----------------------------- | --------------------- |
    | us-west-1<br />               | Oregon<br />          |
    | asia-south1<br />             | Mumbai<br />          |
    | asia-south2<br />             | Delhi<br />           |
    | asia-east1<br />              | Changhua County<br /> |
    | asia-east2<br />              | Hong Kong<br />       |
    | asia-northeast1<br />         | Tokyo<br />           |
    | asia-northeast2<br />         | Osaka<br />           |
    | asia-northeast3<br />         | Seoul<br />           |
    | asia-southeast1<br />         | Jurong West<br />     |
    | australia-southeast1<br />    | Sydney<br />          |
    | australia-southeast2<br />    | Melbourne<br />       |
    | europe-central2<br />         | Warsaw<br />          |
    | europe-north2<br />           | Hamina<br />          |
    | europe-southwest1<br />       | Madrid<br />          |
    | europe-west1<br />            | St. Ghislain<br />    |
    | europe-west2<br />            | London<br />          |
    | europe-west3<br />            | Frankfurt<br />       |
    | europe-west4<br />            | Eemshaven<br />       |
    | europe-west6<br />            | Zurich<br />          |
    | europe-west8<br />            | Milan<br />           |
    | europe-west9<br />            | Paris<br />           |
    | northamerica-northeast1<br /> | Montréal<br />        |
    | northamerica-northeast2<br /> | Toronto<br />         |
    | southamerica-east1<br />      | Osasco<br />          |
    | us-central1<br />             | Council Bluffs<br />  |
    | us-east1<br />                | Moncks Corner<br />   |
    | us-east4<br />                | Ashburn<br />         |
    | us-west1<br />                | The Dalles<br />      |
    | us-west2<br />                | Los Angeles<br />     |
    | us-west3<br />                | Salt Lake City<br />  |
    | us-west4<br />                | Las Vegas<br />       |
  </Tab>

  <Tab title="Azure">
    | Region ID<br />          | Name<br />                 |
    | ------------------------ | -------------------------- |
    | eastus<br />             | East US<br />              |
    | westus2<br />            | West US 2<br />            |
    | westeurope<br />         | West Europe<br />          |
    | southeastasia<br />      | Southeast Asia<br />       |
    | japaneast<br />          | Japan East<br />           |
    | australiaeast<br />      | Australia East<br />       |
    | germanywestcentral<br /> | Germany West Central<br /> |
    | uaenorth<br />           | UAE North<br />            |
  </Tab>
</Tabs>


# What Is AutoMQ Cloud
Source: https://docs.automq.com/automq-cloud/overview/what-is-automq-cloud

AutoMQ Cloud is the portal for creating and managing AutoMQ BYOC environments in your cloud account.

## Overview

AutoMQ Cloud is an environment management portal provided by AutoMQ HK Limited. Use the portal to obtain control plane installation commands and to create and manage AutoMQ BYOC (Bring Your Own Cloud) environments in your cloud account.

AutoMQ BYOC deploys Apache Kafka-compatible control plane and data plane workloads in your cloud account. AutoMQ provides managed operations capabilities for deployment, version upgrades, scaling, observability, and incident response.

Because the AutoMQ BYOC control plane and data plane run in your cloud account, Kafka data remains in the virtual private cloud (VPC) and object storage resources that you control.

<img />

## Core Advantages

AutoMQ Cloud offers the following advantages:

#### Pay-as-you-go, with a 50% Reduction on Cloud Bills

AutoMQ's new cloud-native architecture fully leverages **object storage** and cloud **elastic computing** resources, providing significant cost advantages over Apache Kafka and other competitor solutions.

By enabling AutoMQ Cloud, you can fully utilize resources on demand and pay-as-you-go, with a 50% reduction on cloud bills.

#### Managed Operations and SLA Support

AutoMQ BYOC includes managed operations for deployment, configuration changes, version upgrades, monitoring, and incident response. The AutoMQ team provides 24/7 operations support and the service-level commitments defined in the applicable agreement.

#### Supports BYOC, Resources Deployed in User VPC, No Cross-network Data

The BYOC model of AutoMQ Cloud means that cloud resources are deployed in the user's VPC. Users can access AutoMQ without cross-network connectivity and can also utilize existing cloud account resource discounts.

#### Enterprise-Level Capabilities Out of the Box

AutoMQ Cloud enhances AutoMQ Open Source with enterprise-level features such as auto-scaling, seconds-level partition reassignment, and integrated system observability. Users can use these enhancements immediately upon service activation without additional development.

#### Native Multi-Cloud Support

The cloud-native architecture of AutoMQ relies solely on Public Cloud-based object storage and virtual machine products, inherently supporting multi-cloud deployments. Enterprise customers can activate and use AutoMQ Cloud within mainstream cloud environments.

## Choosing the Service Version

AutoMQ offers AutoMQ BYOC and AutoMQ Software for enterprise customers. AutoMQ BYOC is a fully managed cloud service deployed in your cloud account. AutoMQ Software is a software distribution for private data center deployments.

Enterprise customers can refer to the table below to select the most suitable service version.

| Comparison item        | AutoMQ BYOC                                                                                                                   | AutoMQ Software                                                                          |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Service model          | Fully managed BYOC service \[1]                                                                                               | Self-managed software \[2]                                                               |
| Deployment environment | Public cloud                                                                                                                  | Private data center                                                                      |
| Resource ownership     | IaaS resources are owned by your cloud account                                                                                | Infrastructure resources are owned by you                                                |
| Data sovereignty       | Data remains within your VPC                                                                                                  | Data remains within your private network                                                 |
| Operations             | AutoMQ provides deployment, version upgrades, parameter tuning, monitoring and alerts \[3], and online incident response \[4] | You perform all operations; AutoMQ provides remote technical support                     |
| Payment method         | Pay-as-you-go or annual subscription                                                                                          | Annual subscription or perpetual license                                                 |
| Cost structure         | AutoMQ subscription fee and cloud resource charges paid to the cloud provider                                                 | Software license fee and customer-provided infrastructure costs                          |
| Technical support      | Expert technical support and 24/7 online emergency response                                                                   | Expert technical support and 8/5 online emergency response                               |
| Recommended use        | Public cloud workloads that require resources and data to remain in the customer's cloud account                              | Workloads that require private data center deployment and complete environment isolation |

**Annotation Notes:**

\[1] AutoMQ BYOC deploys the AutoMQ control plane and data plane in your cloud account and provides a fully managed service.

\[2] AutoMQ Software deploys the control plane and data plane in your private data center. You are responsible for operations, and AutoMQ provides technical support and version upgrades.

\[3] In BYOC environments, logs, metrics, and diagnostic data are stored in a separate Ops Bucket. You authorize the AutoMQ service account to access this bucket so that the AutoMQ operations platform can analyze service health and support managed operations.

\[4] AutoMQ Cloud monitors instance health and can isolate unhealthy nodes automatically. The AutoMQ engineering team performs subsequent root-cause analysis when required.

## Related Content

* **Quick Experience of AutoMQ Cloud:** [Overview▸](/automq-cloud/getting-started/overview)


# Prepaid Billing Overview
Source: https://docs.automq.com/automq-cloud/subscriptions-and-billings/byoc-env-billings/billing-instructions-for-byoc

AutoMQ's detailed billing guide explains prepaid subscription fees based on AKU cluster size in a BYOC environment, emphasizing cloud-native scalability, Kafka compatibility, and cost-efficient resource allocation.

The billing overview section offers detailed information about the billing components associated with the AutoMQ Cloud product in a BYOC (Bring Your Own Cloud) environment.

<Info>
  The terms cloud vendors and public cloud vendors mentioned in this document refer to major cloud service providers such as AWS, Google Cloud, Azure, Alibaba Cloud, OCI, etc.
</Info>

## AutoMQ Subscription Fees (Payable to AutoMQ)

To use AutoMQ Cloud in a BYOC environment, users are required to pay subscription fees based on the actual cluster size. AutoMQ evaluates subscription fees with the following scale indicators:

* **Message Processing Specification AKU (AutoMQ Kafka Unit):** Mandatory, the AKU message processing specification is used to measure the scale of computing resources allocated during the sending and receiving of messages within a Kafka cluster. The AKU specification is directly proportional to the cluster's scale pressure.

### Message Processing Specification AKU Costs

#### Metric Constraints

The Message Processing Specification AKU refers to the computational processing resources allocated within a cluster during message transmission. AKU considers resource consumption factors such as computational power, storage IOPS, and network throughput.

Allocating the appropriate AKU specifications for each AutoMQ instance ensures that the required message transmission throughput capacity is achieved. According to benchmark performance test results, adding 1 AKU specification delivers the following performance capabilities:

<table>
  <thead>
    <tr>
      <th>
        Capabilities Provided by 1 AKU

        <br />
      </th>

      <th>
        Description

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Read/Write Throughput

        <br />
      </td>

      <td>
        Write 30 MiB/s or Read 60 MiB/s

        <br />
      </td>

      <td>
        Core performance metrics and load for message read/write calls. If the predefined specifications are exceeded, it may result in slower service response times, increased RT, or rate-limiting failures.

        <br />

        <Tip>
          <br />

          *Example: For a specific AutoMQ instance, the write throughput is 60 MiB/s and the read throughput is 240 MiB/s. This instance's write throughput consumes 60/30 = 2 AKU, read throughput consumes 240/60 = 6 AKU, totaling 8 AKU.*

          <br />
        </Tip>

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Client Request Rate

        <br />
      </td>

      <td>
        800 requests per second

        <br />
      </td>

      <td>
        The application interacts with the server using the Kafka Producer and Consumer SDK, managing request rate and load. If the rate surpasses the predefined specifications, it may lead to throttling or longer response times (RT).

        <br />

        Request types include:

        <br />

        <ul>
          <li>Produce</li>

          <li>FetchConsumer</li>

          <li>CommitOffset</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Upper limit on partition count

        <br />
      </td>

      <td>
        1125 pieces

        <br />
      </td>

      <td>
        The number of partitions limits the amount of metadata an instance can handle simultaneously. Exceeding the specified limit may result in the following situations:

        <br />

        <ul>
          <li>Inability to create new topics or expand partitions.</li>
        </ul>

        <Tip>
          <br />

          AutoMQ, using an S3-based shared storage architecture, compared to Apache Kafka at the same cluster scale,
          supports 10 times
          Partition performance, without concerns about performance degradation due to an excessive number of partitions.

          <br />

          If the default number of partitions is insufficient for your requirements, please
          <a href="/automq-cloud/support/obtaining-services">Obtaining Services▸</a>
          contact us.

          <br />
        </Tip>

        <br />
      </td>
    </tr>
  </tbody>
</table>

<Info>
  In a production environment, it is recommended that applications strictly adhere to the aforementioned specifications for processing capacity to evaluate resource consumption. This will allow for prompt scaling up or down as needed. This approach helps prevent excessive usage from overloading the cluster, which could impact service stability.
</Info>

#### Calculation Rules

Each AutoMQ instance (cluster) allows you to specify the desired AKU specifications when creating or modifying its specifications. The billing system records the real-time consumption of AKU numbers for each instance.

* Scope of Statistics: The count of AKUs for each AutoMQ Kafka instance.

* Statistics Method: An hourly cycle is used to record the peak value within each cycle.

* Aggregation Method: Statistics are aggregated at the instance level.

## Cloud resource fees (paid to cloud providers)

Deploying an AutoMQ instance involves resource consumption costs in addition to the subscription fee. Using Alibaba Cloud as an example:

| **Cloud Product**<br />                  | **Specifications and Usage**<br />                                                                                                                                                   | **Dependency Description**<br />                                                                                                                                                                |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| VM (Virtual Machine)<br />               | <ul><li>Specifications: Varies depending on the region.</li><li>Usage: Scales with the size of the user cluster.</li></ul>                                                           | <ul><li>AutoMQ Kafka's VM consumption changes with the size of the user cluster.</li><li>At least 1 VM is consumed to deploy the management interface even if no instance is created.</li></ul> |
| Elastic Block Service (Cloud Disk)<br /> | <ul><li>Specifications: PL1 type</li><li>Usage: Each Kafka data node uses 40GB.</li></ul>                                                                                            | <ul><li>AutoMQ Kafka uses cloud disks to store temporary message data.</li></ul>                                                                                                                |
| Object Storage<br />                     | <ul><li>Specifications: Standard disaster recovery storage</li><li>Usage: Two buckets per environment; storage capacity scales with user environment resource consumption.</li></ul> | <ul><li>Core storage dependency of the message cluster.</li><li>Charges to cloud provider are based on actual space usage and call volume.</li></ul>                                            |


# Usage-Based Billing for AWS
Source: https://docs.automq.com/automq-cloud/subscriptions-and-billings/byoc-env-billings/usage-based-billing

AutoMQ BYOC supports usage-based billing in AWS environments, where you pay only for what you consume with automatic tiered pricing discounts.

This document describes the usage-based billing model for AutoMQ BYOC (Bring Your Own Cloud) environments. With usage-based billing, you pay only for the resources you actually consume, with no upfront commitment or capacity reservation required.

<Info>
  As of August 2026, usage-based billing is available only for AWS environments. For other cloud environments, refer to [Prepaid Billing](/automq-cloud/subscriptions-and-billings/byoc-env-billings/billing-instructions-for-byoc).
</Info>

## Overview

AutoMQ BYOC supports a usage-based billing model that charges based on your actual resource consumption. This model is designed for customers who prefer pay-as-you-go flexibility without the need to estimate capacity in advance.

Key characteristics of usage-based billing:

* **Pay for what you use.** Charges are calculated based on actual consumption metrics, including data ingress, data egress, data retention, and cluster uptime.
* **Tiered pricing.** Unit prices decrease automatically as your monthly usage increases. The more you use, the lower the per-unit cost.
* **No partition fees.** AutoMQ does not charge for the number of partitions in your cluster.

## Billing Items

The following table summarizes the billing items under the usage-based billing model.

| **Billing Item** | **Unit** | **Description**                                                                         |
| ---------------- | -------- | --------------------------------------------------------------------------------------- |
| Data Ingress     | Per GiB  | The volume of data written to the cluster, measured after compression.                  |
| Data Egress      | Per GiB  | The volume of data read from the cluster, measured after compression.                   |
| Data Retention   | Per GiB  | The volume of data retained in the cluster.                                             |
| Cluster Uptime   | Per hour | The running duration of the cluster. As of August 2026, only the Pro tier is available. |

<Tip>
  All data volume metrics (ingress, egress, and retention) are measured based on compressed data size. For detailed pricing, visit the [AutoMQ Pricing](https://www.automq.com/pricing) page.
</Tip>

## Billing Rules

### Tiered Pricing

Usage-based billing applies a tiered pricing structure. As your monthly usage increases, the applicable unit price for each billing item decreases automatically. You do not need to select a pricing tier manually — the system applies the appropriate tier based on your cumulative monthly usage.

### Billing Cycle

* **Metering period:** Usage is metered continuously and aggregated on a monthly basis.
* **Invoice cycle:** Invoices are generated monthly through AWS Marketplace.
* **Settlement:** Payments are processed through your AWS Marketplace account according to your AWS billing schedule.

### Cluster Tier

As of August 2026, AutoMQ usage-based billing provides only the Pro cluster tier.

| **Cluster Tier** | **Description**                                                          |
| ---------------- | ------------------------------------------------------------------------ |
| Pro              | Suitable for production workloads with higher availability requirements. |

## Configure Usage-Based Billing

### Prerequisites

Before configuring usage-based billing, ensure the following:

* You have registered an [AutoMQ Cloud](https://www.automq.com) account.
* You have an AWS account with permissions to subscribe to products on AWS Marketplace.

### Configure Payment Method

Usage-based billing is settled through an AWS Marketplace subscription. The AWS Marketplace subscription corresponds to a payment method in AutoMQ Cloud. You can complete the configuration through either of the following approaches:

* **Option 1: Subscribe directly from AWS Marketplace.** Visit the [AutoMQ BYOC - Usage-Based Billing](https://aws.amazon.com/marketplace/pp/prodview-mtkwt73meb222?trk=176b570f-20dd-4b84-aa7e-cae53990fe91\&sc_channel=el\&source=automq) product page on AWS Marketplace, complete the subscription, and bind it to your AutoMQ Cloud account.
* **Option 2: Add a payment method from the AutoMQ Cloud Console.** Sign in to the AutoMQ Console, navigate to the **Billing** page, choose **Add Payment Method**, and follow the prompts to subscribe and bind through AWS Marketplace.

#### Steps

1. Visit the [AutoMQ BYOC - Usage-Based Billing](https://aws.amazon.com/marketplace/pp/prodview-mtkwt73meb222?trk=176b570f-20dd-4b84-aa7e-cae53990fe91\&sc_channel=el\&source=automq) product page and review the product information and pricing terms.
2. Choose **Subscribe** to complete the subscription.
3. After the subscription is created, choose **Set up your account** and follow the on-screen instructions to bind the subscription to your AutoMQ Cloud account.

<Info>
  Ensure that your AWS account has a valid payment method configured so that AWS Marketplace can process charges.
</Info>

### Start Using AutoMQ

After the payment method is configured, you can create clusters in your BYOC environment. Usage metering begins automatically when a cluster is running. No additional configuration is required.

### View Invoices and Usage

You can review your usage and billing details through the following channels:

* **AutoMQ Console:** Navigate to the **Billing** page to view usage summaries and billing breakdowns by cluster.
* **AWS Marketplace:** Sign in to the [AWS Marketplace Console](https://console.aws.amazon.com/marketplace) and navigate to **Manage subscriptions** to view invoices and payment history.


# Overview
Source: https://docs.automq.com/automq-cloud/subscriptions-and-billings/overview

Billing items, billing models, and payment methods for AutoMQ BYOC.

This article describes the billing components and payment methods for AutoMQ BYOC (Bring Your Own Cloud).

<Info>
  In this article, the terms AutoMQ product service provider, AutoMQ service provider, and AutoMQ specifically refer to AutoMQ HK Limited.
</Info>

## Billing Items

Using AutoMQ BYOC incurs the following two types of fees:

| **Fee Type**            | **Coverage**                                                                                                                                                              | **Payment Relationship**                  |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| **Subscription Fees**   | AutoMQ provides AutoMQ BYOC product and technical services                                                                                                                | Users pay AutoMQ                          |
| **Cloud Resource Fees** | Fundamental cloud resources consumed by the environment, including but not limited to:<ul><li>Virtual Hosts</li><li>Block Storage</li><li>Network Traffic, etc.</li></ul> | Users pay public cloud providers directly |

## Billing Models

AutoMQ BYOC supports two billing models for subscription fees: **Usage-Based Billing** and **Prepaid Billing**. Choose the model that meets your workload and cost management requirements.

### Usage-Based Billing

Usage-based billing charges you based on your actual resource consumption, with no upfront commitment required. This model applies automatic tiered pricing — the more you use, the lower the per-unit cost.

<Info>
  As of August 2026, usage-based billing is available only for AWS environments. Support for Google Cloud, Azure, and additional cloud environments is planned.
</Info>

Under this model, the following billing items apply:

| **Billing Item** | **Unit** | **Description**                                                                                                 |
| ---------------- | -------- | --------------------------------------------------------------------------------------------------------------- |
| Data Ingress     | Per GiB  | The volume of data written to the cluster, measured after compression.                                          |
| Data Egress      | Per GiB  | The volume of data read from the cluster, measured after compression.                                           |
| Data Retention   | Per GiB  | The volume of data retained in the cluster.                                                                     |
| Cluster Uptime   | Per hour | The running duration of the cluster, billed by cluster tier. As of August 2026, only the Pro tier is available. |

For detailed information, refer to [Usage-Based Billing▸](/automq-cloud/subscriptions-and-billings/byoc-env-billings/usage-based-billing).

### Prepaid Billing (AKU-Based)

The prepaid billing model requires you to purchase a subscription plan in advance on a **monthly or yearly** basis. Subscription fees are based on the AKU (AutoMQ Kafka Unit) specification, which measures the computing resources allocated for message processing within a Kafka cluster. No additional subscription fees are incurred during the subscription period.

This model is available for all supported cloud environments.

For detailed information about AKU specifications and calculation rules, refer to [Prepaid Billing▸](/automq-cloud/subscriptions-and-billings/byoc-env-billings/billing-instructions-for-byoc).

## Payment Methods

### Usage-Based Billing

Usage-based billing is settled through AWS Marketplace on a monthly basis. After subscribing to the AutoMQ product on AWS Marketplace, usage is metered automatically and invoices are generated monthly. For configuration instructions, refer to [Usage-Based Billing▸](/automq-cloud/subscriptions-and-billings/byoc-env-billings/usage-based-billing).

### Prepaid Billing

Depending on your requirements, prepaid subscription plans can be purchased through the following channels:

* Purchase subscription plans via the cloud marketplace.

* Purchase subscription plans through offline business contracts.

In prepaid mode, purchase a subscription and use the corresponding ProductSecret to activate the AutoMQ BYOC environment.


# Overview
Source: https://docs.automq.com/automq/architecture/overview

AutoMQ's Shared Storage architecture leverages cloud-native advantages to offer 10x cost efficiency and 100x operational gains over Kafka, with streamlined scalability and stateless brokers.

# Architecture

Apache Kafka was developed in 2011 for traditional data centers, employing a classic Shared Nothing architecture to solve horizontal scalability challenges. Over time, it has evolved into a Tiered Storage architecture to capitalize on the cost advantages of cloud storage. Today, AutoMQ introduces a Shared Storage architecture that fully leverages the benefits of cloud-native technologies. Compared to Apache Kafka, it provides ten times more cost efficiency and a hundredfold increase in operational efficiency.

## Shared Nothing Architecture

<img />

The Shared Nothing architecture, a hallmark of Apache Kafka, addresses scalability issues in distributed storage software within traditional data center environments via a compute-storage integration approach. Kafka utilizes an ISR-based replication mechanism to ensure data reliability and system availability.

As cloud computing has matured, the demand for business elasticity has grown, revealing the limitations of the classic Shared Nothing architecture. Scaling out Apache Kafka’s Broker nodes involves extensive data replication to complete reassignment, which typically takes several hours.

Additionally, Apache Kafka relies on three-replica replication. When deployed on the cloud, users are presented with two storage options:

* Selecting cloud storage like EBS for Brokers is costly. EBS's inherent three-replica mechanism, coupled with ISR replication, results in data being stored nine times, leading to significant waste of storage space.

* Opting for local disks as the storage medium for Brokers offers more manageable costs, but users face high operational expenses, which negates the benefits of cloud adoption.

## Tiered Storage Architecture

<img />

As cloud computing has matured and expanded in scale, the initial beneficiary has been object storage. Featuring low storage costs and a pay-as-you-go model, object storage has driven significant evolution in storage software towards the Tiered Storage architecture.

As the name suggests, this architecture introduces a secondary storage layer. Here, data is asynchronously transferred to the secondary storage after being initially stored in the primary storage. This approach can partially leverage the cost advantages of object storage while mitigating the elasticity issues of the Shared Nothing architecture.

However, the Tiered Storage architecture doesn't fundamentally address the challenges of Apache Kafka for several reasons:

* While space usage for primary storage can be decreased, the scope of this reduction varies depending on the scenario, necessitating a thorough capacity evaluation. Additionally, the high EBS costs related to ISR replication are not fully alleviated.

* The problem of slow scalability persists; scaling up or down requires reassigning data in primary storage, potentially reducing the time needed from several tens of hours to just a few hours.

In simple terms, the primary storage in the Tiered Storage architecture isn't fundamentally different from a Shared Nothing architecture; aside from space reductions, the partition storage layout on the file system and the ISR replication mechanism remain the same.

## Shared Storage Architecture

<img />

AutoMQ's Shared Storage architecture entirely replaces the Apache Kafka storage layer, innovatively offloading data to cloud storage, which makes Brokers stateless. AutoMQ's shared storage architecture comprises both WAL storage and object storage, with all data being stored in near real-time into object storage. In this architecture:

* Object storage serves as the primary data repository, offering elastic, pay-as-you-go, and low-cost data storage solutions.

* Due to the high latency and low IOPS characteristics of object storage, AutoMQ introduces a WAL storage layer to enhance data write efficiency and reduce IOPS consumption.

* WAL storage can utilize multiple storage services across different cloud platforms. Options include Regional EBS services with multi-AZ disaster recovery, file storage services like AWS EFS and FSx, and even object storage serving simultaneously as WAL and main data storage.

AutoMQ encapsulates these storage modules into a custom-developed streaming storage library—**S3Stream**—and replaces Apache Kafka's native Log storage with S3Stream, resulting in completely stateless Broker nodes. This facilitates features like second-level partition reassignment, automatic scaling, and continuous traffic rebalancing. For this purpose, AutoMQ integrates several Controller components within the kernel, such as Auto Scaling and Auto Balancing components, which are responsible for cluster scaling and traffic rebalancing, respectively.

It's important to note that AutoMQ Open Source supports only S3-compatible storage services as WAL storage options. This means AutoMQ Open Source is a cloud-native Kafka implementation that relies solely on object storage, leading to a simplified deployment architecture. In contrast, AutoMQ commercial editions offer different WAL options across various cloud providers to support a wider range of workloads.

## Reference

\[1]. Kafka ISR Replication Mechanism: [https://kafka.apache.org/documentation/#replication](https://kafka.apache.org/documentation/#replication)

\[2]. WAL Wiki：[https://en.wikipedia.org/wiki/Write-ahead\_logging](https://en.wikipedia.org/wiki/Write-ahead_logging)


# Overview
Source: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/overview

AutoMQ's S3Stream offloads Kafka's log storage to cloud storage, combining flexible WAL options and S3 for low-latency, high-throughput, cost-effective stream storage with near-infinite capacity.

# S3Stream: Shared Streaming Storage

S3Stream is a core stream storage component in AutoMQ, adhering to AutoMQ's concept of decoupling storage and computation. It offloads Apache Kafka's built-in ISR-based log storage layer to object storage and provides accelerated WAL storage capabilities.

S3Stream is a stream storage library rather than a distributed storage service. AutoMQ innovatively implements a set of core stream storage APIs on top of object storage, including position management, Append, Fetch, and Trim data. The following code snippet shows several core interfaces of these APIs.

```java theme={null}
public interface Stream {
    /**
     * Get stream id
     */
    long streamId();

    /**
     * Get stream start offset.
     */
    long startOffset();

    /**
     * Get stream next append record offset.
     */
    long nextOffset();

    /**
     * Append RecordBatch to stream.
     */
    CompletableFuture<AppendResult> append(RecordBatch recordBatch);

    /**
     * Fetch RecordBatch list from a stream. 
     */
    CompletableFuture<FetchResult> fetch(long startOffset, long endOffset, int maxBytesHint);

    /**
     * Trim stream.
     */
    CompletableFuture<Void> trim(long newStartOffset);
}

```

## Core Characteristics of Stream Storage

All data on the internet is generated in a streaming manner, then stored and computed in streams to extract the business value of real-time data. This also implies that stream data imposes at least the following requirements on storage:

* Low Latency: The greatest value of streaming data lies in its freshness. For example, businesses related to ad recommendations have very high real-time requirements. The faster data is stored and computed, the more value it can provide.

* High Throughput: Since all data is generated in a streaming manner, stream storage must support extremely high throughput. Many businesses require at least GiB/s bandwidth.

* Low Cost: Massive amounts of streaming data mean high storage costs. Additionally, many businesses need data playback and re-computation capabilities, making daily storage of stream data a business norm.

Object storage offers significant advantages in terms of scalability and cost for big data, particularly within data lake ecosystems. However, inherent characteristics of object storage, such as 100ms-level latency per API call and billing by API usage, limit its application in low-latency and high-IOPS storage scenarios, which are critical requirements for stream storage.

To overcome this limitation and make object storage suitable for stream storage scenarios, AutoMQ introduces a WAL (Write-Ahead Log) storage module within S3Stream. This module leverages centralized storage to mix data from all partitions of a node into a single WAL file or object, achieving high write efficiency and low IOPS consumption.

AutoMQ innovatively combines WAL storage and object storage, addressing the deficiencies of object storage in stream storage scenarios by providing low latency, high throughput, low cost, and nearly unlimited capacity for stream storage.

## S3Stream Architecture

<img />

In the core architecture of S3Stream, data is first durably written to the WAL and then almost in real-time uploaded to S3 storage. Additionally, to efficiently support both Tailing Read and Catch-up Read models, S3Stream incorporates a data caching component to accelerate read operations.

* WAL Storage: Select different storage media based on your business's latency and durability requirements; options include block storage, object storage, or file storage.

* S3 Storage: The largest object storage service offered by cloud providers is selected to provide high-throughput and cost-effective primary data storage services.

* Data Caching: Both hot data and prefetched cold data are stored in the cache to accelerate reads. At the same time, an efficient eviction mechanism based on consumer focus ensures optimal memory utilization.

Notably, for AutoMQ Open Source, only S3-compatible storage services are supported as a WAL storage option. This indicates that AutoMQ Open Source is a cloud-native Kafka implementation dependent solely on object storage, offering a highly streamlined deployment architecture. For AutoMQ commercial editions, various WAL storage options are provided across different cloud providers to support a broader range of workloads.


# S3 Storage
Source: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/s3-storage

AutoMQ leverages S3 storage for cost-efficient, scalable data storage, ensuring rapid fault recovery with Kafka compatibility. Achieve seamless, real-time traffic balancing and effortless scaling.

In the S3Stream repository, S3 storage serves as a core component. The Write-Ahead Log (WAL) is used solely for write acceleration and fault recovery, while S3 is the actual storage location for data. With vast amounts of data migrating to the cloud, object storage has become the de facto storage engine for big data and data lake ecosystems. Today, we see numerous data-intensive software applications transitioning from file APIs to object APIs, and streaming data integration into data lakes, represented by Kafka, is a prevailing trend.

S3Stream leverages the Object API to provide efficient streaming data ingestion and retrieval. By employing a storage-compute separation architecture, it integrates Apache Kafka's storage layer with object storage. This approach fully capitalizes on the technical and cost advantages brought by shared storage:

* Based on AWS public pricing, Amazon S3 Standard storage is priced at approximately $0.023/GiB/month — roughly one-ninth the price of Amazon EBS gp3 ($0.20/GiB/month) on a per-GiB single-replica basis. Additionally, object storage inherently offers multi-availability zone availability and durability, eliminating the need for additional data replication. Compared to the traditional 3-replica architecture based on EBS gp3 (approximately \$0.60/GiB/month total), this can reduce storage costs by roughly 26×.

* The shared storage architecture, in contrast to the Shared-Nothing architecture, truly embodies storage-compute separation, where data is not bound to computing nodes. Consequently, AutoMQ can perform partition reassignment without data duplication, achieving truly lossless partition reassignment in seconds. This capability is fundamental to supporting AutoMQ's real-time traffic self-balancing and second-level node horizontal scaling.

## S3 Storage Architecture

<img />

All data in AutoMQ is stored in object storage through S3Stream, defining two types of Objects within object storage:

* Stream Set Object: When uploading WAL data, the majority of smaller Streams are consolidated and uploaded as a single Stream Set Object.

* Stream Object: The Stream Object contains data from a single Stream, facilitating precise data deletion for Streams with different lifecycles.

Data from a single Kafka partition is mapped onto multiple Streams, with two core components:

* **Metadata Stream**: This stream stores data indices, Kafka LeaderEpoch snapshots, Producer snapshots, and other metadata.

* **Data Stream**: This stream holds the complete Kafka data within the partition.

Metadata information of objects on S3 is stored in KRaft. To reduce the scale of metadata in scenarios involving a large number of partitions, the S3Stream internally provides two compaction mechanisms to merge small objects and thereby reduce the metadata scale.

### StreamSet Object Compact

**StreamSet Object Compaction** is performed periodically in the background on the Broker at 20-minute intervals. Similar to RocksDB SST, StreamSet Object Compaction selects an appropriate list of StreamSet Objects on the current Broker based on certain policies and merges them using a merge sort method:

* Streams that exceed 16MiB after merging will be split into individual Stream Objects and uploaded separately;

* The remaining Streams are merged and written into a new StreamSet Object using merge sort.

Merge sorting for StreamSet Object compaction can merge up to 15TiB of StreamSet Objects under the constraints of 500MiB memory usage and a 16MiB read/write range size.

Through StreamSet Object Compact, fragmented small stream data segments are merged into larger data segments, significantly reducing API calls and enhancing read efficiency for cold data catch-up read scenarios.

### Stream Object Compact

The core purpose of Stream Object Compact is to minimize the total amount of metadata required to maintain object mapping in the cluster and to improve the aggregation of Stream Object data, thereby reducing API call costs for cold data catch-up read.

Stream Objects involved in Compact are typically already 16MB, meeting the minimum part limit of object storage. Stream Object Compact uses the MultiPartCopy API of object storage to directly perform range copy uploads, avoiding the waste of network bandwidth from reading and then writing back to object storage.

## Multi-Bucket Architecture

As one of the most critical storage services offered by various cloud providers, object storage provides twelve nines of data durability and up to four nines of availability. However, software failures can never be completely eliminated, and object storage can still experience significant software failures, rendering AutoMQ services unavailable.

On the other hand, multi-cloud, multi-region, and even hybrid cloud architectures are gradually emerging to meet enterprises' more flexible IT governance needs.

In view of this, AutoMQ commercial editions innovatively adopt a multi-bucket architecture to further enhance system availability and meet enterprises' more flexible IT governance needs.

<img />

AutoMQ commercial editions support configuring one or multiple buckets for data storage, and S3Stream supports four different write strategies to meet various business scenarios.

### **Round Robin**

Multiple Buckets are treated equally, and data is written to them in a round-robin manner. This method is generally used to bypass the bandwidth limitations imposed by Cloud providers on a single Bucket or account. For instance, if a single Bucket supports only 5 GiB/s of bandwidth, combining two Buckets can achieve 10 GiB/s of bandwidth, supporting ultra-high traffic business scenarios.

### **Failover**

Object storage can still experience failures, and software-level failures can sometimes be more severe than zone-level failures. For business scenarios with extremely high availability requirements, data can be written to two Buckets using a failover approach. The Bucket configuration in a failover scenario might be:

* One as the primary Bucket, located in the same region as the business, where data is preferentially written.

* Another as a standby Bucket, created in a different region or even on a different Cloud. Connectivity between the primary and standby regions is established through dedicated lines or the public internet. When the primary region's object storage is unavailable, new data is submitted to the standby Bucket. Although the standby link incurs higher network costs, these costs are relatively controllable since they only occur when the primary Bucket is unavailable.

This write strategy significantly enhances AutoMQ's disaster recovery scenarios, enabling AutoMQ to build multi-region disaster recovery, multi-cloud disaster recovery, and even hybrid cloud disaster recovery architectures at a low cost.

### **Replication**

A failover-based multi-bucket architecture only redirects write traffic during failures, while read operations must wait for delayed reads. If your business cannot tolerate read delays during failures and requires an active-active read/write architecture, you can configure the write strategy to be in a replication mode, where data is synchronously written to multiple buckets.

This approach is very costly and is only suitable for mission-critical applications.

### Dynamic Sharding

Data is written to multiple buckets in a dynamic, adjustable ratio, with round-robin being a specific scenario of this strategy. Dynamic sharding write methods are often suitable for multi-cloud and hybrid cloud architectures. In this architecture, you configure buckets across different cloud services or combine Public Cloud and Private Cloud object storage. By dynamically adjusting the traffic ratio, the architecture can always maintain the flexibility for multi-cloud migration or even cloud exit.


# WAL Storage
Source: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/wal-storage

AutoMQ’s WAL storage offers scalable, low-latency data persistence with Kafka compatibility. Supports EBS, S3 WAL, and innovative multi-WAL solutions for cost efficiency.

In the stream storage repository S3Stream, WAL is one of the core components. WAL storage primarily addresses issues such as high latency when using object storage as the main data store and low efficiency in writing to multiple partitions. It has two primary responsibilities:

* Providing low-latency, high-performance data persistence writes. Once the data is successfully written to WAL, confirmation is returned to the client.

* During a Broker node failure that requires failover, it recovers data from WAL that has not been timely uploaded to S3.

## WAL Storage Implementation

**WAL** is implemented as a fixed-size, cyclically written storage space. Various storage media can be selected, but the implementation mainly considers the following points:

* Centralized writes, unlike Apache Kafka®, AutoMQ does not need to write separate log files for each partition. By mixing the data from all partitions into the WAL, it supports efficient writing in scenarios with a large number of partitions.

* Sequential writes and group commits, data is written sequentially into the WAL, combined with the group commit mechanism, requiring only a small amount of IOPS to achieve high-throughput writes.

* **Direct IO Writing:** Data is directly written through to the storage medium to fully utilize the durability of cloud storage. Write operations are considered successful only upon confirmation, unaffected by the operating system's Page Cache dirty page collection.

* **Support for Raw Device Writing\[1]:** AutoMQ needs to write only one file. If EBS is used as the storage medium, it can be used directly as a raw device for writing, without the need to mount a file system, thereby avoiding additional overhead brought by the file system. This ensures optimal performance and latency.

Through the aforementioned technology, WAL storage significantly reduces the dependency of stream storage systems on IOPS, addressing the issues of low write efficiency and high API call costs associated with object storage, thus making S3Stream widely applicable. AutoMQ supports both latency-sensitive and high-throughput systems.

## WAL Storage Medium Selection

Public Cloud providers generally offer three types of storage services, namely:

* **Block storage**, such as AWS EBS, Azure Zone-redundant Disk, GCP Regional Persistent Disk, and Alibaba Cloud Regional ESSD. Among these services, EBS adopts a single AZ multi-copy architecture, while the other block storages use a multi-AZ multi-copy architecture, collectively referred to as Regional EBS.

* **Object storage** is the most standard storage service provided by cloud providers, with virtually all cloud providers supporting the standard S3 protocol.

* File storage, primarily NFS protocol-based file storage services, is also widely used in big data and other business scenarios, such as AWS EFS and AWS FSx series.

<img />

The aforementioned storage mediums can be used as WAL storage mediums, with specific differences as shown in the table below.

| -<br />                    | EBS & Regional EBS WAL<br />                                                                                                    | S3 WAL<br />                                                                          | NFS WAL<br />                                                                   |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Multi AZ<br />             | <ul><li>EBS has a single AZ multi-replica architecture</li><li>Regional EBS has a multi AZ multi-replica architecture</li></ul> | S3 offers both single AZ and multi AZ product options<br />                           | NFS offers both single AZ and multi AZ product options<br />                    |
| Durability<br />           | Between 5 nines and 9 nines<br />                                                                                               | Around 11 nines<br />                                                                 | Around 11 nines<br />                                                           |
| Latency<br />              | Sub-millisecond<br />                                                                                                           | Hundred-millisecond<br />                                                             | Millisecond-level<br />                                                         |
| Cost<br />                 | Low<br />                                                                                                                       | Low<br />                                                                             | Moderate<br />                                                                  |
| Applicable Scenarios<br /> | Regional EBS is suitable for all Kafka use cases<br />                                                                          | Suitable for most latency-insensitive scenarios, such as logging and monitoring<br /> | Low-latency solutions on AWS, such as core transaction matching scenarios<br /> |

In scenarios where AutoMQ chooses S3 WAL, S3 simultaneously offers both WAL storage and primary storage services. The WAL storage is optimized for writes, requiring fewer S3 API calls. Especially in environments with numerous partitions, S3 WAL provides efficient data ingestion capabilities, while the primary storage component supports efficient data consumption through various compaction mechanisms.

Notably, for AutoMQ Open Source, only S3-compatible storage services are supported as WAL storage options. This indicates that AutoMQ Open Source is a cloud-native Kafka implementation that relies solely on object storage, with a minimal deployment architecture. For AutoMQ commercial editions, different WAL options are available on various cloud providers to support more workloads.

## References

\[1]. How AutoMQ Achieves High-Performance WAL Based on Raw Devices: [https://www.automq.com/blog/principle-analysis-how-automq-implements-high-performance-wal-based-on-raw-devices](https://www.automq.com/blog/principle-analysis-how-automq-implements-high-performance-wal-based-on-raw-devices)

\[2]. Azure Regional EBS: [https://learn.microsoft.com/en-us/azure/virtual-machines/disks-redundancy#zone-redundant-storage-for-managed-disks](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-redundancy#zone-redundant-storage-for-managed-disks)

\[3]. GCP Regional EBS: [https://cloud.google.com/compute/docs/disks/regional-persistent-disk](https://cloud.google.com/compute/docs/disks/regional-persistent-disk)

\[4]. Alibaba Cloud Regional EBS: [https://developer.aliyun.com/special/live/regionalessd\_bdrc](https://developer.aliyun.com/special/live/regionalessd_bdrc)

\[5]. AWS EFS: [https://aws.amazon.com/efs/](https://aws.amazon.com/efs/)

\[6]. AWS FSx: [https://aws.amazon.com/fsx/netapp-ontap/](https://aws.amazon.com/fsx/netapp-ontap/)


# 5x Catch-up Read Efficiency
Source: https://docs.automq.com/automq/architecture/technical-advantage/5x-catch-up-read-efficiency

AutoMQ offers 5x catch-up read efficiency over Apache Kafka with cold and hot data isolation, cloud-native scalability, Kafka compatibility, and better cost efficiency.

Apache Kafka users have long been plagued by KAFKA-7504\[1], a performance issue that remains unresolved to this day. When a cold read occurs in an Apache Kafka cluster, if it cannot be quickly resolved, more and more tail read operations will also be slowed down, gradually exerting a significant impact on the traffic write operations.

## Apache Kafka Cold Read Issue

<img />

Apache Kafka's read and write paths introduce two key technologies: Page Cache\[2] and zero-copy SendFile\[3] system call.

* The Page Cache greatly simplifies Kafka's memory management burden, which is entirely handled by the kernel. However, there is an issue where hot and cold data cannot be separated. If a service continuously performs cold reads, it will compete for memory resources with hot data, leading to a continuous decline in tail read capabilities.

* SendFile is a critical technology for Kafka's zero-copy feature, but this call occurs in Kafka's network thread pool. If SendFile needs to copy data from the disk (in a cold read scenario), it will partially block this thread pool. Since this thread pool handles Kafka requests, including write requests, the blocking behavior of SendFile will significantly impact Kafka's write operations.

## AutoMQ Cold and Hot Data Isolation Architecture

<img />

From the very first day of its design, AutoMQ fully considered the cold and hot data isolation issue faced by asynchronous message middleware. In AutoMQ's architecture, there are three key data paths:

* Write Path: Data is persistently written into WAL storage in the form of Direct IO, without relying on Page Cache. The data is written through to WAL before returning success to the client, completely separating it from the data read path.

* Tail Read Path

* Cold Read Path

## Cold Read Performance Evaluation of AutoMQ

The following table results are derived from the actual benchmark of AutoMQ vs. Kafka ([Benchmark: AutoMQ vs. Apache Kafka▸](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)), demonstrating that under the same load and machine types, AutoMQ maintains the same level of cold read performance as Kafka without affecting write throughput and latency.

| Comparison Item<br /> | Send Latency During Cold Read<br /> | Impact on Send Traffic During Cold Read<br />       | Cold Read Efficiency<br />(Time to Read 4TiB Data)<br /> |
| --------------------- | ----------------------------------- | --------------------------------------------------- | -------------------------------------------------------- |
| AutoMQ<br />          | Less than **3ms**<br />             | Read-write isolation, maintains **800 MiB/s**<br /> | 42 minutes<br />                                         |
| Apache Kafka<br />    | Approximately **800ms**<br />       | Mutual impact, drops to **150 MiB/s**<br />         | 215 minutes<br />                                        |

From the results, it can be seen that AutoMQ has no impact on send latency during cold reads, whereas Apache Kafka deteriorates to second-level latency, and write traffic continuously drops. In terms of cold read efficiency, reading 4TiB of data, AutoMQ offers a 5-fold improvement in efficiency.

## Advantages of Hot and Cold Data Isolation

After resolving performance issues related to hot and cold data isolation, AutoMQ's architecture becomes more tenant-friendly. Although AutoMQ benefits from second-level scalability, it is recommended to configure separate clusters for each business to avoid mixing workloads, thereby reducing costs. However, for businesses with very low traffic, mixing workloads within a single cluster can effectively reduce the number of clusters. AutoMQ's hot and cold data isolation feature completely addresses stability issues in multi-tenant scenarios.

Additionally, AutoMQ fully leverages the high throughput characteristics of object storage, achieving a 5-fold increase in cold read efficiency. This allows for quick data replay to meet business needs promptly.

## Reference

\[1] Kafka Cold Read Performance Issue: [https://issues.apache.org/jira/browse/KAFKA-7504](https://issues.apache.org/jira/browse/KAFKA-7504)

\[2] Linux Page Cache: [https://en.wikipedia.org/wiki/Page\_cache](https://en.wikipedia.org/wiki/Page_cache)

\[3] Linux SendFile: [https://man7.org/linux/man-pages/man2/sendfile.2.html](https://man7.org/linux/man-pages/man2/sendfile.2.html)


# Continuous Self-Balancing
Source: https://docs.automq.com/automq/architecture/technical-advantage/continuous-self-balancing

AutoMQ offers cloud-native scalability and seamless Kafka compatibility with second-level partition reassignment, ensuring efficient, automatic data self-balancing and optimal traffic distribution.

In an online Apache Kafka® cluster, fluctuations in traffic, Topic creation and deletion, and Broker failures and restarts are constantly occurring. These changes can lead to an uneven distribution of traffic across cluster nodes, resulting in resource wastage and impacting business stability. It is essential to proactively reassign different partitions of a Topic across nodes to balance traffic and data.

## Challenges Faced by Open Source Solutions

Apache Kafka has always faced significant challenges in achieving self-balancing of data. The community has two solutions:

* The official Apache Kafka partition reassignment tool requires operations personnel to devise specific reassignment plans. For Kafka clusters with hundreds or thousands of nodes, it is nearly impossible to manually monitor the cluster state and create a comprehensive partition reassignment plan.

* The community also offers third-party external plugins such as Cruise Control\[1] to assist in generating reassignment plans. However, due to the extensive variables involved in Apache Kafka's self-balancing process (replica distribution, leader traffic distribution, node resource utilization, etc.) and the resource contention and hours-to-days-long duration caused by data synchronization, existing solutions are complex and have low decision timeliness. Implementing data self-balancing strategies still relies on operations personnel's review and continuous monitoring, failing to truly address the challenges posed by Apache Kafka data self-balancing.

## AutoMQ's Architectural Advantages

Thanks to AutoMQ's deep integration with cloud-native capabilities, we have reimplemented Apache Kafka's underlying storage entirely based on cloud object storage, upgrading from a Shared Nothing architecture to a Shared Storage architecture. This enables second-level partition reassignment capabilities, greatly simplifying the factors involved in reassignment planning:

* There is no need to consider node disk resources.

* No need to consider the leader distribution and replica distribution of partitions.

* The reassignment of partitions does not involve data synchronization and copying.

Therefore, we have the opportunity to implement a built-in, lightweight automatic data balancing component within AutoMQ, which continuously monitors cluster status and automatically performs partition reassignment.

## AutoMQ Data Self-Balancing Implementation

<img />

The figure above is the architecture diagram of AutoMQ's built-in Auto Balancing component. This architecture collects cluster metrics, automatically generates partition reassignment plans, and continuously balances the cluster's traffic.

AutoMQ implements Apache Kafka's MetricsReporter interface, monitoring all built-in metrics information of Apache Kafka. It periodically samples metrics of interest (such as network ingress and egress traffic, CPU utilization, etc.) and pre-aggregates these metrics on the Broker side. The aggregated metrics are serialized into a Kafka message and sent to a designated internal Topic.

The AutoMQ Controller maintains an in-memory cluster state model to describe the current state of the cluster, including Broker status, Broker resource capacity, and the traffic information of Topic-Partition managed by each Broker. By listening to KRaft Log event information, the cluster state model can promptly sense changes in the status of Broker and Topic-Partition and update the model accordingly to remain consistent with the actual cluster state.

Meanwhile, the metrics collector in the AutoMQ Controller consumes messages from the internal Topic in real-time, deserializes the messages into specific metrics, and updates them into the cluster state model, thereby constructing all the prerequisite information needed for data self-balancing.

AutoMQ Controller's scheduler periodically captures snapshots of the cluster state model and identifies overutilized or underutilized brokers based on predefined "goals." It then attempts to reassign or exchange partitions to achieve traffic self-balancing.

## AutoMQ Data Self-Balancing Example

<img />

Consider a three-node AutoMQ cluster. Initially, each of the three nodes handles approximately 40MiB/s of write traffic. In the second phase, the traffic for Broker-0 is increased to about 80MiB/s, Broker-1 to about 120MiB/s, and Broker-2 remains at 40MiB/s. As shown in the figure above, automatic load balancing is triggered in the second phase, and the traffic across the three brokers gradually converges to a balanced range.

<Note>
  NOTE: In this scenario, to facilitate observation, the partition reassignment cooldown time is manually extended. With the default configuration, the traffic balancing time is approximately 1 minute.
</Note>

## Quick Experience

Refer to [Example: Continuous Data Self-Balancing▸](/automq/getting-started/explore-automq/example-continuous-data-self-balancing) to experience the continuous data self-balancing capabilities of AutoMQ.

## References

\[1]. LinkedIn's open-source Cruise Control tool: [https://github.com/linkedin/cruise-control](https://github.com/linkedin/cruise-control)


# Native Compatible with Apache Kafka
Source: https://docs.automq.com/automq/architecture/technical-advantage/native-compatible-with-apache-kafka

AutoMQ ensures 100% compatibility with Apache Kafka's protocol and semantics, providing cloud-native scalability, cost-efficient storage solutions, and seamless integration.

Currently, many industry products are compatible with the Apache Kafka® protocol, such as Redpanda and Kafka on Pulsar. AutoMQ believes that starting from scratch to adapt to the Kafka protocol can hardly achieve complete compatibility in every detail, and would also involve a significant amount of repetitive and unnecessary intellectual effort.

The Kafka protocol has 113 ErrorCodes and 68 APIs, with the Fetch API alone having 15 versions. Achieving 100% compatibility with the Kafka protocol and semantics is extremely challenging. Furthermore, as Apache Kafka® evolves, maintaining compatibility with the Kafka protocol over time is also a major challenge.

Compatibility with the Kafka protocol and semantics is a critical consideration for users when choosing Kafka products. Therefore, the architectural design of AutoMQ mandates 100% compatibility with the Apache Kafka® protocol and semantics, and ensures continuous alignment with Apache Kafka®.

## Current State of the Apache Kafka® Protocol

Apache Kafka® has been developed for over 10 years, with contributions from over 1000+ contributors, leading to 1059 KIPs \[1]. The entire codebase contains hundreds of thousands of lines of code, incorporating numerous features, optimizations, and fixes. Building a Kafka-compatible API protocol and semantics from scratch would not only require extensive development effort but is also prone to errors. The Apache Kafka® architecture is composed of a compute layer and a storage layer:

* Compute Layer: Constitutes 98% of the total codebase, carrying Kafka's API protocol and features. Additionally, the compute layer has numerous system optimizations tailored for stream storage, such as end-to-end batch design and zero-copy mechanisms, enabling 1GiB/s throughput with just 2 CPU cores.

* Storage Layer: Makes up 2% of the total codebase and is responsible for the high-durability storage of messages. As a stream processing pipeline, Apache Kafka® stores vast amounts of data over time. The majority of the cost for an Apache Kafka® cluster stems from data storage expenses and the costs associated with machines deployed for compute-storage integration.

## AutoMQ Natively Supports the Kafka Protocol

AutoMQ aims to upgrade Apache Kafka® to a shared storage architecture by adopting a compute-storage separation architecture. The optimal solution is to replace Kafka’s storage layer while retaining its native compute layer. The advantages of this approach include:

* It allows for the reuse of 98% of Apache Kafka's compute layer code, ensuring API protocol & semantic compatibility and feature alignment.

* It enables the replacement of the storage layer with cloud-native storage services, leveraging the technical and cost benefits of shared storage and cloud-native technologies.

<img />

Although Apache Kafka exposes a stream abstraction modeled by Partitions at the business logic layer, internally, Kafka’s log recovery, transaction indexing, timestamp indexing, and reading operations are all based on Log Segments. This means that Log Segment is the smallest operational unit of Kafka storage.

Therefore, Segment is the optimal cut point to implement the compute-storage separation architecture in AutoMQ. By implementing shared Segment semantics based on S3Stream, we can reuse the logic of upper-layer LocalLog, LogCleaner, and Partition, thereby maximizing the reuse of Apache Kafka code.

In addition to achieving native support for Kafka protocol in terms of architectural design, AutoMQ has also passed Apache Kafka's 500+ system test cases (KRaft mode). This test suite covers Kafka functionalities (message sending/receiving, consumer management, Topic Compaction, etc.), client compatibility (>= 0.9), operations (partition reassignment, rolling restart, etc.), Stream, and Connector testing. This ensures 100% protocol and semantic compatibility for AutoMQ from a practical operational standpoint.

## References

\[1]. Apache Kafka KIP List: [https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals](https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals)


# Overview
Source: https://docs.automq.com/automq/architecture/technical-advantage/overview

AutoMQ offers cloud-native scalability and efficiency with 100% Kafka API compatibility, enabling second-level scaling, seamless partition reassignment, and continuous self-balancing.

AutoMQ adopts a storage-compute separation approach, evolving Apache Kafka® from a Shared Nothing architecture to a Shared Storage architecture. This shift enables full utilization of cloud-native technology and cost advantages. Technically, the shared storage architecture offers inherent benefits in addressing operational challenges like partition reassignment and automatic scaling. AutoMQ leverages these benefits through product capabilities, as shown in the table below.

| Advantages                                  | AutoMQ                                                                                   | Apache Kafka®                                                                                                   |
| ------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Native support for Kafka protocol           | ✅ Natively supported, reuses protocol layer code                                         | ✅ Natively supported                                                                                            |
| Second-level partition reassignment         | ✅ Shared storage, no need to copy data during reassignment                               | ❌ Reassigns local data, typically takes hours per partition                                                     |
| Second-level smooth scaling                 | ✅ Achieves second-level scaling by bulk reassigning partitions                           | ❌ Due to data copying during partition reassignment, cannot support fast scaling, typically takes tens of hours |
| Continuous self-balancing                   | ✅ Built-in Auto Balancing component, dynamically adjusts traffic to eliminate hotspots   | ❌ Prone to local hotspots, affecting stability                                                                  |
| Stateless Broker                            | ✅ Data offloaded to shared storage, Brokers are stateless and can use Spot instances     | ❌ Local disk state, requires extensive data transfer for decommissioning                                        |
| Cold-hot isolation, 5x cold read efficiency | ✅ Out-of-the-box cold-hot isolation capability, supports multi-tenant business scenarios | ❌ Cold and hot data affect each other, exacerbating system degradation, cannot support multi-tenant scenarios   |


# Partition Reassignment in Seconds
Source: https://docs.automq.com/automq/architecture/technical-advantage/partition-reassignment-in-seconds

AutoMQ enables second-level partition reassignments with its shared storage architecture, ensuring real-time scalability, high cost efficiency, and full Kafka compatibility for seamless cloud-native operations.

Partitions are the core resource model in Apache Kafka®, serving as the primary conduit for client read and write traffic. As the number of partitions in a cluster increases, uneven partition distribution and hotspot partitions can occur. Consequently, partition reassignments are a common high-frequency operation in the day-to-day maintenance of Apache Kafka®.

In the Shared Nothing architecture of Apache Kafka®, partition reassignment involves replicating large amounts of data. For example, a Kafka partition with a 100MiB/s traffic rate generates approximately 8.2TB of data in one day. If this partition needs to be reassigned to another broker, the entire dataset must be replicated. Even with nodes that have 1 GBps bandwidth, this reassignment would take hours to complete, rendering Apache Kafka® clusters nearly incapable of real-time elasticity.

Thanks to AutoMQ's shared storage architecture, partition reassignments require syncing only a small amount of data, reducing reassignment time to seconds.

## How AutoMQ Achieves Second-Level Partition Reassignments

<img />

In AutoMQ's shared storage architecture, almost all data is stored in object storage, with minimal data temporarily residing in WAL storage. During a partition reassignment, only the data that has not yet been uploaded to object storage and is temporarily held in WAL needs to be forcibly uploaded. This process allows the partition to be safely transferred to another node, typically within about 1.5 seconds.

The specific reassignment process is illustrated in the figure above, using the example of reassigning partition P1 from Broker-0 to Broker-1:

1. When the KRaft Controller receives a partition reassignment command, it constructs the corresponding PartitionChangeRecord and commits it to the KRaft Log layer. This action removes Broker-0 from the Leader Replica list and adds Broker-1 to the Follower Replica list. Broker-0, upon syncing the KRaft Log and detecting the P1 partition change, initiates the partition shutdown process.

2. During the partition shutdown, if P1 contains data that has not yet been uploaded to object storage, a forced upload is triggered. In a stable running cluster, this data typically amounts to a few hundred megabytes. Given the burst network bandwidth capabilities provided by current cloud providers, this process usually takes only seconds. Once P1's data upload is complete, the partition can be safely closed and deleted from Broker-0.

3. After the Broker completes its shutdown, it will proactively trigger a leader election. At this point, Broker-1, being the sole Replica, is promoted to the Leader of P1, and the partition recovery process begins.

4. During partition recovery, the metadata corresponding to P1 is fetched from the object storage to restore the relevant Checkpoint for P1. Depending on P1's shutdown state (whether it was a Cleaned Shutdown), the corresponding data recovery is performed.

5. At this point, the partition reassignment is complete.

## Significance of Second-Level Partition Reassignment

In a production environment, a Kafka cluster typically serves multiple applications. Fluctuations in application traffic and partition distribution can cause cluster capacity issues or machine hotspots. Kafka operations personnel need to expand the cluster and reassign hotspot partitions to idle nodes to ensure the availability of cluster services.

The time taken for partition reassignment determines the efficiency of emergency response and maintenance:

* The shorter the partition reassignment time, the shorter the duration from cluster expansion to meeting capacity demands, and the shorter the service disruption time.

* Faster partition reassignment results in shorter observation times for operations personnel, enabling quicker operational feedback and decision-making for subsequent actions.

In the architecture of AutoMQ, partition reassignment within seconds is the foundation for many automation capabilities, including automatic scaling, descaling, and continuous self-balancing of traffic.

## Quick Experience

Refer to [Example: Partition Reassignment in Seconds▸](/automq/getting-started/explore-automq/example-partition-reassignment-in-seconds) to experience the partition reassignment capabilities of AutoMQ in seconds.


# Scale-out/in in Seconds
Source: https://docs.automq.com/automq/architecture/technical-advantage/scale-outin-in-seconds

AutoMQ offers rapid, cloud-native scalability in seconds with seamless Kafka compatibility, reducing resource waste and ensuring cost efficiency with automatic scaling based on business traffic.

In the cloud-native era, leveraging the elasticity provided by cloud providers, we can efficiently scale a cluster at the node level using AWS Auto Scaling Groups \[1] or Alibaba Cloud's ESS Auto Scaling Groups \[2]. However, due to the need for traffic reassignment, Apache Kafka® clusters often cannot directly utilize the elasticity provided by cloud providers and require manual intervention from operations personnel to move the traffic, which typically takes hours. For online clusters with frequent traffic fluctuations, this makes it almost impossible to scale on demand. To ensure cluster stability, operations personnel have to pre-deploy according to maximum capacity to avoid risks associated with untimely scaling during traffic peaks, leading to significant resource wastage.

## How AutoMQ Achieves Smooth Scaling in Seconds

<img />

AutoMQ's capability for rapid scaling relies on an atomic capability, namely partition reassignment in seconds (see [Partition Reassignment in Seconds▸](/automq/architecture/technical-advantage/partition-reassignment-in-seconds)).

When using Auto Scaling Groups (ASG) or Kubernetes' Horizontal Pod Autoscaler (HPA)\[3] to add nodes, you only need to reassign some partitions in the cluster to the new nodes to achieve traffic self-balancing (see [Continuous Self-Balancing▸](/automq/architecture/technical-advantage/continuous-self-balancing)), which can often be completed within ten seconds.

### Triggering Scaling

Taking AWS Auto Scaling Groups (ASG) as an example, by configuring traffic threshold monitoring, when the cluster traffic reaches the scaling threshold, new broker nodes are automatically launched. At this point, the Controller detects the traffic imbalance and automatically moves partitions to the newly created brokers, completing the traffic redistribution.

The figure below shows the change in the number of brokers in an AutoMQ Kafka cluster as the traffic increases. It can be seen that brokers are dynamically created and added to the cluster to balance the load as traffic increases linearly.

<img />

The figure below shows the traffic changes across broker nodes during traffic increases. It can be seen that the newly created brokers complete traffic rebalancing within ten seconds.

<img />

### Triggering Scale Down

Taking AWS Auto Scaling Groups as an example, when the cluster traffic reaches the scale-down threshold, the Broker node to be scaled down will undergo a graceful shutdown process. During this time, the partitions on the Broker will be reassigned in a round-robin manner to the remaining Brokers within seconds, completing the graceful shutdown and traffic transfer.

The figure below shows the change in the number of Brokers in an AutoMQ Kafka cluster as traffic decreases. You can observe that as traffic linearly decreases, Brokers are dynamically shut down to save resources.

<img />

The figure below illustrates the changes in traffic across Broker nodes during traffic decline. It is evident that the load on the shutting down Broker is transferred to the remaining Brokers (whenever a Broker is shut down, the traffic on the remaining Brokers significantly increases).

<img />

<Note>
  In the above example, to facilitate observation, the ASG's scale-up and scale-down cooldown times were artificially increased, and process startup and destruction delays were added.
</Note>

## Advantages of Automatic Scaling

AutoMQ's shared storage architecture inherently supports rapid automatic scaling, which is also the foundation for achieving Serverless. The automatic scaling capabilities of AutoMQ provide at least the following advantages:

* Cost advantages: There is no need to prepare resources based on peak demand. Resources automatically scale according to business traffic, effectively handling tidal and burst-type workloads, with payment based on usage and no wasted idle resources.

* Stability advantages: Seamless scaling without causing additional traffic pressure on the cluster. This allows for lossless scaling even under high watermarks. In contrast, scaling in Apache Kafka® is a high-risk operation that can only be performed under low watermarks.

* Multi-tenant advantages: Clusters with automatic scaling capabilities eliminate the need to mix multiple businesses to improve resource utilization. It is entirely possible to configure an independent cluster for each business. Each independent cluster can scale according to its own traffic model. This ensures cost advantages while preventing global impact if a particular business encounters issues.

## References

\[1]. AWS Auto Scaling Groups: [https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html)

\[2]. Alibaba Cloud ESS Elastic Scaling Group: [https://www.aliyun.com/product/ecs/ess](https://www.aliyun.com/product/ecs/ess)

\[3]. Kubernetes HPA Component for Scaling: [https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)


# Stateless Broker
Source: https://docs.automq.com/automq/architecture/technical-advantage/stateless-broker

AutoMQ leverages cloud-native scalability by separating storage and compute, making Broker nodes stateless and reducing costs with Spot instances. Fully Kafka-compatible.

AutoMQ utilizes the separation of storage and compute to offload Kafka's storage layer to cloud storage via S3Stream, making Broker nodes stateless. Stateless Brokers have significant advantages in operations and scalability. Additionally, stateless AutoMQ can be deployed using Spot instances on the cloud, further reducing computing costs.

## Advantages of Statelessness

Storage software that adopts a storage-compute integrated architecture is generally stateful software, facing significant challenges in operations, scaling, and downsizing. AutoMQ transforms Apache Kafka® into stateless storage software, making operating AutoMQ as simple as managing a microservice application.

* Simplified Operations: For AutoMQ, daily operations become sufficiently simple. After a Broker node shuts down, its state is completely transferred, with clients entirely unaffected. Operations personnel have ample time to decide whether the shut-down Broker needs to be brought back online or decommissioned permanently. Cluster upgrades can also be completed at low cost and risk through rolling updates.

* Automatic Scaling: Stateless AutoMQ can scale up or down freely, similar to a microservice application or a Kubernetes Deployment, achieving true auto-scaling and saving significant costs.

* Use of Spot Instances: Cloud providers offer Spot instances at significantly lower prices than regular virtual machines in many regions. However, due to the nature of Spot instances being subject to termination at any time, only stateless applications can take advantage of them.


# AutoMQ vs. Apache Kafka Benchmarks and Cost
Source: https://docs.automq.com/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost

AutoMQ offers cloud-native scalability with complete Kafka API compatibility, achieving up to 14x cost efficiency compared to Apache Kafka, maintaining a 1GiB/s throughput.

AutoMQ is a cloud-native reimplementation of Apache Kafka's storage layer utilizing cloud storage. It not only maintains complete compatibility with Apache Kafka's API but also achieves a 17x cost efficiency compared to Apache Kafka under 1GiB/s traffic. The monthly TCO decreases from 226,671 USD for Apache Kafka to 12,899 USD. The detailed cost breakdown is shown in the table below:

| Comparison Item<br />                | AutoMQ<br />  | Apache Kafka<br /> |
| ------------------------------------ | ------------- | ------------------ |
| Compute                              | \$2,859       | \$18,447           |
| Storage                              | \$5,961       | \$69,984           |
| S3 API<br />                         | \$3,952<br /> | \$0<br />          |
| Cross-availability zone traffic cost | \$128         | \$138,240          |
| Total                                | \$12,899      | \$226,671          |

## Benchmark Preparation

Conduct performance testing on a Topic with 256 partitions at a traffic speed of 1 GiB/s for both production and consumption.

### AutoMQ Cluster

AutoMQ can be deployed on Kubernetes using the Bitnami Kafka Helm Chart. You can refer to [Install AutoMQ by Helm Chart](https://github.com/AutoMQ/automq/blob/main/chart/bitnami/README.md) to deploy a 3-availability zone AutoMQ cluster on AWS:

* A total of 6 nodes, including 3 servers and 3 brokers, are evenly distributed across 3 availability zones;

* The selected physical node type is m7g.4xlarge, featuring a specification of 16 cores and 64GB of RAM, a network baseline of 960 MiB/s, and priced at \$476.544 per month.

### Load Tester

Stress machines are deployed across 3 availability zones, with one m7g.4xlarge in each zone, simulating multi-availability zone business loads. The performance testing is conducted using the `automq-perf-test.sh` script provided by AutoMQ, configured as follows:

```bash theme={null}
KAFKA_HEAP_OPTS="-Xmx32g -Xms32g" nohup ./bin/automq-perf-test.sh --bootstrap-server $bootstrap_server  --record-size 1024 --topics 1 --partitions-per-topic 256 --topic-prefix perf --producers-per-topic 20 --groups-per-topic 1 --consumers-per-group 20 --send-rate 349525 --warmup-duration 2 --test-duration 30 --producer-configs batch.size=1048576 linger.ms=100 buffer.memory=67108864 max.request.size=33554432 client.id='automq_az=apse1-az1' --consumer-configs fetch.max.bytes=104857600 max.partition.fetch.bytes=104857600 client.id='automq_az=apse1-az1' --reset &> nohup.log &

KAFKA_HEAP_OPTS="-Xmx32g -Xms32g" nohup ./bin/automq-perf-test.sh --bootstrap-server $bootstrap_server ... client.id='automq_az=apse1-az2' --await-topic-ready false &> nohup.log &

KAFKA_HEAP_OPTS="-Xmx32g -Xms32g" nohup ./bin/automq-perf-test.sh --bootstrap-server $bootstrap_server ... client.id='automq_az=apse1-az3' --await-topic-ready false &> nohup.log &

```

With this configuration, the load tester simulates the following scenario:

* A total of 60 Producers and 60 Consumers are evenly distributed across 3 availability zones, marked with `client.id=automq_az=apse1-az1` to indicate the availability zone affiliation of the client.

* Send 1,048,575 Records per second, each of 1 KiB in size, totaling 1 GiB/s of data.

* Sending parameters are additionally configured with `batch.size=1048576` and `linger.ms=100` to achieve better batching performance, suitable for most high-throughput Kafka scenarios.

## Costs

In a scenario where data is retained for 3 days (`log.retention.hours=72`), using AWS us-east-1 region as an example, the total cost of ownership for AutoMQ is \$12,899 per month. The total cost includes expenses for computation, storage, S3 API, and cross-availability zone traffic.

* Computation: AutoMQ utilized 6 m7g.xlarge instances in this benchmarking scenario, totaling a computing resource cost of 6 \* \$476.544 = \$2,859 per month.

* Storage: Traffic \* 3 days \* S3 price per unit = 1 \* (60 \* 60 \* 24 \* 3) \* 0.023 = \$5,961 per month.

* S3 API: Under this traffic, the average Get is 1,278/s and Put is 202.662/s, resulting in a cost of \$3,952 per month.

<img />

* Cross-availability zone traffic: Although AutoMQ is designed for clients to send and receive messages only with Brokers within the same availability zone, there is still a small amount of RPC requests between Brokers to synchronize KRaft metadata and forward ZoneRouterProduceRequest. We collected cross-availability zone traffic on Broker nodes using `iftop -t -s 60 -L 100`. The average cross-availability zone Network In + Network Out for one node is 0.84 MiB/s. Therefore, the cross-availability zone traffic cost for 6 nodes is 0.84 / 1024 \* 6 \* (60 \* 60 \* 24 \* 30) \* 0.01 \~= \$127 per month.

The total cost of AutoMQ is calculated as follows: compute + storage + S3 API + cross Availability Zone (AZ) traffic \~= \$12,899 per month.

## Compared to Apache Kafka, This Represents a 14x Cost Reduction.

The same workload on Apache Kafka requires \$65,385 per month, which is 14 times that of AutoMQ. The total cost of ownership includes compute, storage, and Inter-Zone traffic.

* Storage: To store 253.125 TiB of data (calculated as 1 / 1024 \* (60 \* 60 \* 24 \* 3)), and considering three replicas, 759.375 TiB of space is needed. Taking into account buffer and uneven data distribution, and assuming a 50% effective disk utilization rate, 1,518.75 TiB of disk space is necessary. To save on disk storage costs, st1 storage medium is used, leading to a storage cost calculation of 1,518.75 \* 1024 \* 0.045 = \$69,984 per month.

* Compute: With a maximum EBS volume size of 16 TiB, at least 95 volumes are needed, corresponding to 95 Brokers. Following production practices, the minimum recommended instance type is r4.xlarge, costing \$18,447.1 per month.

* Cross Availability Zone traffic: Assuming a uniform distribution of partitions and traffic, 2/3 of production traffic is sent across AZs. Consumption can be optimized to avoid cross AZ traffic using Fetch From Follower. Fetch From Follower implies that replicas are distributed across 3 AZs, generating two additional sets of production traffic among Broker ISR. Overall, cross AZ traffic is calculated as: production traffic \* (2 / 3 + 2) \* 30d = 1024 \* (2 / 3 + 2) / 1024 \* (60 \* 60 \* 24 \* 30) \* 0.02 = \$138,240 per month.

The total cost of Apache Kafka is: compute + storage + cross AZ traffic = \$226,671 per month.

In addition to reducing static costs, the stateless architecture of AutoMQ delivers further cost advantages:

* AutoBalancing: AutoMQ includes a load balancing component within the Controller, which automatically balances the load based on traffic between nodes. Partition reassignment is completed in less than 2 seconds, eliminating the need for manual load balancing by operations personnel.

* No Overprovisioning: With stateless nodes and second-level partition reassignments, AutoMQ can scale up or down within minutes—most of the time is spent on resource preparation. After expansion, load balancing is completed in seconds. AutoMQ does not require pre-reserving resources for peak times or handling load balancing at the hour level.

## Performance:

The CPU utilization of each AutoMQ Broker is at 50%. There is ample idle CPU capacity to handle traffic spikes and more small packet requests.

<img />

AutoMQ uses 8MiB / 250ms batching parameters by default when writing to S3. The Producer uses ack=all, and a record is only acknowledged as successful to the client after it is persisted to S3. In this test, the latency performance is as follows:

| -<br />       | AVG<br />   | P50<br />   | P99<br />    | P999<br />   |
| ------------- | ----------- | ----------- | ------------ | ------------ |
| Produce<br /> | 472ms<br /> | 465ms<br /> | 823ms<br />  | 1049ms<br /> |
| E2E<br />     | 631ms<br /> | 602ms<br /> | 1189ms<br /> | 1549ms<br /> |

If lower latency is needed, you can adjust the `batchInterval` parameter in the configurations of `s3.wal.path` and `automq.zonerouter.channels` to reduce the wait time for batching writes to S3. However, achieving lower latency will increase the cost of S3 APIs.


# Broker and Controller Configuration
Source: https://docs.automq.com/automq/configuration/broker-and-controller-configuration

Optimize AutoMQ deployment with advanced broker and controller configurations, leveraging cloud-native scalability and Kafka compatibility to reduce costs.

This document aims to explain the configuration parameters involved in the deployment of AutoMQ. It includes configuration definitions, descriptions, setting ranges, and specifications to aid developers in making necessary custom adjustments in production environments.

AutoMQ implements a storage-compute separation based on object storage and is fully compatible with Apache Kafka. Therefore, Kafka's functional configurations (e.g., ACL, network, etc.) can be referred to from the [official configuration documentation](https://kafka.apache.org/documentation/#brokerconfigs). This document only lists the configuration parameters related to the new storage module added by AutoMQ.

## Public Configuration

### elasticstream.enable

| Item<br />                      | Description<br />                                                   |
| ------------------------------- | ------------------------------------------------------------------- |
| Configuration Description<br /> | Whether to enable AutoMQ, this parameter must be set to true.<br /> |
| Value Type<br />                | boolean<br />                                                       |
| Default Value<br />             | false<br />                                                         |
| Valid Input Range<br />         | N/A<br />                                                           |
| Importance Level<br />          | High, requires careful configuration<br />                          |

### automq.zonerouter.channels

By enabling cross-AZ (Availability Zone) request routing, you can reduce cross-AZ data transmission and decrease traffic costs. For more details, refer to [Overview▸](/automq/eliminate-inter-zone-traffics/overview).

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                 |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | Configuration for Inter-Zone channel. By configuring Inter-Zone routing components, the cost caused by cross-zone traffic can be significantly reduced. Currently, only object storage is supported. The format is: `0@s3://$bucket?region=$region[&batchInterval=250][&maxBytesInBatch=8388608]` |
| Value Type<br />          | string<br />                                                                                                                                                                                                                                                                                      |
| Default Value<br />       | null<br />                                                                                                                                                                                                                                                                                        |
| Valid Input Range<br />   | N/A<br />                                                                                                                                                                                                                                                                                         |
| Importance Level<br />    | High, requires careful configuration<br />                                                                                                                                                                                                                                                        |

## S3Stream Related Configuration

### s3.data.buckets

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The URI for data plane object storage. The format is: `0@s3://$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey][&checksumAlgorithm=$checksumAlgorithm]`.<br />For configurations from different vendors, see: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                                                                       |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                                                                         |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                                                                          |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                         |

### s3.ops.buckets

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The URI for control plane object storage. The format is: `1@s3://$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey][&checksumAlgorithm=$checksumAlgorithm]`.<br />For configurations from different vendors, see: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                                                                          |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                                                                            |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                                                                             |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                            |

### s3.wal.path

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                    |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The mount path for block storage devices used for storing local WAL, with the format: <br />`0@s3://$bucket?region=$region[&batchInterval=250][&maxBytesInBatch=8388608]`<br />For details on different vendors, refer to: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration)<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                         |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                           |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                            |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                           |

### s3.wal.cache.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | WAL (Write-Ahead Logging) cache is a FIFO (first-in-first-out) queue that contains data which has not been uploaded to object storage, as well as data that has been uploaded but not yet evicted from the cache. When the cache data that has not been uploaded fills up the entire capacity, storage will exert backpressure on subsequent requests until the data upload is completed. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                                                                                                                                                                |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                                                         |

### s3.wal.upload.threshold

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The threshold that triggers WAL uploads to object storage. The configuration value needs to be less than `s3.wal.cache.size`. The larger the configuration value, the higher the data aggregation degree, leading to lower metadata storage costs. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                    |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                         |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                  |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                  |

### s3.block.cache.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | `s3.block.cache.size` is the size of the block cache. The block cache is used to cache cold data read from object storage. It is recommended to set this configuration item to more than 4MB \* the number of concurrent cold reads per partition to achieve better cold read performance. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                            |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                                                                 |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                          |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                          |

### s3.stream.object.compaction.interval.minutes

| Item<br />                      | Description<br />                                                                                                                                                    |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The interval period for compaction in Stream objects. The larger the interval, the lower the cost of API calls, but it increases the size of metadata storage.<br /> |
| Value Type<br />                | int, measured in minutes<br />                                                                                                                                       |
| Default Value<br />             | 30<br />                                                                                                                                                             |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                      |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                      |

### s3.stream.object.compaction.max.size.bytes

| Item<br />                      | Description<br />                                                                                                                                                                        |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Stream object compaction allows for the maximum size of synthetic objects. The larger this value, the higher the cost of API calls, but the smaller the scale of metadata storage.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                            |
| Default Value<br />             | 10737418240<br />                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                          |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                          |

### s3.stream.set.object.compaction.interval.minutes

| Item<br />                      | Description<br />                                                                                                                                                                                                                              |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Sets the interval for stream object compaction. The smaller this value, the smaller the scale of the metadata storage, and the data becomes compacted sooner. However, the final stream objects will undergo compaction more frequently.<br /> |
| Value Type<br />                | int, measured in minutes<br />                                                                                                                                                                                                                 |
| Default Value<br />             | 5<br />                                                                                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                |

### s3.stream.set.object.compaction.cache.size

| Item<br />                | Description<br />                                                                                                                 |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | The size of memory available during the stream object compaction process. The larger this value, the lower the cost of API calls. |
| Value Type                | long, in bytes                                                                                                                    |
| Default Value             | 209715200                                                                                                                         |
| Legal Input Range<br />   | \[1048576, ...]<br />                                                                                                             |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                   |

### s3.stream.set.object.compaction.stream.split.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | During the Stream object compaction process, if the data volume within a single Stream exceeds this threshold, the Stream's data will be directly split and written into individual Stream objects. The smaller this value, the earlier the data is split from the Stream set object, resulting in lower subsequent API call costs for Stream object compaction, but leading to higher API call costs for the split.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                                                                                              |
| Default Value<br />             | 8388608<br />                                                                                                                                                                                                                                                                                                                                                                                                              |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                            |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                            |

### s3.network.baseline.bandwidth

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The total available bandwidth for object storage requests. This is used to prevent stream set object compaction and catch-up reads from occupying normal read-write traffic. Production and consumption will also individually consume inbound and outbound traffic.<br /> For instance, if this value is set to 100MB/s and normal read-write traffic is 80MB/s, then the available traffic for stream set object compaction would be 20MB/s.<br /> |
| Value Type<br />                | long, measured in byte/s<br />                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Default Value<br />             | 1073741824<br />                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                                                      |

### s3.stream.allocator.policy

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                  |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | S3Stream memory allocator policy. Note that when configured to use DIRECT memory, the heap size in the virtual machine options (e.g., -Xmx) and the direct memory size (e.g., -XX:MaxDirectMemorySize) need to be adjusted. You can set them via the environment variable KAFKA\_HEAP\_OPTS.<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                       |
| Default<br />                   | POOLED\_HEAP<br />                                                                                                                                                                                                                                                                                 |
| Valid Values<br />              | POOLED\_HEAP, POOLED\_DIRECT<br />                                                                                                                                                                                                                                                                 |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                    |

### s3.telemetry.metrics.level

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                                            |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | Sets the level of Metrics logging. The "INFO" level includes metrics that most users should be concerned with, such as throughput and latency of common stream operations. The "DEBUG" level includes detailed metrics useful for diagnostics, such as latencies at various stages when writing to underlying block devices. |
| Value Type<br />          | string<br />                                                                                                                                                                                                                                                                                                                 |
| Default Value             | INFO                                                                                                                                                                                                                                                                                                                         |
| Valid Input Range         | INFO, DEBUG                                                                                                                                                                                                                                                                                                                  |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                              |

### s3.telemetry.exporter.report.interval.ms

| Item<br />                      | Description<br />                           |
| ------------------------------- | ------------------------------------------- |
| Configuration Description<br /> | Sets the interval for Metrics export.<br /> |
| Value Type<br />                | int, in milliseconds<br />                  |
| Default Value<br />             | 30000<br />                                 |
| Valid Input Range<br />         | N/A<br />                                   |
| Importance Level<br />          | Low, set relatively broad<br />             |

### s3.telemetry.metrics.base.labels

| Item<br />                      | Description<br />                                                                                                                                                                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Multi-dimensional labels for metrics, used to attach static multi-dimensional labels to all monitoring metrics, enabling categorization, aggregation, and fine-grained analysis of metrics. The format is: key1=value1,key2=value2.<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                              |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                 |
| Importance level<br />          | Medium, requires careful configuration<br />                                                                                                                                                                                              |

### s3.telemetry.metrics.exporter.uri

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                           |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration description<br /> | The export URI for Metrics. The format is: `$type://?$param1=$value1&$param2=$value2`. Currently the supported types for 'type' are prometheus and otlp.<br />The format for prometheus: `prometheus://?host=$hostname&port=$port`<br />The format for otlp: `otlp://?endpoint=$endpoint&protocol=$protocol&compression=$compression`<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                  |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                   |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                  |

## Persistent Data Rebalancing Configuration

### metric.reporters

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                                                                       |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | A list of classes for metrics reporters. By implementing the `org.apache.kafka.common.metrics.MetricsReporter` interface, you can dynamically load new metrics. `JmxReporter` is always included to register JMX statistics. To enable AutoBalancing, `metric.reporters` must include `kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter`. |
| Value Type<br />          | list<br />                                                                                                                                                                                                                                                                                                                                              |
| Default Value<br />       | ""<br />                                                                                                                                                                                                                                                                                                                                                |
| Valid Input Range<br />   | N/A<br />                                                                                                                                                                                                                                                                                                                                               |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                         |

### autobalancer.reporter.metrics.reporting.interval.ms

| Item<br />                      | Description<br />                                      |
| ------------------------------- | ------------------------------------------------------ |
| Configuration Description<br /> | Interval for reporting data by Metrics Reporter.<br /> |
| Value Type<br />                | long, unit is milliseconds<br />                       |
| Default Value<br />             | 10000<br />                                            |
| Legal Input Range<br />         | \[1000, ...]<br />                                     |
| Importance Level<br />          | High, requires careful configuration<br />             |

### autobalancer.controller.enable

| Item<br />                      | Description<br />                              |
| ------------------------------- | ---------------------------------------------- |
| Configuration Description<br /> | Whether to enable automatic rebalancing.<br /> |
| Value Type<br />                | boolean<br />                                  |
| Default Value<br />             | true<br />                                     |
| Valid Input Range<br />         | N/A<br />                                      |
| Importance Level<br />          | High, requires careful configuration<br />     |

### autobalancer.controller.anomaly.detect.interval.ms

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration description<br /> | The minimum interval for the Controller to check if a data rebalance is needed. The actual time for the next rebalance also depends on the number of partitions that have been reassigned. Reducing the minimum check interval can increase the sensitivity of data rebalancing. This value should be greater than the broker metrics reporting interval to prevent the controller from missing recent reassignment results.<br /> |
| Value Type<br />                | long, unit is milliseconds<br />                                                                                                                                                                                                                                                                                                                                                                                                   |
| Default Value<br />             | 60000<br />                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                                                         |

### autobalancer.controller.exclude.topics

| Item<br />                      | Description<br />                                          |
| ------------------------------- | ---------------------------------------------------------- |
| Configuration Description<br /> | List of Topics to be excluded from data rebalancing.<br /> |
| Value Type<br />                | list<br />                                                 |
| Default Value<br />             | ""<br />                                                   |
| Valid Input Range<br />         | N/A<br />                                                  |
| Importance Level<br />          | High, requires careful configuration<br />                 |

### autobalancer.controller.exclude.broker.ids

| Item<br />                      | Description<br />                                                       |
| ------------------------------- | ----------------------------------------------------------------------- |
| Configuration Description<br /> | List of Broker Ids that should be excluded from data rebalancing.<br /> |
| Value Type<br />                | list<br />                                                              |
| Default Value<br />             | ""<br />                                                                |
| Valid Input Range<br />         | N/A<br />                                                               |
| Importance Level<br />          | High, requires careful configuration<br />                              |

## Table Topic

<Tip>
  Table Topic is a core feature designed by AutoMQ for modern data lake architectures. It integrates streaming data with data lake storage through architectural innovation, addressing traditional challenges such as the separation of streaming and batch processing, complex ETL processes, and high costs. For more details, see [Overview▸](/automq/table-topic/overview).
</Tip>

For the current Table Topic configuration model, including topic-level settings, record conversion, transformations, error handling, Schema Registry client options, Catalog authentication, and Catalog-specific options, see [Table Topic Configuration▸](/automq/table-topic/table-topic-configuration).


# Object Storage Configuration
Source: https://docs.automq.com/automq/configuration/object-storage-configuration

Optimize your data storage with AutoMQ, the cloud-native Kafka-compatible solution utilizing leading cloud object storage for superior cost efficiency and scalability.

AutoMQ is a cloud-native, redesigned distribution of Kafka that fully utilizes cloud object storage services. It can build the storage layer using object storage from major cloud providers or storage software.

### Object Storage URI Construction

The URI is a unique identifier used by AutoMQ to represent writes to object storage, adhering to established standards to reconcile differences among various object storage service providers. The URI components are as follows: `s3` specifies the protocol; `xxx_bucket` indicates the specific bucket; `region` and `endpoint` are additional parameters used by different vendors to locate the bucket.

Ultimately, replace the variables in the configuration templates `global.config` and `global.envs` from [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) to generate the final configuration files.

| Provider     | Type                   | URI Example                                                                                                                                                                                                                              |
| ------------ | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AWS          | Cloud Provider         | s3://xxx\_bucket?region=us-east-1                                                                                                                                                                                                        |
| AWS-CN       | Cloud Provider         | s3://xxx\_bucket?region=cn-northwest-1\&endpoint=[https://s3.amazonaws.com.cn](https://s3.amazonaws.com.cn)                                                                                                                              |
| ALIYUN       | Cloud Provider         | s3://xxx\_bucket?region=oss-cn-shanghai\&endpoint=[https://oss-cn-shanghai.aliyuncs.com](https://oss-cn-shanghai.aliyuncs.com)                                                                                                           |
| OCI<br />    | Cloud Provider<br />   | s3://xxx\_bucket?region=us-ashburn-1\&endpoint=[https://xxx\_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com\&pathStyle=true](https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com\&pathStyle=true)<br /> |
| CEPH<br />   | Storage Software<br /> | s3://xxx\_bucket?region=us-west-2\&endpoint=[http://127.0.0.1:80](http://127.0.0.1:80)<br />                                                                                                                                             |
| CubeFS<br /> | Storage Software<br /> | s3://xxx\_bucket?region=us-west-2\&endpoint=[http://10.1.0.240:17410](http://10.1.0.240:17410)<br />                                                                                                                                     |
| MinIO<br />  | Storage Software<br /> | s3://xxx\_bucket?region=us-west-2\&pathStyle=true\&endpoint=[http://10.1.0.240:9000](http://10.1.0.240:9000)<br />                                                                                                                       |

<Tip>
  If the `s3.wal.path` is set, you can also include the `batchInterval` and `maxBytesInBatch` parameters in the URI to control the submit interval and the maximum size of bytes per batch, respectively. The default settings are `batchInterval=250` milliseconds and `maxBytesInBatch=8388608` bytes.
</Tip>

### Cloud Object Storage Configuration

<Tabs>
  <Tab title="AWS">
    It is recommended to refer to the [Policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html) and [Endpoint & Region](https://docs.aws.amazon.com/general/latest/gr/s3.html) links in the AWS documentation to get detailed configuration instructions for IAM and S3.

    <Info>
      In a production environment, it is advisable to scope IAM Policy permissions to specific buckets to avoid unintended authorizations.
    </Info>

    ```json theme={null}
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:AbortMultipartUpload",
                    "s3:DeleteObject",
                    "s3:ListBucket"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }

    ```
  </Tab>

  <Tab title="Aliyun">
    It is recommended to refer to the official Alibaba Cloud documentation on [RAM Permission Policy Language Overview](https://www.alibabacloud.com/help/en/ram/user-guide/policy-overview) and [OSS Endpoints and Regions](https://www.alibabacloud.com/help/en/oss/regions-and-endpoints) for detailed instructions on configuring RAM permissions and OSS.

    <Info>
      In a production environment, it is advisable to scope RAM policy permissions to specific buckets (OSS storage spaces) to avoid overly broad authorization and reduce security risks.
    </Info>

    ```json theme={null}
    {
      "Version": "1",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "oss:PutObject",
            "oss:AbortMultipartUpload",
            "oss:GetObject",
            "oss:DeleteObject",
            "oss:PutObjectTagging"
          ],
          "Resource": [
             "acs:oss:*:*:examplebucket"
          ]
        }
      ]
    }

    ```
  </Tab>

  <Tab title="GCP">
    Since the AWS S3 SDK cannot directly access GCP Cloud Storage, AutoMQ Open Source cannot run directly on Google Cloud Storage. If you need to run AutoMQ on GCP, [contact the AutoMQ team](https://www.automq.com/contact) for support. AutoMQ BYOC supports GCP Cloud Storage.
  </Tab>

  <Tab title="Azure">
    [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) is not compatible with the AWS S3 network protocol; therefore, AutoMQ Open Source cannot run directly on Azure Blob Storage. Users can address this as follows:

    * Set up your own S3 Proxy to access Blob Storage.

    * [Contact the AutoMQ team](https://www.automq.com/contact) for support. AutoMQ BYOC supports Azure Blob Storage.
  </Tab>

  <Tab title="OCI">
    We recommend checking the official website for more detailed information.

    It is advisable to consult the [Policy](https://docs.oracle.com/en-us/iaas/Content/Security/Reference/objectstorage_security.htm#iam-policies), [Detailed Policy](https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/objectstoragepolicyreference.htm#Details), and [Endpoint & Region](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm#usingAPI) links, and review OCI documentation for more detailed configuration instructions on IAM and Object Storage.

    ```text theme={null}
    Allow group 'AutoMQ PoC' to manage objects in tenancy where any {target.bucket.name='automq-data', target.bucket.name='automq-ops'}
    ```
  </Tab>

  <Tab title="Other Cloud Providers">
    To deploy AutoMQ in other public cloud environments, make sure that the cloud provider's object storage service is compatible with the S3 protocol. You can contact the AutoMQ team via [this form](https://www.automq.com/contact) to obtain the necessary assistance and best practices.
  </Tab>
</Tabs>

### Storage Software Configuration

If you wish to deploy AutoMQ in a private data center, you can also use storage software like MinIO, Ceph, or CubeFS that supports the S3 protocol to provide object storage services.


# Performance Tuning for Client
Source: https://docs.automq.com/automq/configuration/performance-tuning-for-client

Optimize AutoMQ for peak performance with its cloud-native scalability and Kafka compatibility, ensuring cost efficiency and enterprise-grade low-latency support.

AutoMQ does not rely on local disks; instead, it writes data directly to object storage. In comparison to writing to local disks, file creation operations in object storage exhibit higher latency. For example, our empirical data shows that when writing a 4MiB file to S3, the P99 latency is approximately 400 milliseconds. Due to this, in some throughput-prioritized scenarios, it is necessary to suitably adjust client parameters to meet the system's performance goals ([Little’s Law](https://en.wikipedia.org/wiki/Little's_law)).

<Tip>
  Note:

  The configurations listed in this document are for reference only. Users need to consider their specific scenario demands and system goals comprehensively to reasonably adjust the corresponding parameter values.

  If your scenario is sensitive to message read and write latency, contact the AutoMQ team through [this form](https://www.automq.com/contact) for support. AutoMQ BYOC and AutoMQ Software support single-digit millisecond low-latency data reads and writes.
</Tip>

### Producer Configuration

| **Producer Configuration Option**<br /> | **Description**<br />                                                                                                                 | **Recommended Value**<br /> |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| linger.ms<br />                         | The delay time for the Producer to batch send messages, enhancing the efficiency of each request by accumulating more messages.<br /> | 100<br />                   |
| metadata.max.age.ms<br />               | The forced refresh time for metadata to prevent routing errors due to metadata expiration.<br />                                      | 60000<br />                 |
| batch.size<br />                        | The maximum number of bytes in a single batch, directly affecting the number of network requests and throughput.<br />                | 1048576<br />               |
| max.request.size<br />                  | The maximum number of bytes in a single request, limiting the size of messages the Producer can send.<br />                           | 16777216<br />              |

### Consumer Configuration

| **Consumer Configuration Options** | **Description**                                                                                                                                            | **Recommended Value** |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| metadata.max.age.ms                | The forced refresh time for metadata to prevent route errors due to expired metadata.                                                                      | 60000                 |
| max.partition.fetch.bytes          | Limits the maximum amount of data returned in a Fetch request from a single partition, working together with fetch.max.bytes to control fetch granularity. | 8388608               |


# AutoMQ Benchmark
Source: https://docs.automq.com/automq/deployment/automq-benchmark

Test AutoMQ cluster performance using Kafka CLI tools or the AutoMQ performance testing tool. Covers tail read and catch-up read scenarios.

AutoMQ uses an S3 shared storage architecture that separates compute and storage. It provides 100% compatibility with Apache Kafka while offering rapid scalability, cost efficiency, and high performance, especially in cold read scenarios. This document describes how to conduct performance testing on an AutoMQ cluster.

## Prerequisites

Before conducting cluster performance testing, the following conditions must be met:

1. **Complete the installation and deployment of the AutoMQ cluster.** You can refer to the following methods for installing and deploying AutoMQ.

* [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux)

* [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes)

2. **Prepare necessary load generators.** It is recommended to create a certain number of load generators within the same VPC network to ensure that network throughput and the CPU of the load generators do not become bottlenecks in the testing scenario.

## Test Scenario

### Scenario 1: Tail Read

Tail Read, commonly referred to as "hot read," evaluates conditions where the offset difference between the Producer and Consumer is minimal. The essential metrics for hot write and hot read scenarios are as follows:

* **Read/Write Throughput:** Real-time read/write scenarios are typical for Kafka. Achieving higher throughput with the same hardware resources implies improved performance.

* **Write and E2E Latency:** Given constant network throughput, lower message write latency is desirable. Similarly, reduced E2E latency (the time taken from production to consumption of a message) is advantageous.

In the Tail Read scenario, messages sent by the Producer are consumed immediately after being written to the Broker. Here, messages consumed by the Consumer are retrieved directly from the AutoMQ Log Cache, which eliminates the need to read from object storage, thereby reducing resource consumption.

### Scenario 2: Catch-Up Read

Catch-Up Read, also known as a "catch-up read" or "cold read," evaluates the scenario where the Consumer's consumption offset significantly lags behind the Producer's offset. In this scenario, the Consumer is initially paused and, after accumulating a certain number of messages, consumption resumes. At this point, messages consumed by the Consumer are fetched from object storage, with Block Cache handling pre-reading and caching.

In this test scenario, the following metrics are primarily focused on:

* **Is the catch-up read speed fast enough**? Determine whether the consumption speed of each Consumer Group exceeds the Producer's write speed. Only if it does can it be said that the Consumer is capable of catching up with the Producer.

* **Is Write Throughput Affected During Catch-Up Reads?** Check to see if there's a reduction in the message flow from the Producer during catch-up reads and if there's an increase in send latency.

## Testing with Kafka CLI

Users can perform performance tests using Kafka CLI tools by running `kafka-producer-perf-test.sh` and `kafka-consumer-perf-test.sh`.

If your previous AutoMQ cluster was deployed following the guide [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux), the resulting cluster bootstrap address will look something like "**192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092**".

<Tip>
  Adjust the bootstrap-server addresses below to match your actual cluster configuration.
</Tip>

### Create Topic

```bash theme={null}
./kafka-topics.sh --create --topic test-topic --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

```

### Send Message

```bash theme={null}
./kafka-producer-perf-test.sh --topic test-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
```

<Tip>
  The parameters `linger.ms` and `batch.size` are crucial for performance tuning. You can consult [Performance Tuning For Client▸](/automq/configuration/performance-tuning-for-client) to adjust these settings according to your specific needs.
</Tip>

### Consume Message

```bash theme={null}
./kafka-consumer-perf-test.sh --topic test-topic --show-detailed-stats --timeout 300000 --messages=1024000 --reporting-interval 1000 --bootstrap-server=192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
```

## Testing with the AutoMQ Performance Testing Tool (Recommended)

The AutoMQ team has enhanced the Kafka performance testing CLI tool provided by the community and released a one-click performance testing tool. This tool draws on the OpenMessaging Benchmark framework, providing the following advantages:

* **Support for single-process multiple clients to enhance benchmarking density and efficiency:** Unlike Apache Kafka's built-in kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh scripts, the automq-perf-test.sh script allows for multiple Producers and Consumers to run within a single process, thereby sending and receiving messages across multiple Topics. This approach better mirrors real-world scenarios and improves ease of use.

* **One-click execution without distributed deployment:** In contrast to the OpenMessaging Benchmark testing framework, automq-perf-test.sh eliminates the need for distributed deployment of multiple Workers, enabling you to execute tests with a single click on a single machine. This simplifies deployment and use in testing scenarios where large scale is not required.

* **Provides Catch Up Read test scenarios:** Furthermore, the automq-perf-test.sh script can handle more complex cold read test scenarios by permitting multiple Consumer Groups to start consumption at different offsets. This setup helps prevent cache reuse during cold reads, allowing for performance testing under more rigorous conditions.

* **Open and neutral:** This testing script exclusively utilizes the Apache Kafka Client, which ensures compatibility with performance testing on stream systems like Apache Kafka, MSK, and other platforms supporting the Kafka protocol.

### Scenario 1: Tail Read

The following use case tests the Tail Read performance of AutoMQ. This test case involves:

* A production and consumption traffic ratio of 1:1.

* Data is written to a total of 1,280 partitions across 10 topics.

* Writing 1600 messages per second, each 51 KiB in size (without any batching), resulting in a write speed of 80 MiB/s.

Note that before executing the script below, you need to replace the `--bootstrap-server` address with the actual AutoMQ access point address.

```bash theme={null}
KAFKA_HEAP_OPTS="-Xmx12g -Xms12g" ./bin/automq-perf-test.sh \
--bootstrap-server 0.kf-v8tj9bmunqdo1og8.wanshao-for-aws.automq.private:9092,1.kf-v8tj9bmunqdo1og8.wanshao-for-aws.automq.private:9092,2.kf-v8tj9bmunqdo1og8.wanshao-for-aws.automq.private:9092 \
--producer-configs batch.size=0 \
--consumer-configs fetch.max.wait.ms=1000 \
--topics 10 \
--partitions-per-topic 128 \
--producers-per-topic 1 \
--groups-per-topic 1 \
--consumers-per-group 1 \
--record-size 52224 \
--send-rate 1600 \
--warmup-duration 10 \
--test-duration 5 \
--reset
```

### Scenario 2: Catch-Up Read

The following use case tests the Catch-Up Read performance of AutoMQ, structured as follows:

* The production-to-consumption traffic ratio is 1:3.

* Data is written to a total of 1,280 partitions across 10 topics.

* 800 messages, each sized at 64 KiB, are written per second (with no batching), resulting in a write speed of 50 MiB/s.

* Buffer 600 seconds of data (\~30 GiB) before beginning to catch up, then initiate 3 Consumer Groups to start catching up, with each Group's starting point staggered by 30 seconds (\~1.5 GiB).

Note that before executing the script below, you need to replace the `--bootstrap-server` address with the actual AutoMQ access point address.

```bash theme={null}
KAFKA_HEAP_OPTS="-Xmx12g -Xms12g" ./bin/automq-perf-test.sh \
--bootstrap-server 0.kf-hsd29pri8q5myud5.wanshao-for-aws.automq.private:9092,1.kf-hsd29pri8q5myud5.wanshao-for-aws.automq.private:9092,2.kf-hsd29pri8q5myud5.wanshao-for-aws.automq.private:9092 \
--producer-configs batch.size=0 \
--consumer-configs fetch.max.wait.ms=1000 \
--topics 10 \
--partitions-per-topic 128 \
--producers-per-topic 1 \
--groups-per-topic 3 \
--consumers-per-group 1 \
--record-size 65536 \
--send-rate 800 \
--backlog-duration 600 \
--group-start-delay 30 \
--warmup-duration 5 \
--reset
```

### Startup Parameters

* `--bootstrap-server`: Specifies the initial connection nodes for the Kafka cluster, provided in the format "host1:port1,host2:port2". It should be noted that these addresses are only used for the initial connection to obtain cluster metadata, so you do not need to provide the addresses of all Brokers in the cluster. Just a few running and accessible addresses will suffice.

* `--common-configs`: Specifies the common configuration for Kafka Admin Client, Producer, and Consumer, including authentication-related configurations.

* `--topic-configs`: Specify configurations related to topics, such as message retention time, etc.

* `--producer-configs`: Specify configurations related to producers, like batch size, batch duration, compression method, etc.

* `--consumer-configs`: Specify configurations related to consumers, including the maximum size of messages fetched in a single operation, etc.

* `--reset`: Determines whether to delete all existing topics in the cluster before executing the benchmark.

* `--topic-prefix`: The prefix for topics used during the test.

* `--topics` : The number of topics created during testing.

* `--partitions-per-topic` : The number of partitions in each topic. The total number of partitions used for testing is calculated as `--topics` \* `--partitions-per-topic`.

* `--producers-per-topic` : The number of producers created for each topic. The total number of producers used for testing is calculated as `--topics` \* `--producers-per-topic`.

* `--groups-per-topic` : The number of consumer groups created for each topic, which also represents the read-write ratio (fan-out) during testing.

* `--consumers-per-group` : The number of consumers in each consumer group. The total number of consumers used for testing is calculated as `--topics` \* `--groups-per-topic` \* `--consumers-per-group`.

* `--record-size`: The size of each message sent by the Producer, measured in bytes.

* `--send-rate`: The total number of messages sent per second by all Producers. The product of `--record-size` and `--send-rate` represents the write throughput during testing.

* `--random-ratio`: The proportion of random data in messages, commonly used for testing scenarios where the Producer has enabled compression. The value ranges between 0.0 and 1.0; the larger the value, the more random data in messages, theoretically resulting in poorer compression efficiency. The default value is 0.0, meaning each message is entirely identical.

* `--random-pool-size`: The size of the random message pool. A message is randomly selected from this pool each time a message is sent. This option is only effective when `--random-ratio` is greater than 0.

* `--backlog-duration`: Used in catch-up read test scenarios to control the duration of message accumulation, measured in seconds. The product of `--record-size`, `--send-rate`, and `--backlog-duration` represents the size of accumulated messages before the catch-up reading begins.

* `--group-start-delay`: Used in catch-up read test scenarios to control the interval at which each Consumer Group begins consumption. Measured in seconds, this option can stagger the consumption progress of each Consumer Group to prevent cache reuse, thereby more accurately simulating real catch-up read scenarios.

* `--send-rate-during-catchup`: Used in catch-up read test scenarios to control the sending rate of the Producer during the catch-up read period. By default, it mirrors the setting of `--send-rate`.

* `--warmup-duration`: Specifies the duration of the warm-up period before conducting the test, measured in minutes. During the warm-up period, the first 50% gradually increases the Producer's sending rate to the level specified by `--send-rate`, and the latter 50% maintains it at `--send-rate`. Metrics collected during the warm-up period are excluded from the final results. To effectively warm up the JVM, it is advised to set `--warmup-duration` to at least 10 minutes.

* `--test-duration`: Indicates the duration of the actual test, measured in minutes. This parameter is applicable only in non-catch-up read test scenarios (when `--backlog-duration` is less than or equal to 0).

* `--reporting-interval`: Determines the frequency for collecting statistics of related metrics during the test, measured in seconds.

## Performance Optimization and Technical Support

The performance of an AutoMQ cluster is influenced by various factors, including computing resource specifications and kernel parameter tuning. Performance optimization and baseline calibration for production environments can be quite complex.

You can reach out to the AutoMQ team via [this form](https://www.automq.com/contact) to acquire performance test reports and best practices for performance optimization in production scenarios.


# Deploy Multi-Nodes Cluster on Kubernetes
Source: https://docs.automq.com/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes

Deploy a scalable, cost-efficient AutoMQ cluster on Kubernetes. Explore Kafka-compatible features and a 30-day free cloud trial. Contact us for guidance.

This topic introduces how to deploy a multi-node AutoMQ cluster using Kubernetes, allowing users to validate cluster-related features such as partition reassignment and data auto-balancing in this development environment.

In addition to the Kubernetes deployment solution, users can refer to the following documentation to explore other deployment options:

* [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux)

<Tip>
  Deploying AutoMQ and tuning parameters for production load is relatively complex. You can contact the AutoMQ team through [this form](https://www.automq.com/contact) to receive necessary assistance and best practices.
</Tip>

Moreover, if you wish to completely avoid the burden of installation and deployment, you can experience the fully managed cloud service provided by the AutoMQ team through the following link. Currently, all cloud markets offer a free 30-day trial.

* [Experience AutoMQ Cloud for free from AWS Marketplace](/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws).

## Prerequisites

This document provides examples for deploying a 5-node AutoMQ cluster. In this setup, 3 nodes will run both the Controller and Broker, while the other 2 nodes will run only the Broker.

Ensure the following conditions are met:

* Prepare a Kubernetes cluster with at least 5 nodes, recommending network-optimized virtual machines with 4 cores and 16GB of RAM for subsequent Pod creation and other operations.

* Helm chart requires version v3.8.0 or later. Refer to the [Helm Chart Quickstart](https://helm.sh/docs/intro/quickstart/).

* Utilize the Bitnami Helm repository. AutoMQ is fully compatible with Bitnami's Helm Charts, enabling you to customize the AutoMQ Kubernetes cluster using Bitnami's values.yaml.

* Prepare 2 object storage buckets: one for storing message data and another for storing system logs and metric data.

## Deploy AutoMQ Cluster

### Step 1: Edit the Configuration File

Create an empty `automq-values.yaml` file, edit it, and add specific parameters. For guidance, refer to [demo-values.yaml](https://github.com/AutoMQ/automq-labs/blob/main/opensource-examples/setup/kubernetes/bitnami/demo-values.yaml) and check recommended configuration examples for different scenarios in its subdirectories. For more details, consult the [README.md](https://github.com/AutoMQ/automq-labs/blob/main/opensource-examples/setup/kubernetes/bitnami/README.md).

* Substitute `${ops-bucket}`, `${data-bucket}`, `${region}`, and `${endpoint}` with specific values for object storage. For more details, refer to [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).

* Replace `${access-key}` and `${secret-key}` with the actual values. You can also choose alternative authorization methods such as IAM Role.

* For production-grade deployments, it is recommended to use dedicated nodes for AutoMQ (to avoid network bandwidth and other resource competition with other Pods). It is advisable to match using node affinity (nodeAffinity) and tolerations tags.

* For multi-availability zone deployments, you can use the `topologySpreadConstraints` parameter to ensure Pods are evenly distributed across the specified availability zones.

```yaml theme={null}
controller:
  topologySpreadConstraints:
    - maxSkew: 1             
      topologyKey: "topology.kubernetes.io/zone"
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: automq

```

* To avoid cross Availability Zone traffic, `brokerRackAssignment` will ultimately set the AutoMQ Broker's `broker.rack`. Clients need to be configured accordingly to eliminate cross-Availability Zone traffic costs. For example, the following configuration can be used in AWS EKS:

```yaml theme={null}

brokerRackAssignment: aws-az

```

* Several other server-side parameters can be set as needed in `controller.extraConfig` and `broker.extraConfig`. For more details, refer to: [Broker And Controller Configuration▸](/automq/configuration/broker-and-controller-configuration).

### Step 2: Install AutoMQ

Install or upgrade the AutoMQ Helm Chart using a custom YAML file: It is recommended to use the `--version` flag to specify the Bitnami Helm Chart version [31.x.x (31.1.0 \~ 31.5.0)](https://artifacthub.io/packages/helm/bitnami/kafka) when installing AutoMQ.

```bash theme={null}
helm install automq-release oci://registry-1.docker.io/bitnamicharts/kafka -f demo-values.yaml --version 31.5.0 --namespace automq --create-namespace
```

Wait for the AutoMQ cluster to be ready:

```bash theme={null}
kubectl --namespace automq rollout status statefulset --watch
```

When the AutoMQ cluster is ready, the output should look like the following:

```text theme={null}
statefulset rolling update complete 2 pods at revision automq-kafka-broker-6c756696dd...
statefulset rolling update complete 3 pods at revision automq-kafka-controller-c574d5fd5...

```

Check the Pod list:

```bash theme={null}
~/.kube kubectl get pods
NAME                                          READY   STATUS    RESTARTS   AGE
data-automq-kafka-controller-0                1/1     Running   0          16m
data-automq-kafka-controller-1                1/1     Running   0          16m
data-automq-kafka-controller-2                1/1     Running   0          16m
data-automq-kafka-broker-0                    1/1     Running   0          13m
data-automq-kafka-broker-0                    1/1     Running   0          13m
```

## Test Message Sending and Receiving

After the Helm execution is complete, the access address for the cluster and the commands for testing message sending and receiving will be displayed. This allows you to perform Topic message sending and consumption tests using `kafka-console-producer.sh` and `kafka-console-consumer.sh`.

<img />

## Stop and Uninstall the AutoMQ Cluster

* After completing the tests, AutoMQ clusters can be stopped and uninstalled using `helm uninstall`.

```bash theme={null}
helm uninstall automq-release -n automq
```

* If historical data is no longer needed, it's necessary to delete the cluster's PVC and Bucket data altogether to prevent leftover data from affecting the next deployment.

## Precautions for Production Environment

### Lock Chart Version

To avoid unexpected changes during deployment, it is recommended to lock the Helm Chart version. Locking the version means specifying an exact version at deployment instead of using the latest or unspecified version. Locking the Helm Chart version aids in:

* **Ensure Compatibility**: Make sure the application's performance in the deployed environment aligns with its behavior during testing, even if new Chart versions are released.

* **Prevent Unintended Updates**: Prevent automatic updates that may introduce changes incompatible with your current deployment or operational practices.

### Name Override

When deploying multiple instances of the same Helm Chart within a Kubernetes cluster, name conflicts may occur. Use `nameOverride` and `fullnameOverride` to distinguish between different instances. For example, using distinct names for your production and staging environments can help avoid confusion.

* Using `nameOverride`, the StatefulSet name will be `<release-name>-<nameOverride>`.

* Using `fullnameOverride`, the StatefulSet name will be `<fullnameOverride>`.

```yaml theme={null}

nameOverride: 'automq-prod'
fullnameOverride: 'automq-instance-prod'

```

### Docker Image

Bitnami provides the Docker image for AutoMQ deployment, with the default image being `bitnami/kafka:latest`. You should replace it with a custom AutoMQ image specifying a particular version:

```yaml theme={null}
global:
  security:
    allowInsecureImages: true
image:
  registry: automqinc
  repository: automq
  tag: 1.5.5-bitnami
  pullPolicy: Always

```

### Scheduling Strategy

For AutoMQ, a refined scheduling strategy in Kubernetes can be implemented using node affinities and tolerations. We suggest that a production-level AutoMQ operates exclusively without co-locating with other applications. It is advisable to tailor label matching rules based on node types:

#### Tolerance

It is recommended to add a taint to the Kubernetes node group: `key: "dedicated", operator: "Equal", value: "automq", effect: "NoSchedule"`.

```yaml theme={null}
controller:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "automq"
      effect: "NoSchedule"   
broker:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "automq"
      effect: "NoSchedule"   

```

#### Node Affinity

Override the default values in the controller/broker configuration to match the node labels (e.g., `node-type: m7g.xlarge`):

```yaml theme={null}
controller:  
  affinity:
    nodeAffinity:  
      requiredDuringSchedulingIgnoredDuringExecution:  
        nodeSelectorTerms:  
        - matchExpressions:  
          - key: "node-type"  
            operator: In  
            values: ["m7g.xlarge"]  

```

#### Pod Anti-affinity

Ensure that the controller component and the broker component are not scheduled on the same node by using the `podAntiAffinity` parameter:

```yaml theme={null}
controller:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/instance
                operator: In
                values:
                  - automq
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - controller-eligible
                  - broker
          topologyKey: kubernetes.io/hostname

```

### Scaling

#### Controller

The number of instances is configured through `controller.replicas`, which supports horizontal scaling. By default, the cluster deploys 3 Controller Pods, but users can customize the number of Controller replicas.

<Danger>
  Note: Once the cluster deployment is complete, adjusting the replicas for the Controller is not recommended to avoid unexpected risks.
</Danger>

#### Broker

The number of instances is configured through `broker.replicas`, which supports horizontal scaling.

#### AutoScaling (HPA)

HPA is disabled by default. To enable it, you can configure parameters in `broker.autoscaling.hpa`:

```yaml theme={null}
broker:
  autoscaling:
    hpa:
      enabled: true        # Enable HPA
      minReplicas: "1"     # Minimum Replicas
      maxReplicas: "3"     # Maximum Replicas
      targetCPU: "60"      # Target CPU Utilization Rate (%)
      targetMemory: ""     # Target Memory Utilization (% Optional)

```

<Danger>
  Note: We do not recommend configuring Controller HPA because the Controller in Kafka KRaft mode relies on the Raft protocol to maintain metadata consistency, which does not support automated Raft membership changes. Therefore, configuring Controller HPA could lead to quorum failure or make the cluster unavailable.
</Danger>

### Resource Configuration

It is recommended that each Pod for AutoMQ runs on resources of 4Core16GB. Adjust resource parameters through the following configurations:

```yaml theme={null}
controller:
  replicaCount: 3
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"
  heapOpts: -Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m

```

```yaml theme={null}
broker:
  replicaCount: 2
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"
  heapOpts: -Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m

```

### Security and Authentication

Each listener configured in Kafka can have a different authentication protocol. For instance, you can use `sasl_tls` authentication for client communications and `tls` for inter-Controller and Broker communications. The table below lists available protocols and their security features (see more details in [Kafka Security Authentication](https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md#enable-security-for-kafka)):

| Method          | Authentication Method       | Encrypted via TLS |
| --------------- | --------------------------- | ----------------- |
| plaintext       | None                        | No                |
| tls             | None                        | Yes               |
| mtls            | Yes (Mutual Authentication) | Yes               |
| sasl            | Yes (via SASL)              | No                |
| sasl\_tls<br /> | Yes (via SASL)<br />        | Yes<br />         |

### External Access

Additional listeners and advertised listeners must be configured, and a specific service must be created for each Kafka Pod.

There are three ways to configure external access: by using LoadBalancer services, NodePort services, or ClusterIP services. For more information, refer to the [Kafka External Access](https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md#accessing-kafka-brokers-from-outside-the-cluster) section.

### **Monitoring**

The primary focus is on the integration of this Chart with Prometheus. For more details, refer to the section [Enable Prometheus Metrics](https://github.com/bitnami/charts/tree/main/bitnami/kafka#prometheus-metrics).

### Table Topic Feature

The Table Topic feature supports integration between topic streaming data and a static data lake. To enable the Table Topic feature in an AutoMQ cluster, refer to [Overview▸](/automq/table-topic/overview).


# Deploy Multi-Nodes Cluster on Linux
Source: https://docs.automq.com/automq/deployment/deploy-multi-nodes-cluster-on-linux

Deploy a scalable, multi-node AutoMQ cluster on Linux with full Kafka compatibility. Benefit from object storage integration and cloud-native performance.

This document outlines how to deploy a multi-node AutoMQ cluster on a Linux host (utilizing object storage services from cloud vendors or S3-compatible storage software). In this development environment, users can test cluster-related features such as partition reassignment and automatic data balancing.

In addition to the Linux host deployment plan, users can refer to the following documents to explore other deployment options:

* [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes)

<Tip>
  Deploying AutoMQ for production workloads and optimizing its parameters can be relatively complex. You can reach out to the AutoMQ team through [this form](https://www.automq.com/contact) for necessary assistance and best practices.
</Tip>

Moreover, if you want to completely eliminate the installation and deployment effort, you can try the fully managed cloud services provided by the AutoMQ team through the link below. Currently, all cloud markets offer a free 30-day trial.

* [Get a free AutoMQ Cloud trial from AWS Marketplace](https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws).

## **Prerequisites**

This documentation example is intended to deploy a 5-node AutoMQ cluster, where 3 nodes run both the Controller and Broker, and the remaining 2 nodes run only the Broker.

Ensure the following prerequisites:

* Five Linux hosts, preferably network-optimized virtual machines with 4 CPUs and 16GB of RAM, for deploying the AutoMQ cluster. Make sure the system disk storage space is at least 20GB, and data volume disk space is no less than 20GB (required only for deploying Controller nodes). An example is as follows:

  | Role                      | IP                | Node ID | System Volume  | Metadata Volume |
  | ------------------------- | ----------------- | ------- | -------------- | --------------- |
  | Controller + Broker<br /> | 192.168.0.1<br /> | 0<br /> | EBS 20GB<br /> | EBS 20GB<br />  |
  | Controller + Broker<br /> | 192.168.0.2<br /> | 1<br /> | EBS 20GB<br /> | EBS 20GB<br />  |
  | Controller + Broker<br /> | 192.168.0.3<br /> | 2<br /> | EBS 20GB<br /> | EBS 20GB<br />  |
  | Broker                    | 192.168.0.4       | 1000    | EBS 20GB       | Not applicable  |
  | Broker                    | 192.168.0.5       | 1001    | EBS 20GB       | Not applicable  |

* Download the binary installation package to set up AutoMQ. Refer to [Software Artifact▸](/automq/releases/software-artifact).

* Create two custom-named object storage buckets, such as automq-data and automq-ops, as used in this document.

* Create an IAM user and generate an Access Key ID and Access Key Secret for it. Then, ensure that the IAM user has full read and write permissions to the previously created object storage bucket.

* Install jemalloc to optimize memory allocation. Refer to [Getting-started](https://github.com/jemalloc/jemalloc/wiki/getting-started).

<Tip>
  AutoMQ relies on the Kafka KRaft component for maintaining cluster metadata; therefore, in production scenarios, each cluster requires the deployment of at least 3 nodes (running both Controller and Broker).
</Tip>

## Install and Launch the AutoMQ Cluster

### Step 1: Create a Cluster Deployment Project

AutoMQ provides the `automq-cli.sh` tool for managing AutoMQ clusters. By using the command `automq-cli.sh cluster create [project]`, a cluster configuration template will be automatically created in the directory `clusters/[project]/topo.yaml`.

```bash theme={null}
bin/automq-cli.sh cluster create poc

```

An example of the execution result is as follows:

```text theme={null}
Success create AutoMQ cluster project: poc
========================================================
Please follow the steps to deploy AutoMQ cluster:
1. Modify the cluster topology config clusters/poc/topo.yaml to fit your needs
2. Run ./bin/automq-cli.sh cluster deploy --dry-run clusters/poc , to deploy the AutoMQ cluster
```

### Step 2: Edit the Cluster Configuration Template

Edit the configuration template generated in Step 1 during cluster creation. An example of the configuration template is as follows:

```yaml theme={null}
global:
  clusterId: ''
  # Bucket URI Pattern: 0@s3://$bucket?region=$region&endpoint=$endpoint
  # Bucket URI Example:
  # AWS : 0@s3://xxx_bucket?region=us-east-1
  # OCI: 0@s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
  config: |
    s3.data.buckets=0@s3://xxx_bucket?region=us-east-1
    s3.ops.buckets=1@s3://xxx_bucket?region=us-east-1
    s3.wal.path=0@s3://xxx_bucket?region=us-east-1
    log.dirs=/root/kraft-logs
  envs:
    - name: KAFKA_S3_ACCESS_KEY
      value: 'xxxxx'
    - name: KAFKA_S3_SECRET_KEY
      value: 'xxxxx'
controllers:
  # The Controllers Default Are Combined Nodes Which Roles Are Controller and Broker.
  # The Default Controller Port Is 9093 and the Default Broker Port Is 9092
  - host: 192.168.0.1
    nodeId: 0
  - host: 192.168.0.2
    nodeId: 1
  - host: 192.168.0.3
    nodeId: 2
brokers:
  - host: 192.168.0.5
    nodeId: 1000
  - host: 192.168.0.6
    nodeId: 1001
```

* `global.clusterId`: A randomly generated unique ID, which does not require modification.

* `global.config`: This is the custom incremental configuration for all nodes within the cluster. You need to specify actual values for `s3.data.buckets`, `s3.ops.buckets`, and `s3.wal.path`. Additional configuration items can be added with line breaks. For details on configuring object storage parameters for different cloud providers, refer to [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).

* `global.envs`: These are the environment variables for the node. Replace `KAFKA_S3_ACCESS_KEY` and `KAFKA_S3_SECRET_KEY` with actual values.

* `controllers`: This is a list of Controller nodes that must be updated with actual values.

* `brokers`: This is a list of Broker nodes that also need to be updated with actual values.

### Step 3: Start AutoMQ

Execute the cluster pre-check command to generate the final startup command.

```bash theme={null}
bin/automq-cli.sh cluster deploy --dry-run clusters/poc
```

This command will initially verify the S3 configuration for accuracy, ensure successful access to S3, and then output the startup commands for each node. An example output is as follows:

```text theme={null}
Host: 192.168.0.1
KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'

...
```

To start the cluster, sequentially execute the list of commands provided in the previous step on the designated CONTROLLER or BROKER hosts. For instance, on 192.168.0.1, begin the first CONTROLLER process by executing the corresponding command from the generated startup command list for that host.

```bash theme={null}
KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'
```

## Test Message Sending and Receiving

After installing and starting the AutoMQ cluster, you can test message sending, consumption, and other functionalities by executing the Kafka CLI commands available in the bin directory of the installation package.

1. Execute the following command to run kafka-topics.sh and create a Topic.

```bash theme={null}
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

```

1. Execute the following command to run kafka-console-producer.sh and send test messages.

```bash theme={null}
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
```

1. Execute the following command to run kafka-console-consumer.sh and consume test messages.

```bash theme={null}
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092
```

## Stop and Uninstall the AutoMQ Cluster.

After completing the tests, you can refer to the following steps to stop and uninstall the AutoMQ cluster.

1. Run the following command on each node to terminate the process.

```bash theme={null}
bin/kafka-server-stop.sh
```

2. You can automatically clear data in `s3-data-bucket` and `s3-ops-bucket` by setting lifecycle rules for object storage, and then remove these buckets.

3. Delete the created compute instances along with their respective system volumes and data volumes.

4. Remove the IAM user set up for testing along with its Access Key ID and Access Key Secret.

<Tip>
  Table Topic is a core feature designed by AutoMQ for modern data lake architectures. It integrates streaming data with static data lakes and addresses traditional stream-batch separation, complex ETL processes, and high costs through architectural innovation. For instructions on enabling Table Topic in the cluster, refer to [Overview▸](/automq/table-topic/overview).
</Tip>


# Introduction and Deployment Recommendations for AutoMQ Open Source Image
Source: https://docs.automq.com/automq/deployment/deployment-recommendations

AutoMQ offers cloud-native scalability and Kafka protocol compatibility, enabling flexible deployments in complex environments with Kubernetes lifecycle tools for efficient resource management.

## Introduction

AutoMQ is a cloud-native streaming platform that is fully compatible with the Apache Kafka protocol.

This topic describes that when deploying a multi-node AutoMQ cluster based on Kubernetes, it is possible to use Kubernetes application lifecycle management tools such as custom  `  Helm Chart  ` ,  `  Operator  `  to flexibly transform and adapt relevant dependent resources and configurations, and finally complete the deployment in a complex environment.

## Analysis of Differences between AutoMQ & Apache Kafka Docker Images

### Init & Startup

| -<br />                    | Apache Kafka Open Source Image<br />                                                                                                                                                                                                                                      | AutoMQ Open Source Image<br />                                                                                                                                                                                                                                                                                                         |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Startup Mode<br />         | encapsulates the default startup logic and is Out Of The Box<br />                                                                                                                                                                                                        | Compatible with the startup logic of the Apache Kafka Docker image, Out Of The Box<br />                                                                                                                                                                                                                                               |
| Configuration Method<br /> | <ul><li>Provides configuration conversion logic for environment variables with specific [rules](https://github.com/apache/kafka/blob/trunk/docker/examples/README.md#using-environment-variables)</li><li>Fixed mount path: /opt/kafka/config/server.properties</li></ul> | <ul><li>Configuration conversion logic for environment variables compatible with Apache Kafka docker image</li><li>server.properties can also be flexibly injected using other methods such as Helm Chart/Operator</li><li>The mount path of the configuration file is consistent with that of the Apache Kafka Docker image</li></ul> |
| Simple example<br />       | [Apache Kafka Docker Hub](https://hub.docker.com/r/apache/kafka#overriding-the-default-broker-configuration)<br />                                                                                                                                                        | [AutoMQ for Apache Kafka](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/setup/docker-compose/kafka)<br />                                                                                                                                                                                                        |

### AutoMQ-specific configuration

AutoMQ employs a shared storage architecture and requires additional startup configuration parameters. It is important to note that all parameters detailed later in this chapter fall under these additional requirements and are mandatory for proper operation.

#### S3 primary storage

Required shared storage parameters:

```properties theme={null}
# Object Storage settings. You will need to replace the placeholder values (marked with `${...}`), such as the S3 bucket names (`ops-bucket`, `data-bucket`), bucket region, and endpoint.
s3.data.buckets=0@s3://${data-bucket}?region=${region}&endpoint=${endpoint}
s3.wal.path=0@s3://${data-bucket}?region=${region}&endpoint=${endpoint}
s3.ops.buckets=1@s3://${ops-bucket}?region=${region}&endpoint=${endpoint}
elasticstream.enable=true
```

#### Kraft

##### node.id

Among the many Kubernetes application lifecycle management tools, there is already a feature that dynamically injects node.id into server.properties based on the Kubernetes Statefulset ID prefix; however, additional necessary generation rules are still required:

* Controller node.id needs to increment starting from 0 (min.id=0)

* Broker node.id needs to increment starting from 1000 (min.id=1000)

```properties theme={null}
# controller node.id start with 0
node.id=0

# broker node.id start with 1000
node.id=1000
```

##### role

We recommend that Controller serve as both controller and broker roles simultaneously

```properties theme={null}
process.roles=broker,controller
```

#### Listener

AutoMQ is fully compatible with the Kafka Listener system and can be customized to set `PLAINTEXT` , `SASL` , `TLS` , `mTLS`  protocols, with the default being `PLAINTEXT` .

It is worth noting that if the  `inter-broker`  protocol is not PLAINTEXT, you need to set the corresponding inter-broker authentication protocol (taking SASL\_PLAINTEXT as an example) for the  `AutoBalancer`  component of AutoMQ in the configuration:

```properties theme={null}
# need replace SASL_PLAINTEXT "inter_broker_user" and "interbroker-password-placeholder"
autobalancer.client.auth.sasl.mechanism=PLAIN
autobalancer.client.auth.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="${inter_broker_user}" password="${interbroker-password-placeholder}" user_inter_broker_user="${interbroker-password-placeholder}";
autobalancer.client.auth.security.protocol=SASL_PLAINTEXT
autobalancer.client.listener.name=INTERNAL
```

### AutoMQ-specific Env

Configure and use the `AK/SK`  with read and write permissions to access your Bucket:

```yaml theme={null}
- name: "AWS_ACCESS_KEY_ID"
  value: "yourAccessKey"
- name: "AWS_SECRET_ACCESS_KEY"
  value: "yourSecretKey"
```

> The actual AK/SK can be stored in Kubernetes Secrets and referenced to ensure the security of AK/SK

### No configuration required

AutoMQ, which builds its core architecture on object storage, has fundamental differences from traditional Kafka: object storage itself provides high durability and redundancy capabilities. Therefore, some replica-related parameters in traditional Kafka that are used to ensure data reliability, the following replica-related configurations are no longer needed or have no effect in AutoMQ:

```properties theme={null}
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
default.replication.factor=1
min.insync.replicas=1
```

### Resource request

Due to the difference in architecture, AutoMQ relies more on object storage compared to Kafka. In contrast, AutoMQ no longer requires local Data Log disk storage

Therefore, when only  `  Metadata PV  `  is retained for the Controller, all other  `  PVCs  `  can be removed.

### Observability

AutoMQ provides a complete set of methods for building an observable system. For details, see:

[AutoMQ Observability](https://docs.automq.com/automq/observability/overview)

# Deployment Recommendations

When using your custom Helm Chart or Operator, it is best to first try using [ AutoMQ Labs ](https://hub.docker.com/r/apache/kafka#overriding-the-default-broker-configuration) for deployment experience, which demonstrates the unified configuration pass-through method by setting Env and reusing the Apache Kafka Docker Image. This will help you better use your custom Kubernetes application lifecycle management tool to deploy AutoMQ.

### Image

Specify the AutoMQ Docker Image for your main [container](https://hub.docker.com/r/automqinc/automq/tags?name=-kafka)

Reference

```text theme={null}
automqinc/automq:1.6.0-rc0-kafka
```

### Config & Env

We recommend uniformly using Env to set the above-mentioned specific dependency configuration for AutoMQ (the way to pass in the configuration of the Apache Kafka Docker Image), and `using KAFKA_` as the prefix to set the key-value pairs that we ultimately want to pass through to server.properties.

```yaml theme={null}
- name: "KAFKA_PROCESS_ROLES"
  value: "controller,broker"
- name: "KAFKA_S3_DATA_BUCKETS"
  value: "0@s3://${data-bucket}?region=${region}&endpoint=${endpoint}"
- name: "KAFKA_S3_WAL_PATH"
  value: "0@s3://${data-bucket}?region=${region}&endpoint=${endpoint}"
- name: "KAFKA_S3_OPS_BUCKETS"
  value: "1@s3://${ops-bucket}?region=${region}&endpoint=${endpoint}"
- name: "KAFKA_ELASTICSTREAM_ENABLE"
  value: "true"
```

and the Env actually needed

```yaml theme={null}
- name: "AWS_ACCESS_KEY_ID"
  value: "yourAccessKey"
- name: "AWS_SECRET_ACCESS_KEY"
  value: "yourSecretKey"
```

## Other production-level recommendations

### Production Deployment

#### Server Resource Configuration

We recommend that each Pod for resource deployment of AutoMQ be deployed exclusively and run on resources of 2Core16GB or higher, and appropriately adjust the  `requests`  and  `limits`  of the AutoMQ server (Statefulset) in Kubernetes.

<Tip>
  Note: The specific resource configuration method depends on the capabilities and entry points provided by your Kubernetes application lifecycle management tool
</Tip>

#### Scheduling Policy

Through `  nodeAffinities  ` ,  `  tolerations  ` ,  `  podAntiAffinities  `  strategies, AutoMQ can implement fine-grained scheduling strategies in Kubernetes. We recommend that production-level AutoMQ be exclusive and non-mixed deployment.

<Tip>
  Note: The specific way to set up scheduling depends on the capabilities and entry points provided by your Kubernetes application lifecycle management tool.
</Tip>

#### Replicas

Through the Kafka cluster of your Kubernetes application lifecycle management tool, it supports modifying configurations such as  `  replicas  `  to achieve the purpose of horizontal volume expansion and contraction.

<Danger>
  Note: However, we recommend that the cluster be deployed with 3 Controller Pods and several Broker Pods by default, and do not recommend dynamically adjusting the replicas of the Controller to avoid unexpected risks.
</Danger>

#### AutoScaling (HPA)

Use your Kubernetes application lifecycle management tool to set and enable  `HPA ` for dynamic volume expansion and contraction.

<Danger>
  Note: We also do not recommend configuring Controller HPA because the Controller in Kafka KRaft mode relies on the Raft protocol to maintain metadata consistency and does not support automated Raft member changes. Therefore, configuring Controller HPA may cause Quorum failure or cluster unavailability.
</Danger>

### Performance optimization

AutoMQ also supports setting additional properties parameters and ENV for performance optimization. For details, see: [Performance Tuning for Broker](/automq/deployment/performance-tuning-for-broker)


# Deploy Multi-Nodes Cluster on Kubernetes with Strimzi
Source: https://docs.automq.com/automq/deployment/multi-nodes-cluster-with-strimzi

Deploy a multi-node AutoMQ cluster on Kubernetes with Strimzi. Covers Kafka-compatible features, scheduling, scaling, and a free 30-day AWS trial.

This topic describes how to use Strimzi to deploy a multi-node AutoMQ cluster, where users can verify cluster-related features such as partition migration and automatic data balancing in this Development Environment.

In addition to the Strimzi-based Kubernetes deployment solution, users can refer to the following documents to experience other deployment solutions:

* [Deploy Multi-Nodes Cluster on Kubernetes](https://docs.automq.com/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes)

* [Deploy Multi-Nodes Cluster on Linux](https://docs.automq.com/automq/deployment/deploy-multi-nodes-cluster-on-linux)

<Tip>
  Deploying AutoMQ for production environment workloads and tuning its parameters are relatively complex. You can contact the AutoMQ team via [the form here](https://www.automq.com/contact) to obtain the necessary assistance and best practices.
</Tip>

Additionally, if you wish to completely avoid the workload of installation and deployment, you can experience the full-service Cloud as a Service provided by the AutoMQ team via the following link. Currently, all cloud marketplaces offer a free 30-day trial experience.

* [Experience AutoMQ Cloud for free from AWS Marketplace](https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws).

## Prerequisites

This document example is used to deploy a 4-node AutoMQ cluster, where 3 nodes run Controller ( Controller and Broker), and the remaining 1 node runs only Broker.

The following conditions need to be checked in advance:

* Prepare a Kubernetes cluster, ensure it has at least 4 Nodes, and it is recommended to use network-optimized virtual machines with 2 cores and 16GB of memory, on which Pod creation and other operations will be performed later.

* Ensure that the Helm Chart version is v3.8.0 or higher. Refer to [Helm Chart Quick Start](https://helm.sh/docs/intro/quickstart/).

* Prepare 2 Object Storage Buckets, one for storing message data and one for storing system logs and Metric data.

> Strimzi provides multiple types of Operators. Unless otherwise specified below, all references refer to the Cluster Operator.

## Deploy Strimzi Operator

### Step 1:  Edit the configuration file

Create an empty  `strimzi-values.yaml`  file, which is recommended to be consistent with the example configuration [strimzi-values.yaml](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/setup/kubernetes/strimzi/strimzi-values.yaml) , and then add additional parameters as needed.

Strimzi supports multiple Kafka-related components, and AutoMQ is mainly adapted to the  `KAFKA ` component, and supports only version 3.9.0.

### Step 2: Install the Strimzi Operator

Use the configuration file just created and install the Strimzi Operator via `helm install` .

* Specify  `version`  as 0.47 to ensure stability and avoid unnecessary issues.

```bash theme={null}

helm install automq-strimzi-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator \
  --version 0.47.0 \
  --namespace automq \
  --create-namespace \
  --values strimzi-values.yaml

```

Waiting for Operator to be ready

```bash theme={null}

kubectl --namespace automq rollout status deployment strimzi-cluster-operator --watch

```

When the Operator is ready, it should output something similar to the following:

```text theme={null}

deployment "strimzi-cluster-operator" successfully rolled out

```

View Pod List:

```text theme={null}

kubectl get pods -n automq -w       
                                                            
NAME                                       READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-8dfd94b6d-nb9z6   1/1     Running   0          82s

```

## Deploy AutoMQ Cluster

### Step 1: Edit the configuration file

Create an empty `automq-demo.yaml` file, edit the file, and add specific parameters. You can refer to the recommended configuration examples for different scenarios under [automq-demo.yaml](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/setup/kubernetes/strimzi/automq-demo.yaml) and its subdirectories. For more details, refer to [README.md](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/setup/kubernetes/strimzi/README.md).

* Replace  `${ops-bucket}` ,  `${data-bucket}` ,  `${region}` ,  `${endpoint}`  with the specific values of Object Storage. For details, see [Object Storage Configuration](https://docs.automq.com/automq/configuration/object-storage-configuration) .

* Replace  `${access-key}`  and  `${secret-key}`  with actual values. You can also choose other authorization methods such as IAM Role.

* Multi-Available Zone deployment can use the `topologySpreadConstraints` parameter to ensure that Pods are evenly distributed across specified Available Zones.

```yaml theme={null}

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              strimzi.io/pool-name: controller or broker

```

* Avoid cross-Available Zone traffic. The following configuration will obtain the topology.kubernetes.io/zone of the current node label, ultimately achieving the effect of eliminating cross-Available Zone traffic costs.

```yaml theme={null}

spec:
  kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone

```

* Disable the shrinkage check. Enabling this configuration allows automq to quickly complete shrinkage, and details can be found in the volume expansion and contraction section below.

```yaml theme={null}

metadata:
  name: my-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
    strimzi.io/skip-broker-scaledown-check: "true"

```

* It is recommended to deploy AutoMQ exclusive nodes at the production level (to avoid competing for resources such as internet bandwidth with other Pods), and it is advisable to match them through labels for node affinity and tolerations.

* Other server level parameters can be set as needed in  `spec.kafka.config` . For details, refer to: [Broker and Controller Configuration](https://docs.automq.com/automq/configuration/broker-and-controller-configuration).

### Step 2: Install AutoMQ

Install AutoMQ using the custom `automq-demo.yaml` file

```bash theme={null}

kubectl apply -f automq-demo.yaml -n automq

```

Waiting for AutoMQ cluster to be ready:

```bash theme={null}

kubectl --namespace automq get kafka my-cluster --watch

```

When the AutoMQ cluster is ready, the output should be similar to the following:

```text theme={null}

NAME         READY   METADATA STATE   WARNINGS
my-cluster   True    KRaft            True

```

View Pod List:

```text theme={null}

kubectl get pods -n automq -w   
NAME                                          READY   STATUS    RESTARTS   AGE
my-cluster-broker-1000                        1/1     Running   0          9m28s
my-cluster-controller-0                       1/1     Running   0          9m28s
my-cluster-controller-1                       1/1     Running   0          9m28s
my-cluster-controller-2                       1/1     Running   0          9m28s
my-cluster-entity-operator-6948bd66c4-4962d   2/2     Running   0          103s
strimzi-cluster-operator-678cc9fbc8-x46z4     1/1     Running   0          11m

```

## Test message sending and receiving

* When accessing the Kafka cluster within the k8s cluster, the format of its server address is:

```text theme={null}

<kafka_cluster_name>-kafka-bootstrap

```

For example, in our example, the access address is: `my-cluster-kafka-bootstrap:9092`

> For details, see: [Strimzi ref list of kafka cluster resources str](https://strimzi.io/docs/operators/latest/deploying#ref-list-of-kafka-cluster-resources-str)

Next, you can perform Topic message sending and consumption tests based on `kafka-console-producer.sh` , `kafka-console-consumer.sh` .

* Start the producer and test sending messages (the topic will be automatically created by default):

```bash theme={null}

kubectl -n automq run automq-producer -ti --image=automqinc/automq:1.5.5 --rm=true --restart=Never -- /opt/kafka/kafka/bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
If you don't see a command prompt, try pressing enter.
>Hi, this is AutoMQ!

```

* Start the consumer and test receiving messages:

```bash theme={null}

kubectl -n automq run kafka-consumer -ti --image=automqinc/automq:1.5.5 --rm=true --restart=Never -- /opt/kafka/kafka/bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
Hi, this is AutoMQ!

```

## Uninstall

* After completing the test, you can stop and uninstall the Strimzi Operator via  `  helm uninstall  ` .

```bash theme={null}

helm uninstall automq-strimzi-operator --namespace automq

```

* Stop and uninstall automq

```bash theme={null}

kubectl delete -f automq-demo.yaml -n automq

```

* If historical data is no longer needed, the PVC and Bucket data of the cluster must be deleted together to prevent dirty data from affecting the next deployment.

## Precautions for the production environment

### Fix the Strimzi Operator version

To avoid unexpected changes during deployment, it is recommended to fix the Strimzi Operator version to `0.47` . This helps:

* **Ensure Compatibility** : Ensure that the behavior of the deployed application is consistent with that in testing and is not affected by the release of new versions.

* **Avoid accidental updates** : Prevent automatic updates from introducing changes that are incompatible with current deployment or operational practices.

### Namespace Management

The Strimzi Operator is only responsible for managing AutoMQ clusters created within **the namespace where it resides**  (e.g., automq).

The Operators of Apache Kafka and AutoMQ should be deployed in different Namespaces. To avoid conflicts, it is recommended to create a new Namespace when using AutoMQ,  and completely delete this Namespace after use .

### Docker mirroring

The Strimzi Operator uses the Kafka 3.9 version mirroring address by default as  `  quay.io/strimzi/kafka:latest-kafka-3.9.0  ` , which needs to be modified to the AutoMQ custom mirroring and specific version:

```yaml theme={null}

  - name: STRIMZI_KAFKA_IMAGES
    value: |
      3.9.0=automqinc/automq:1.6.0-rc0-strimzi
      3.9.1=quay.io/strimzi/kafka:0.47.0-kafka-3.9.1
      4.0.0=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0

```

### Storage Class

In this example, we are using AWS's gp2 storage type. If you are using non-AWS or have different requirements, you can make adjustments based on the actual situation.

```yaml theme={null}

storage:
  type: jbod
  volumes:
    - id: 0
      type: persistent-claim
      size: 20Gi
      kraftMetadata: shared
      deleteClaim: false
      class: gp2

```

### Autobalancer Configuration

The  `  autobalancer  `  in PLAINTEXT mode must explicitly configure parameters in the following format:  `  listener.name - port  ` .

```yaml theme={null}

spec:
  kafka:
    config:
        autobalancer.client.listener.name: PLAIN1-9092

```

### Scheduling Policy

Through node affinity (nodeAffinities) and toleration (tolerations), fine-grained scheduling policies can be implemented for AutoMQ in Kubernetes. We recommend that production-level AutoMQ be exclusive and non-mixed workload. It is recommended to customize label matching rules based on node types:

#### Tolerance

It is recommended to add taints to the Kubernetes node group  `  key: "dedicated", operator: "Equal", value: "automq", effect: "NoSchedule"  ` .

```yaml theme={null}

  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "automq"
      effect: "NoSchedule"

```

#### Node Affinity

Override the default values in the controller/broker configuration to match node labels (such as  `node-type: r6in.large` )

```yaml theme={null}

    nodeAffinity:  
      requiredDuringSchedulingIgnoredDuringExecution:  
        nodeSelectorTerms:  
        - matchExpressions:  
          - key: "node-type"  
            operator: In  
            values: ["r6in.large"]  

```

#### Pod Anti-Affinity

Ensure that the controller component and the broker component are not scheduled to the same node, using the  `podAntiAffinity`  parameter:

```yaml theme={null}

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: strimzi.io/component-type
                operator: In
                values:
                  - kafka
          topologyKey: kubernetes.io/hostname

```

### Volume Expansion and Contraction

#### Controller

The number of Controller instances is specified through the  `controller.replicas`  configuration item, and does not support horizontal volume expansion and contraction. It is recommended to configure 3 instances, but it can be adjusted according to actual needs.

<Danger>
  Note: After the Clustered Deployment is completed, adjusting the Replicas of the Controller is not supported to avoid unexpected risks.
</Danger>

#### Broker

The number of instances is configured through  `  broker.replicas  ` , supporting horizontal volume expansion and contraction.

Note that the following parameters must be configured during volume contraction:

```yaml theme={null}

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  annotations:
    strimzi.io/skip-broker-scaledown-check: "true" 

```

When there is Partition data on the Broker to be deleted, Strimzi will default to preventing volume contraction. For traditional Kafka, Cruise Control can be used to complete data migration, ensuring that the Broker has no data before performing volume contraction.

Strimzi also provides the parameter `skip-broker-scaledown-check` to skip this check.

Since AutoMQ adopts **the Shared Storage architecture** , this parameter can be enabled **without**  the risk of data loss.

> For details, see: [Strimzi prevent broker scale down if containing partition replicas](https://strimzi.io/blog/2024/01/03/prevent-broker-scale-down-if-containing-paritition-replicas/)

### Resource Allocation

Each Pod of AutoMQ is recommended to run on resources of 2Core16GB. Adjust the resource parameters through the following configuration:

```yaml theme={null}

spec:
  replicas: 3
  roles:
    - controller
    - broker
  resources:
    requests:
      cpu: 1000m
      memory: 12Gi
    limits:
      cpu: 2000m
      memory: 16Gi
  template:
    kafkaContainer:
      env:
        - name: "KAFKA_HEAP_OPTS"
          value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"

```

```yaml theme={null}

spec:
  replicas: 1
  roles:
    - broker
  resources:
    requests:
      cpu: 1000m
      memory: 12Gi
    limits:
      cpu: 2000m
      memory: 16Gi
  template:
    kafkaContainer:
      env:
        - name: "KAFKA_HEAP_OPTS"
          value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"

```

### **Monitoring**

It mainly involves the Prometheus-related configuration in Strimzi. For details, refer to the [Using Prometheus in Strimzi](https://strimzi.io/docs/operators/in-development/deploying#assembly-metrics-setup-str) section.


# Overview
Source: https://docs.automq.com/automq/deployment/overview

AutoMQ offers cloud-native scalability and 100% Kafka compatibility on AWS, Azure, GCP, and more. It supports both public and private clouds.

## Deployment Architecture

AutoMQ is a Kafka-compatible streaming platform offering storage-compute separation powered by object storage. It is designed to operate on any public cloud or private data center that supports the S3 protocol (requiring S3-compatible storage software). AutoMQ can be installed directly on Linux hosts or deployed on Kubernetes using a Helm Chart or other deployment methods.

<img />

### Related Content

* K8S Deployment: [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes)

* Linux Host Deployment: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux)

* Common Server Performance Tuning: [Performance Tuning for Broker▸](/automq/deployment/performance-tuning-for-broker)

* Performance Benchmarking: [AutoMQ Benchmark▸](/automq/deployment/automq-benchmark)


# Performance Tuning for Broker
Source: https://docs.automq.com/automq/deployment/performance-tuning-for-broker

Optimize AutoMQ's broker performance with memory and network tuning, leveraging cloud-native scalability and Kafka compatibility for efficient resource allocation.

AutoMQ enhances server performance through methods like read/write efficiency optimization and resource isolation, ensuring stable and efficient operation.

<Tip>
  Users should adjust parameters based on their hardware environment and business scenarios.
</Tip>

## Memory-related Parameters

### s3.stream.allocator.policy

Configure the memory allocation strategy for AutoMQ. It is advisable to use `POOLED_DIRECT` to alleviate GC pressure.

### s3.wal.cache.size

Manage the size of the Log Cache, which influences the maximum size of cached data during Tail Read (hot reads).

### s3.block.cache.size

Control the size of the Block Cache; it influences the maximum size of data cached during Catch-up Read (catch-up read, cold read). In Catch-up Read scenarios, consider appropriately increasing this value.

### s3.wal.upload.threshold

Manage the amount of data AutoMQ accumulates in the Log Cache before triggering an S3 upload. It is advisable not to exceed 1/3 of `s3.wal.cache.size`.

### Recommended Parameter Values

When the memory allocation strategy is `POOLED_DIRECT`, the sum of `s3.wal.cache.size` and `s3.block.cache.size` should be about 1/2 of `MaxDirectMemorySize` (if `MaxDirectMemorySize` is large, it can be slightly increased).

| -<br />                          | Tiny<br />           | Small<br />          | Medium<br />         | Large<br />          |
| -------------------------------- | -------------------- | -------------------- | -------------------- | -------------------- |
| Physical Memory<br />            | 4 GiB<br />          | 8 GiB<br />          | 16 GiB<br />         | 32 GiB<br />         |
| s3.stream.allocator.policy<br /> | POOLED\_DIRECT<br /> | POOLED\_DIRECT<br /> | POOLED\_DIRECT<br /> | POOLED\_DIRECT<br /> |
| -XX:MaxHeapSize<br />            | 1g<br />             | 2g<br />             | 6g<br />             | 12g<br />            |
| -XX:MaxDirectMemorySize<br />    | 1500m<br />          | 3g<br />             | 6g<br />             | 14g<br />            |
| s3.wal.cache.size<br />          | 524,288,000<br />    | 1,073,741,824<br />  | 2,147,483,648<br />  | 6,442,450,944<br />  |
| s3.block.cache.size<br />        | 104,857,600<br />    | 536,870,912<br />    | 1,073,741,824<br />  | 4,294,967,296<br />  |
| s3.wal.upload.threshold<br />    | 62,914,560<br />     | 357,913,941<br />    | 536,870,912<br />    | 536,870,912<br />    |

## Network-related Parameters

### s3.network.baseline.bandwidth

The baseline for assessing local network bandwidth should be the network's stable capability to achieve both outbound and inbound bandwidth.

Since the server internally performs operations such as writing the WAL, S3, and Compaction, each data write from a client translates into three network writes by the server. By considering the business's read/write ratio, you can ultimately determine the maximum write load a single machine can manage.


# Broker Configuration
Source: https://docs.automq.com/automq/eliminate-inter-zone-traffics/broker-configuration

Optimize AutoMQ's inter-zone routing and rack configurations to minimize costs and enhance performance. Benefit from cloud-native scalability and 100% Kafka compatibility.

## Enable Inter-Zone Routing

In clouds that charge fees for Inter-Zone traffic (such as AWS), configure `automq.zonerouter.channels` to enable Inter-Zone routing functionality. Once this feature is enabled, clients will connect only to AutoMQ nodes within the same zone for Produce and Consume operations. For example, in the diagram below, assume that a client in AZ2 wants to consume a partition located in AZ1:

* For Producers: AutoMQ in AZ2 will act as a proxy node for AutoMQ in AZ1, forwarding the Produce request via the Rack-Aware Router to AutoMQ in AZ1.

* For Consumers: AutoMQ in AZ2 will directly read the already written data from S3 and return it to the client.

<img />

The format for configuring `automq.zonerouter.channels` is as follows:

```properties theme={null}
automq.zonerouter.channels=0@s3://$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey][&batchInterval=250][&maxBytesInBatch=8388608]
```

Configuration consists of two parts:

* Channel connection details: `$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey]`bash
* Channel Batching Settings: By default, the Proxy node batches Produce requests every 250ms (
  `batchInterval=250`) or when 8MiB (`maxBytesInBatch=8388608`) of data is accumulated, routing them to the Main node. Increasing the configured value reduces the cost of S3 API calls but increases Inter-Zone write latency. Conversely, setting a smaller value increases S3 API call costs but decreases Inter-Zone write latency.

## Configuration Mapping Rules to Identify the Originating CIDR of the Client in the Rack

Due to the limitations of the Kafka protocol, the server cannot detect the specific rack a client belongs to. To address this issue and prevent cross-availability zone production and consumption, AutoMQ offers [Client Configuration▸](/automq/eliminate-inter-zone-traffics/client-configuration) as well as server-based rack configuration options. This section introduces a server-based dynamic configuration method, `automq.zone.cidr.blocks`.

<Tip>
  In a cloud environment, racks are typically defined by [ZoneId](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html). The concepts of rack and zone mentioned later can be used interchangeably.
</Tip>

The configuration format for `automq.zone.cidr.blocks` is as follows:

```text theme={null}

# Ex. use1-az1@10.0.0.0/19,10.0.32.0/19<>use1-az2@10.0.64.0/19<>use1-az3@10.0.96.0/19

$zone1@$CIDRBlock1,$CIDRBlock2<>$zone2@$CIDRBlock3<>$zone3@$CIDRBlock4,$CIDRBlock5

```

* Configurations for different zones are separated by `< >`.

* Each zone's configuration starts with the zone identifier, followed by `@` as a delimiter, and then the current zone's CIDR list. CIDRs in the list are separated by `,`.

This configuration can be dynamically changed using `kafka-config.sh`. A sample command is as follows:

```bash theme={null}
./bin/kafka-configs.sh --broker-defaults --add-config automq.zone.cidr.blocks=$config_value --alter  --bootstrap-server $bootstrap_server
```

AutoMQ determines the Zone each Client belongs to by matching the Client's IP address with CIDR. The server-side configuration `automq.zone.cidr.blocks` is ideal for situations where:

* The Zones where Clients reside have distinct CIDR ranges.

* Computational engines such as Flink, Spark, Presto, etc., face challenges in setting different Client configurations based on the Zone handling the load.

If both server-side and client-side configurations are set, the priority order is: `client.id`  >  `automq.zone.cidr.blocks`  > Consumer `client.rack`.


# Client Configuration
Source: https://docs.automq.com/automq/eliminate-inter-zone-traffics/client-configuration

Optimize cloud-native data flow with AutoMQ using zone identification for seamless Kafka-compatible scalability and efficient data fetching based on client zone ID.

## Setting Zone Identification

In addition to identifying client zones through server-side configuration rules [Broker Configuration▸](/automq/eliminate-inter-zone-traffics/broker-configuration), AutoMQ also supports clients specifying zone information. This can be achieved by appending `automq_az=$zoneId` to `client.id`, indicating the zone to which the client belongs. The `client.id` format is as follows:

```properties theme={null}
client.id=automq_az=$zoneId&existing-client-id
```

<Tip>
  On AWS, it is recommended to use [zoneId](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html) instead of zoneName, as zoneId is consistent across all accounts.
</Tip>

## Follower Fetching

Consumers can set zone identification via `client.id`. Additionally, for clients with versions greater than 2.4.0, the [KIP-329 fetch from closest replicas](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica) feature can be utilized to configure zone affinity for consumption. The configuration method is as follows:

```properties theme={null}
client.rack=$zoneId
```


# Monitor Inter Zone Traffic
Source: https://docs.automq.com/automq/eliminate-inter-zone-traffics/monitor-inter-zone-traffic

Monitor AutoMQ's inter-zone traffic with tools like iftop on Linux and AWS CloudWatch, optimizing transfers in a scalable, Kafka-compatible cloud service.

After setting up Inter-Zone traffic optimization, users can track local network traffic to pinpoint actual Inter-Zone transfers by correlating with the accessed brokers.

## Command Tool for Monitoring Cross-AZ Traffic

1. Monitor network traffic: On Linux, you can use `iftop` to view the inbound and outbound traffic between your machine and other IPs. The command is as follows:

```bash theme={null}

iftop  -t -s 60 -L 100 -n

```

2. Identify node availability zones: Query the cluster nodes of AutoMQ and the zones to which the nodes belong. The command is as follows:

```bash theme={null}
./bin/kafka-broker-api-versions.sh --bootstrap-server $bootstrap_server | grep id

```

Match both outputs to see if Inter-Zone traffic is generated.

## Other

If your application is deployed on AWS, you can also monitor cross-zone traffic using CloudWatch's [Network Flow Monitor](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-NetworkFlowMonitor.html) feature.


# Overview
Source: https://docs.automq.com/automq/eliminate-inter-zone-traffics/overview

AutoMQ reduces Inter-Zone costs with S3-based storage, eliminating Kafka's data transfer traffic across zones while ensuring scalability and cost efficiency.

AutoMQ's S3-based shared storage architecture can help minimize inter-broker and Inter-Zone (AZ) replica replication, as well as client Inter-Zone data read and write traffic. This document outlines how to use AutoMQ to reduce Inter-Zone transmission traffic.

<Info>
  In this document, AZ stands for Availability Zone, specifically referring to a zone provided by cloud service providers in a public cloud environment. Each independent AZ may be supported by a separate data center facility.
</Info>

## Problem Background

### Apache Kafka Inter-Zone Traffic Analysis

When users deploy Apache Kafka across multiple AZs in AWS or GCP cloud environments, enabling multi-AZ service disaster recovery also introduces Inter-Zone data transmission costs. In large-scale clusters, these traffic costs can account for **60-70%**.

* **Inter-Zone Production Traffic**: If the Producer has not specified a ShardingKey, and partitions are distributed equally across cluster nodes, at least two-thirds of the Producer's traffic will traverse across AZs. For instance, Producers located in AZ1 will route one-third of their traffic to both AZ2 and AZ3.

* **Server-Side Inter-Zone Replication Traffic**: After messages reach the Kafka Broker, it ensures high reliability by replicating the data to Brokers situated in different AZs, effectively doubling the Inter-Zone traffic generated by the Produce operation.

* **Inter-Zone Consumption Traffic**: Consumers have the option to configure client.rack to consume partitions/replicas within the same AZ, thus preventing the generation of Inter-Zone traffic.

<img />

### AutoMQ Eliminates Inter-Zone Data Transfer

AutoMQ utilizes an architecture that separates storage and compute by leveraging S3, resulting in zero Inter-Zone traffic costs based on S3:

* Production Traffic: AutoMQ's built-in Proxy layer intercepts and identifies Produce requests that cross Availability Zones (AZs). It establishes Inter-Zone proxy channels based on S3, forwarding these Produce requests to the actual partition Leader. Producers only need to communicate with Broker nodes within the same AZ, thus avoiding Inter-Zone traffic costs.

* Replication Traffic: AutoMQ leverages S3 as a shared storage layer, with S3 utilizing erasure coding to create multiple replicas across various AZs. Consequently, no Apache Kafka replication traffic occurs between AutoMQ Broker nodes.

* Consumption Traffic: Besides the Leader partition, AutoMQ establishes a read-only partition in each additional AZ. These read-only partitions directly access data written by the Leader partition from S3 on-demand. Therefore, consumers only need to communicate with the Leader partition or a read-only partition within the same AZ, avoiding any Inter-Zone traffic costs.

<img />

## Configuration Documentation

To eliminate Inter-Zone data transfer using AutoMQ's Inter-Zone data routing feature, configuration must be enabled on both the server and client sides. The relevant configuration documentation is as follows:

* [Broker Configuration▸](/automq/eliminate-inter-zone-traffics/broker-configuration)

* [Client Configuration▸](/automq/eliminate-inter-zone-traffics/client-configuration)

After configuration is complete, you can conduct data sending and receiving tests, and refer to the [Monitor Inter Zone Traffic▸](/automq/eliminate-inter-zone-traffics/monitor-inter-zone-traffic) document to observe the actual inter-availability zone traffic generated.


# Deploy Multi-Nodes Test Cluster on Docker
Source: https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker

Deploy a scalable AutoMQ test cluster on Docker to evaluate cloud-native message streaming with full Kafka compatibility and feature exploration.

This document explains how to deploy a three-node AutoMQ cluster in a local development environment (by deploying MinIO to provide object storage services). In this setup, users can test cluster-related features such as partition reassignment and data auto-balancing.

Besides the Docker deployment option, users can reference the following documentation for alternative deployment methods to verify cluster-related features:

* [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes)

* [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux)

<Tip>
  Deploying AutoMQ and tuning parameters for production environment loads can be complex. For necessary assistance and best practices, you can contact the AutoMQ team through [this form](https://www.automq.com/contact).
</Tip>

Additionally, if you'd like to completely bypass the installation and deployment process, you can try the fully managed cloud service provided by the AutoMQ team through the link below. A free 30-day trial is available on AWS Marketplace.

* [Experience AutoMQ Cloud for Free from AWS Marketplace](https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws).

## Prerequisites

* Linux/Mac/Windows Subsystem for Linux

* Docker

* Docker Compose version > 2.22.0

* At least 8GB of available memory

## Install and Launch a Three-node AutoMQ Cluster

Use the following command to download the Docker Compose configuration file for the single-node deployment plan and start the single-node AutoMQ cluster using Docker Compose.

```bash theme={null}
curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml up -d
```

After a successful startup, the cluster address will be exposed under the Docker network named automq\_net, and the cluster's bootstrap address will be "**server1:9093,server2:9093,server3:9093**." You can connect to the cluster using this address subsequently.

<img />

<Tip>
  **Note:**

  In the above test environment, AutoMQ utilizes a Docker network named automq\_net. To connect to the cluster, you need to configure the *--network automq\_net* option to use the same Docker network for the connection.
</Tip>

## Test Message Sending and Receiving.

After installing and starting the AutoMQ cluster, you can test message sending, consumption, and other functionalities by running the following commands.

1. Execute the following command to create a Topic using the Kafka CLI tool `kafka-topics.sh` within a Docker environment.

```bash theme={null}
docker run --network automq_net automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"

```

2. Execute the following command to send test messages using the Kafka CLI tool `kafka-console-producer.sh` within a Docker environment.

```bash theme={null}
docker run -it --network automq_net automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"
```

3. Execute the following command to consume test messages using the Kafka CLI tool `kafka-console-consumer.sh` within a Docker environment.

```bash theme={null}
docker run --network automq_net automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server server1:9092,server2:9092,server3:9092"
```

## Explore the Features of the AutoMQ Cluster.

In addition to core message production and consumption features, AutoMQ offers advanced capabilities like second-level partition reassignment and automatic data rebalancing, which address Apache Kafka's challenges with slow reassignment and scaling. You can refer to the following documentation for testing and validation:

* [Example: Partition Reassignment in Seconds▸](/automq/getting-started/explore-automq/example-partition-reassignment-in-seconds)

* [Example: Continuous Data Self-Balancing▸](/automq/getting-started/explore-automq/example-continuous-data-self-balancing)

* [Example: Self-Balancing When Cluster Nodes Change▸](/automq/getting-started/explore-automq/example-self-balancing-when-cluster-nodes-change)

## Stop and Remove a Three-node AutoMQ Cluster.

After completing the functional test, please run the following command to stop and uninstall the current AutoMQ cluster.

```bash theme={null}
curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml down -v
```


# Example: Continuous Data Self-Balancing
Source: https://docs.automq.com/automq/getting-started/explore-automq/example-continuous-data-self-balancing

Experience the power of AutoMQ's continuous data rebalance for seamless load distribution and enhanced cloud-native scalability, all while maintaining Kafka compatibility.

This document explains the data auto-rebalance testing for the AutoMQ cluster using the Kafka CLI tool. The Kafka CLI tool is executed through the Docker image provided by AutoMQ.

1. Create a topic with multiple partitions and manually reassign partitions to specific nodes to induce an imbalance in partition distribution.

2. Subsequently, send a balanced load to all partitions and observe if the partitions automatically reassign between different brokers.

This automatic data balancing is a fundamental feature of AutoMQ, ensuring automatic equitable distribution of data across the cluster. By monitoring the distribution of partitions and the load on brokers, you can verify whether the automatic partition balancing functions as expected.

### Prerequisites

Before conducting automatic partition data rebalance tests, the following conditions must be met:

**Complete the installation and deployment of the AutoMQ cluster**, you can refer to the following methods for installing and deploying AutoMQ:

* [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

<Tip>
  If deploying the cluster via [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) or [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes), be sure to set autobalancer.controller.enable to true when starting the Controller to activate automatic data rebalancing.
</Tip>

**Additionally, the host running the test program** needs to meet the following criteria:

* Linux/Mac/Windows Subsystem for Linux

* Docker

### Experience: Continuous Data Rebalance

If the previous AutoMQ cluster was deployed by following [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker), then the obtained cluster Bootstrap address will be similar to "**server1:9092,server2:9092,server3:9092**", and the AutoMQ cluster will be located within the "**automq\_net**" Docker network.

<Tip>
  Please substitute the bootstrap-server address below with the actual cluster address as per the deployment configuration.
</Tip>

#### Create Topic

```bash theme={null}
docker run  --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --partitions 8 --create --topic continuous-self-balancing-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

```

#### View Partition Distribution

```bash theme={null}
docker run  --network automq_net   automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic continuous-self-balancing-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"
```

```yaml theme={null}
Topic: continuous-self-balancing-topic TopicId: DNZe6gBQTrCOEAruQ_y2tg PartitionCount: 8       ReplicationFactor: 1    Configs: min.insync.replicas=1,segment.bytes=1073741824
    Topic: continuous-self-balancing-topic Partition: 0    Leader: 2   Replicas: 2     Isr: 2
    Topic: continuous-self-balancing-topic Partition: 1    Leader: 1   Replicas: 1     Isr: 1
    Topic: continuous-self-balancing-topic Partition: 2    Leader: 1   Replicas: 1     Isr: 1
    Topic: continuous-self-balancing-topic Partition: 3    Leader: 2   Replicas: 2     Isr: 2
    Topic: continuous-self-balancing-topic Partition: 4    Leader: 1   Replicas: 1     Isr: 1
    Topic: continuous-self-balancing-topic Partition: 5    Leader: 2   Replicas: 2     Isr: 2
    Topic: continuous-self-balancing-topic Partition: 6    Leader: 1   Replicas: 1     Isr: 1
    Topic: continuous-self-balancing-topic Partition: 7    Leader: 2   Replicas: 2     Isr: 2
```

#### Manually Reassign Partitions

To facilitate the observation of continuous data rebalancing, we manually reassign partitions to node2.

1. Create a plan for partition reassignment.

```bash theme={null}
echo '{
    "partitions": [
        {"topic": "continuous-self-balancing-topic", "partition": 0, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 1, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 2, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 3, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 4, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 5, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 6, "replicas": [2]},
        {"topic": "continuous-self-balancing-topic", "partition": 7, "replicas": [2]}
    ],
    "version": 1
}' > move.json

```

2. Implement the partition reassignment plan.

```bash theme={null}
docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --execute"
```

3. After the manual reassignment, view the partition distribution as follows:

```text theme={null}

Topic: continuous-self-balancing-topic TopicId: HtVB3bM7TYaNKKKmm7khQw PartitionCount: 8   ReplicationFactor: 1    Configs: min.insync.replicas=1,segment.bytes=1073741824
    Topic: continuous-self-balancing-topic Partition: 0    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 1    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 2    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 4    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 5    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 6    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 7    Leader: 2   Replicas: 2 Isr: 2
```

#### Start the Producer.

```bash theme={null}

docker run  --network automq_net  automqinc/automq:1.5.5 /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic continuous-self-balancing-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

```

#### Start the Consumer.

```bash theme={null}

docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-consumer-perf-test.sh --topic continuous-self-balancing-topic --show-detailed-stats --timeout 300000 --messages=1024000 --reporting-interval 1000 --bootstrap-server=server1:9092,server2:9092,server3:9092"

```

#### Rechecking Partition Distribution

After some time, you'll notice the producer generating the following logs.

```text theme={null}

[2024-05-16 10:29:50,448] 25622 records sent, 5123.4 records/sec (5.00 MB/sec), 15.7 ms avg latency, 41.0 ms max latency.
[2024-05-16 10:30:00,372] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10354 on topic-partition continuous-self-balancing-topic-7, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,373] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-7 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,373] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10354 on topic-partition continuous-self-balancing-topic-0, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,373] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10356 on topic-partition continuous-self-balancing-topic-7, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-7 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10356 on topic-partition continuous-self-balancing-topic-0, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10356 on topic-partition continuous-self-balancing-topic-6, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,384] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-6 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,385] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10357 on topic-partition continuous-self-balancing-topic-0, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,385] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10358 on topic-partition continuous-self-balancing-topic-7, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-7 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10358 on topic-partition continuous-self-balancing-topic-6, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-6 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10358 on topic-partition continuous-self-balancing-topic-4, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,397] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-4 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,398] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10360 on topic-partition continuous-self-balancing-topic-6, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,398] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-6 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,398] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10360 on topic-partition continuous-self-balancing-topic-4, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,398] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-4 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,411] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10361 on topic-partition continuous-self-balancing-topic-4, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,412] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-4 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,412] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 10362 on topic-partition continuous-self-balancing-topic-4, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:30:00,412] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition continuous-self-balancing-topic-4 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-16 10:29:55,450] 25327 records sent, 5064.4 records/sec (4.95 MB/sec), 15.3 ms avg latency, 80.0 ms max latency.
```

After waiting a few seconds, production will resume normal operations. Check the partition status again afterward.

```text theme={null}

Topic: continuous-self-balancing-topic TopicId: HtVB3bM7TYaNKKKmm7khQw PartitionCount: 8   ReplicationFactor: 1    Configs: min.insync.replicas=1,segment.bytes=1073741824
    Topic: continuous-self-balancing-topic Partition: 0    Leader: 1   Replicas: 1 Isr: 1
    Topic: continuous-self-balancing-topic Partition: 1    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 2    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 4    Leader: 1   Replicas: 1 Isr: 1
    Topic: continuous-self-balancing-topic Partition: 5    Leader: 2   Replicas: 2 Isr: 2
    Topic: continuous-self-balancing-topic Partition: 6    Leader: 1   Replicas: 1 Isr: 1
    Topic: continuous-self-balancing-topic Partition: 7    Leader: 1   Replicas: 1 Isr: 1
```

It has been observed that because we reassigned all partitions to node2, all messages are being sent to node2 during production, creating a local hotspot on node2 and triggering AutoMQ's Self-Balancing. AutoMQ reassigns partitions to achieve a balanced state across the nodes.


# Example: Partition Reassignment in Seconds
Source: https://docs.automq.com/automq/getting-started/explore-automq/example-partition-reassignment-in-seconds

Optimize partition reassignment with AutoMQ's Kafka-compatible service using CLI tools for sub-second performance, leveraging cloud-native scalability and efficiency.

This document outlines the process for using Kafka CLI tools to verify AutoMQ's sub-second partition reassignment capability. The Kafka CLI tool is executed via a Docker image provided by AutoMQ.

First, create a new Kafka topic with a single partition and initiate a producer to write a certain amount of data into it. Next, attempt to reassign the partition among different AutoMQ brokers using the reassign command. Observe the time it takes to complete the partition reassignment.

### Prerequisites

Before carrying out a partition reassignment test, the following conditions must be met:

**Complete the installation and deployment of the AutoMQ cluster**. You can refer to the following methods to install and deploy AutoMQ:

* [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

**Additionally, the host running the test program** must meet the following conditions:

* Linux/Mac/Windows Subsystem for Linux

* Docker

### Experience Partition Reassignment

If the previous AutoMQ cluster was deployed following the [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker) guide, the obtained cluster Bootstrap address will resemble "**server1:9092,server2:9092,server3:9092**," and the AutoMQ cluster is on the "**automq\_net**" Docker network.

<Tip>
  Please replace the bootstrap-server address below with the actual cluster's address based on the deployment configuration.
</Tip>

#### Create Topic

```bash theme={null}
docker run --network automq_net automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

```

#### Send Messages

Execute the following command to send a specified amount of data.

```bash theme={null}
docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"
```

#### View Partition Distribution

```bash theme={null}
docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"
```

#### Reassign Partitions Between Brokers

In this step, it's assumed that partition 0 of the test topic "reassign-topic" is currently on node 1, and the goal is to reassign this partition to node 2. Depending on the actual partition distribution during testing, you might opt to reassign to different nodes.

1. Create the partition reassignment plan.

```bash theme={null}
echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json

```

2. Execute the partition reassignment plan.

```bash theme={null}
docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --execute"
```

#### 3. Check if the Reassignment Is Complete.

```bash theme={null}
docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --verify"
```


# Example: Self-Balancing when Cluster Nodes Change
Source: https://docs.automq.com/automq/getting-started/explore-automq/example-self-balancing-when-cluster-nodes-change

AutoMQ ensures seamless data distribution with automatic partition reassignment during cluster changes, leveraging cloud-native scalability and Kafka compatibility.

This document presents the Kafka CLI tools for verifying automatic partition reassignment and data balancing while scaling an AutoMQ cluster. The Kafka CLI tools run via a Docker image offered by AutoMQ.

1. Create a topic with 16 partitions and distribute a balanced load.

2. As you start and stop brokers, check to see if partitions automatically reassign themselves across different brokers.

This automatic data balancing is an intrinsic feature of AutoMQ, ensuring that data is distributed automatically and evenly throughout the cluster. By observing the distribution of partitions and broker load, you can confirm whether automatic partition balancing functions as anticipated.

### Prerequisites

Before conducting automated partition data rebalance tests, the following conditions must be met:

**Complete the installation and deployment of the AutoMQ cluster**. You can refer to the following methods for installing and deploying AutoMQ:

* [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

<Tip>
  If deploying the cluster through [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) or [Deploy Multi-Nodes Cluster on Kubernetes▸](/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes), you need to ensure that when starting the Controller, autobalancer.controller.enable is set to true to enable automatic data rebalancing.
</Tip>

**Additionally, the host running the test program** needs to meet the following conditions:

* Linux/Mac/Windows Subsystem for Linux

* Docker

### Experience Partition Reassignment Triggered by Cluster Node Changes.

If the previous AutoMQ cluster was deployed by following the guide [Deploy Multi-Nodes Test Cluster on Docker▸](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker), the cluster bootstrap address you would have acquired might look like "**server1:9092,server2:9092,server3:9092**", and the AutoMQ cluster would be in the "**automq\_net**" Docker network.

<Tip>
  Please replace the bootstrap-server address below with the actual address of the cluster based on your deployment configuration.
</Tip>

#### Create Topic

```bash theme={null}
docker run  --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --partitions 16 --create --topic self-balancing-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

```

#### View Partition Distribution

```bash theme={null}

docker run  --network automq_net   automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic self-balancing-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"

```

```text theme={null}

Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 1        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 4        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 5        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 8        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 9        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 12        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 13        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 2        Replicas: 2        Isr: 2
```

#### Launch Producer

```bash theme={null}
docker run  --network automq_net  automqinc/automq:1.5.5 /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic self-balancing-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"
```

#### Start the Consumer

```bash theme={null}
docker run --network automq_net  automqinc/automq:1.5.5 /bin/bash -c "/opt/kafka/kafka/bin/kafka-consumer-perf-test.sh --topic self-balancing-topic --show-detailed-stats --timeout 300000 --messages=1024000 --reporting-interval 1000 --bootstrap-server=server1:9092,server2:9092,server3:9092"
```

#### Stop the Broker

Stop a server, causing its partitions to be reassigned to other nodes. After stopping, you can observe how producers and consumers recover.

```bash theme={null}

docker stop automq-server3

```

After stopping, the producer logs will appear as follows:

```text theme={null}

[2024-04-29 05:00:03,436] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 49732 on topic-partition self-balancing-topic-7, retrying (2147483641 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-04-29 05:00:03,438] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-7 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)

```

After waiting a few seconds, you will see that production and consumption return to normal.

```text theme={null}

2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-3 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Got error produce response with correlation id 42141 on topic-partition self-balancing-topic-3, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,920] WARN [Producer clientId=perf-producer-client] Received invalid metadata error in produce request on partition self-balancing-topic-3 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2024-05-07 11:56:08,588] 25693 records sent, 5138.6 records/sec (5.02 MB/sec), 8.9 ms avg latency, 1246.0 ms max latency.
[2024-05-07 11:56:13,589] 25607 records sent, 5120.4 records/sec (5.00 MB/sec), 1.8 ms avg latency, 44.0 ms max latency.
[2024-05-07 11:56:18,591] 25621 records sent, 5121.1 records/sec (5.00 MB/sec), 1.6 ms avg latency, 10.0 ms max latency.
```

#### Review Partition Distribution Again

After the producer resumes writing, we examine the partition distribution once more and observe that all partitions are located on broker1. AutoMQ efficiently and quickly completes the reassignment of partitions and rebalancing of traffic from the stopped node.

```bash theme={null}

docker run --network automq_net  automqinc/automq:1.5.0 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic self-balancing-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"

```

```text theme={null}

Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 1        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 4        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 5        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 8        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 9        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 12        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 13        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 1        Replicas: 1        Isr: 1
```

#### Restart the Broker

Restart automq-server3 to trigger the automatic reassignment of partitions. After several seconds of retrying, the producer and consumer can resume operations.

```bash theme={null}

docker start automq-server3

```

At this stage, if we review the partition distribution again, we can confirm that the partitions have been automatically reassigned.

```text theme={null}

Topic: self-balancing-topic        TopicId: AjoAB22YRRq7w6MdtZ4hDA        PartitionCount: 16        ReplicationFactor: 1        Configs: min.insync.replicas=1,elasticstream.replication.factor=1,segment.bytes=1073741824
        Topic: self-balancing-topic        Partition: 0        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 1        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 2        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 3        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 4        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 5        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 6        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 7        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 8        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 9        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 10        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 11        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 12        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 13        Leader: 2        Replicas: 2        Isr: 2
        Topic: self-balancing-topic        Partition: 14        Leader: 1        Replicas: 1        Isr: 1
        Topic: self-balancing-topic        Partition: 15        Leader: 1        Replicas: 1        Isr: 1
```


# Apache Doris
Source: https://docs.automq.com/automq/integrations/data-analysis/apache-doris

AutoMQ ensures cloud-native scalability, Kafka compatibility, and cost efficiency for importing data from AutoMQ into Apache Doris, driving fast, real-time analytics.

[Apache Doris](https://doris.apache.org/) is a high-performance, real-time analytical database based on MPP architecture, known for its exceptional ease of use and sub-second response times for queries on massive datasets. It supports both high-concurrency point query scenarios and high-throughput complex analytical scenarios. Consequently, Apache Doris is well-suited for use cases such as report analysis, ad-hoc queries, unified data warehouse construction, and data lake federation query acceleration. Users can build applications on top of this, such as user behavior analysis, A/B testing platforms, log search analysis, user profiling, and order analysis.

This document discusses how to import data from AutoMQ into Apache Doris using Apache Doris Routine Load. For detailed information about Routine Load, refer to the [Routine Load Fundamentals](https://doris.apache.org/docs/4.x/data-operate/import/import-way/routine-load-manual/) document.

## Environment Preparation

### Prepare Apache Doris and Test Data

Ensure that an operational Apache Doris cluster is available. For demonstration purposes, we have set up a test Apache Doris environment on Linux by following the [Docker Deployment for Doris](https://doris.apache.org/docs/3.0/gettingStarted/what-is-apache-doris) document.

Create a database and test table:

```bash theme={null}
create database automq_db;
CREATE TABLE automq_db.users (
                                 id bigint NOT NULL,
                                 name string NOT NULL,
                                 timestamp string NULL,
                                 status string NULL

) DISTRIBUTED BY hash (id) PROPERTIES ('replication_num' = '1');

```

### Prepare Kafka Command-line Tools

Download the latest TGZ package from [AutoMQ Releases](https://github.com/AutoMQ/automq) and extract it. Assuming the extraction directory is `$AUTOMQ_HOME`, this article will use the tools in `$AUTOMQ_HOME/bin` to create topics and generate test data.

### Prepare AutoMQ and Test Data

Refer to AutoMQ [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) to deploy a functional cluster, ensuring network connectivity between AutoMQ and Apache Doris.

Quickly create a topic named `example_topic` in AutoMQ and write a test JSON data into it, following the steps below.

#### **Create Topic**

Use the Apache Kafka command-line tool to create the topic. Make sure you have access to the Kafka environment and that the Kafka service is running. Here is an example command for creating a topic:

```bash theme={null}
$AUTOMQ_HOME/bin/kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 127.0.0.1:9092  --partitions 1 --replication-factor 1

```

<Tip>
  When executing the command, you need to replace the topic and bootstrap-server with the actual AutoMQ Bootstrap Server address.
</Tip>

After creating the topic, you can use the following command to verify if the topic was successfully created.

```bash theme={null}
$AUTOMQ_HOME/bin/kafka-topics.sh --describe example_topic --bootstrap-server 127.0.0.1:9092

```

#### **Generate Test Data**

Generate a JSON formatted test data that corresponds with the previous table.

```json theme={null}
{
  "id": 1,
  "name": "Test User"
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

```

#### **Write Test Data**

Use Kafka command line tools or programming methods to write the test data into a Topic named example\_topic. Below is an example using the command line tool:

```bash theme={null}
echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic example_topic

```

Use the following command to view the data just written to the topic:

```bash theme={null}
sh $AUTOMQ_HOME/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic example_topic --from-beginning

```

<Tip>
  When executing the command, you need to replace the topic and bootstrap-server with the actual AutoMQ Bootstrap Server address.
</Tip>

## Create Routine Load Import Job

Create a Routine Load job in the Apache Doris command line to receive JSON data, which is used for continuously importing data from the AutoMQ Kafka topic. For specific parameters of Routine Load, refer to the Doris Routine Load documentation.

```sql theme={null}
CREATE ROUTINE LOAD automq_example_load ON users
COLUMNS(id, name, timestamp, status)
PROPERTIES
(
    "format" = "json",
    "jsonpaths" = "[\"$.id\",\"$.name\",\"$.timestamp\",\"$.status\"]"
 )
FROM KAFKA
(
    "kafka_broker_list" = "127.0.0.1:9092",
    "kafka_topic" = "example_topic",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);

```

<Tip>
  When executing the command, replace `kafka_broker_list` with the actual AutoMQ Bootstrap Server address being used.
</Tip>

## Verify Data Import

First, check the status of the Routine Load job to ensure the task is running.

```sql theme={null}

show routine load\G;

```

Then, query the relevant table in the Apache Doris database to confirm that the data has been successfully imported.

```sql theme={null}
select * from users;
+------+--------------+---------------------+--------+
| id   | name         | timestamp           | status |
+------+--------------+---------------------+--------+
|    1 | Test User     | 2023-11-10T12:00:00 | active |
|    2 | Test User     | 2023-11-10T12:00:00 | active |
+------+--------------+---------------------+--------+
2 rows in set (0.01 sec)

```


# Databend
Source: https://docs.automq.com/automq/integrations/data-analysis/databend

AutoMQ integrates with Databend, enabling cloud-native scalability and Kafka compatibility for efficient big data analytics. Learn how to ingest data from AutoMQ to Databend.

[Databend](https://github.com/datafuselabs/databend) is a next-generation cloud-native data warehouse developed in Rust. It is specifically designed for cloud architecture. Built on object storage, it offers businesses integration with data lakes and warehouses while providing a big data analytics platform that separates compute from storage.

This document will guide you on how to import data from AutoMQ into Databend using [bend-ingest-kafka](https://github.com/databendcloud/bend-ingest-kafka).

## Environment Preparation

### Prepare Databend Cloud and Test Data

Firstly, visit [Databend Cloud](https://docs.databend.cn/guides/cloud/new-account) to initiate a Warehouse. Then, create databases and test tables within the worksheet.

```sql theme={null}
create database automq_db;
create table users (
    id bigint NOT NULL,
    name string NOT NULL,
    ts timestamp,
    status string
)

```

### Prepare AutoMQ and Test Data

Refer to [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) to set up AutoMQ, making sure there's network connectivity between AutoMQ and Databend.

Quickly create a topic named `example_topic` in AutoMQ and write test JSON data into it by following these steps.

#### **Create Topic**

Use Apache Kafka command-line tools to create a topic. Ensure you have access to the Kafka environment and that the Kafka service is running. Below is an example command for creating a topic:

```bash theme={null}
./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092  --partitions 1 --replication-factor 1

```

<Tip>
  When executing commands, replace `topic` and `bootstrap-server` with the actual Kafka server addresses.
</Tip>

After creating the topic, you can use the following command to verify whether the topic has been successfully created.

```bash theme={null}
./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092
```

#### **Generate Test Data**

Generate test data in JSON format that corresponds to the previously mentioned table.

```json theme={null}
{
  "id": 1,
  "name": "Test User",
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

```

#### **Write Test Data**

Use Kafka's command-line tools or programming methods to write test data into a topic named example\_topic. Here is an example using command-line tools:

```bash theme={null}
echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic

```

<Tip>
  When executing the command, replace the topic and bootstrap-server with the actual Kafka server address being used.
</Tip>

You can view the topic data just written using the following command:

```bash theme={null}
sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning
```

## Create a Bend-ingest-databend Job

[bend-ingest-kafka](https://github.com/databendcloud/bend-ingest-kafka) is designed to monitor Kafka and batch write data into a Databend Table. After deploying `bend-ingest-kafka`, the data import job can be initiated.

```bash theme={null}
bend-ingest-kafka --kafka-bootstrap-servers="localhost:9094" --kafka-topic="example_topic" --kafka-consumer-group="Consumer Group" --databend-dsn="https://cloudapp:password@host:443" --databend-table="automq_db.users" --data-format="json" --batch-size=5 --batch-max-interval=30s

```

<Tip>
  When executing the command, replace kafka-bootstrap-servers with the actual Kafka server address you are using.
</Tip>

### Parameter Description

#### **databend-dsn**

The DSN provided by Databend Cloud for connecting to the warehouse can be found in this documentation.

#### **batch-size**

`bend-ingest-kafka` will gather data until it reaches the specified batch size, then initiate a data synchronization process.

## Verify Data Import

Navigate to the Databend Cloud worksheet and query the `automq_db.users` table. You should confirm that the data has been successfully synchronized from AutoMQ to the Databend Table.

<img />


# Flink
Source: https://docs.automq.com/automq/integrations/data-analysis/flink

Integrate Flink with AutoMQ for real-time data stream analysis with Flink and AutoMQ. Enjoy 100% Kafka compatibility, cloud-native scalability, and over 10x cost savings.

## Introduction

[Apache Flink](https://flink.apache.org/) is a renowned stream processing engine widely used in event-driven, stream-batch analysis scenarios. [AutoMQ](https://github.com/AutoMQ/automq) is a highly elastic cloud-native Kafka that brings over 10x cost reduction and elasticity benefits through cloud-native modifications to the Kafka storage layer. Thanks to AutoMQ's 100% compatibility with Kafka, it can easily leverage existing Kafka ecosystem tools to read and write with Flink. This article will demonstrate how Flink can read data from an AutoMQ Topic, perform data analysis, and then write the results back to AutoMQ using a WordCount example.

## Environment Setup

### Install and Launch Flink

This document uses Flink version v1.19.0. Follow the official documentation [Flink First Step](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/try-flink/local_installation/) to deploy a v1.19.0 Flink service.

### Install and Launch AutoMQ

Refer to the [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) documentation to deploy an AutoMQ cluster. In this example, the AutoMQ version used is v1.0.4.

### Prepare Test Topic and Data

Create a topic `to-flink` to store data that needs to be imported into Flink for analysis and computation.

```bash theme={null}
### The Default Port for Locally Installed AutoMQ Is 9094.
bin/kafka-topics.sh --create --topic to-flink --bootstrap-server localhost:9094 

```

Use the command-line tool to write a batch of data for word count computation.

```bash theme={null}
bin/kafka-console-producer.sh --topic to-flink --bootstrap-server localhost:9094
```

The data to be written is as follows, and you can exit the producer by pressing Ctrl+C after finishing the input.

```text theme={null}

apple
apple
banana
banana
banana
cherry
cherry
pear
pear
pear
lemon
lemon
mango
mango
mango

```

Finally, we hope that the result obtained through Flink computation will be.

```text theme={null}

apple 2
banana 3
cherry 2
pear 3
lemon 2
mango 3

```

After writing is complete, we can try to consume the data to confirm it was successfully written.

```bash theme={null}
bin/kafka-console-consumer.sh --topic to-flink --from-beginning --bootstrap-server localhost:9094
```

Create a topic to receive the results of Flink computations.

```bash theme={null}
bin/kafka-topics.sh --create --topic from-flink --bootstrap-server localhost:9094 

```

## Read Data from AutoMQ into Flink for Analysis.

Thanks to AutoMQ's full compatibility with Kafka, we can directly use the [Kafka Connector](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/kafka/) provided by Flink to write the source and sink code to load data from AutoMQ's Topic.

### POM Dependencies

```xml theme={null}
....
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>1.19.0</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>1.19.0</version>
</dependency>

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients</artifactId>
    <version>1.19.0</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka</artifactId>
    <version>1.17.2</version>
</dependency>

....

<!-- shade plugin support for building jar -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.5.2</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>automq-wordcount-flink-job</finalName>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>com.automq.example.flink.WordCount</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

```

### Writing the Flink Job Code

The following Java code defines an AutoMQ source and sink using KafkaSource and KafkaSink, respectively. It first reads the pre-prepared "fruit list" test data from the topic to-flink. Then, it creates a DataStream to perform the WordCount computation and sinks the result into the AutoMQ topic from-flink.

```java theme={null}
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.automq.example.flink.WordCount;

import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.connector.kafka.sink.KafkaSink;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * This is a re-write of the Apache Flink WordCount example using Kafka connectors.
 * Find the reference example at https://github.com/redpanda-data/flink-kafka-examples/blob/main/src/main/java/io/redpanda/examples/WordCount.java
 */
public class WordCount {
    
    final static String TO_FLINK_TOPIC_NAME = "to-flink";
    final static String FROM_FLINK_TOPIC_NAME = "from-flink";
    final static String FLINK_JOB_NAME = "WordCount";
    
    public static void main(String[] args) throws Exception {
        // Use your AutoMQ cluster's bootstrap servers here
        final String bootstrapServers = args.length > 0 ? args[0] : "localhost:9094";
        
        // Set up the streaming execution environment
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        
        KafkaSource<String> source = KafkaSource.<String>builder()
                .setBootstrapServers(bootstrapServers)
                .setTopics(TO_FLINK_TOPIC_NAME)
                .setGroupId("automq-example-group")
                .setStartingOffsets(OffsetsInitializer.earliest())
                .setValueOnlyDeserializer(new SimpleStringSchema())
                .build();
        
        KafkaRecordSerializationSchema<String> serializer = KafkaRecordSerializationSchema.builder()
                .setValueSerializationSchema(new SimpleStringSchema())
                .setTopic(FROM_FLINK_TOPIC_NAME)
                .build();
        
        KafkaSink<String> sink = KafkaSink.<String>builder()
                .setBootstrapServers(bootstrapServers)
                .setRecordSerializer(serializer)
                .build();
        
        DataStream<String> text = env.fromSource(source, WatermarkStrategy.noWatermarks(), "AutoMQ Source");
        
        // Split up the lines in pairs (2-tuples) containing: (word,1)
        DataStream<String> counts = text.flatMap(new Tokenizer())
                // Group by the tuple field "0" and sum up tuple field "1"
                .keyBy(value -> value.f0)
                .sum(1)
                .flatMap(new Reducer());
        
        // Add the sink to so results
        // are written to the outputTopic
        counts.sinkTo(sink);
        
        // Execute program
        env.execute(FLINK_JOB_NAME);
    }
    
    /**
     * Implements the string tokenizer that splits sentences into words as a user-defined
     * FlatMapFunction. The function takes a line (String) and splits it into multiple pairs in the
     * form of "(word,1)" ({@code Tuple2<String, Integer>}).
     */
    public static final class Tokenizer
            implements
                FlatMapFunction<String, Tuple2<String, Integer>> {
        
        @Override
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
            // Normalize and split the line
            String[] tokens = value.toLowerCase().split("\\W+");
            
            // Emit the pairs
            for (String token : tokens) {
                if (token.length() > 0) {
                    out.collect(new Tuple2<>(token, 1));
                }
            }
        }
    }
    
    // Implements a simple reducer using FlatMap to
    // reduce the Tuple2 into a single string for
    // writing to kafka topics
    public static final class Reducer
            implements
                FlatMapFunction<Tuple2<String, Integer>, String> {
        
        @Override
        public void flatMap(Tuple2<String, Integer> value, Collector<String> out) {
            // Convert the pairs to a string
            // for easy writing to Kafka Topic
            String count = value.f0 + " " + value.f1;
            out.collect(count);
        }
    }
}
```

The following code, after being built using mvn, will generate an automq-wordcount-flink-job.jar, which is the job we need to submit to Flink.

### Submitting the Job to Flink

Execute the following command to submit the task jar to Flink. Through the console, we can see that 15 pieces of data have been received and processed.

```bash theme={null}
./bin/flink run automq-wordcount-flink-job.jar
```

<img />

### Verify the Analysis Results

Use the Kafka bin tools extracted from AutoMQ to consume data from `from-flink` and check the results:

```bash theme={null}
bin/kafka-console-consumer.sh --topic from-flink --from-beginning --bootstrap-server localhost:9094
```

You can see the output results below. Since it's processed in a stream and there is no watermark or window calculation set, the word count result is printed out every time a calculation is performed.

```text theme={null}

apple 1
apple 2
banana 1
banana 2
banana 3
cherry 1
cherry 2
pear 1
pear 2
pear 3
lemon 1
lemon 2
mango 1
mango 2
mango 3

```

Next, we write 5 more data entries to the `to-flink` Topic and observe the stream processing results:

```bash theme={null}
bin/kafka-console-producer.sh --topic to-flink --bootstrap-server localhost:9094
```

The data written is

```text theme={null}

apple
banana
cherry
pear
lemon

```

Then we can see that the `from-flink` Topic correctly outputs the following word count results

```text theme={null}

apple 3
banana 4
cherry 3
pear 4
lemon 3

```

We can also see on the console that 20 data entries were correctly received and processed:

<img />

## Summary

This article demonstrates how AutoMQ integrates with Flink to complete a Word Count analysis workflow. For more configurations and usage of the Kafka Connector, refer to the official Flink documentation [Apache Kafka Connector](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/kafka/).


# RisingWave
Source: https://docs.automq.com/automq/integrations/data-analysis/risingwave

Import data from AutoMQ into RisingWave’s SQL-compatible streaming database. Experience cloud-native scalability, complete Kafka compatibility, and cost efficiency.

[RisingWave](https://risingwave.com/) \[1] is a distributed stream database that provides a standard SQL interface compatible with the PostgreSQL ecosystem, allowing integration without the need for code modification. RisingWave treats streams as tables, enabling users to write complex queries on both streaming and historical data. With RisingWave, users can concentrate on query analytical logic without having to learn Java or the underlying API of specific systems.

This article will guide you on how to import data from AutoMQ into the RisingWave database using [RisingWave Cloud](https://cloud.risingwave.com/) \[2].

## Prepare AutoMQ and Test Data

Refer to [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) for instructions on deploying AutoMQ, ensuring network connectivity between AutoMQ and RisingWave.

Quickly create a topic named `example_topic` in AutoMQ, and write a test JSON data to it by following these steps.

#### **Create Topic**

Use the Apache Kafka command line tool to create the Topic. Ensure you have access to a Kafka environment and the Kafka service is running. Below is an example command to create the Topic:

```bash theme={null}
./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092  --partitions 1 --replication-factor 1

```

<Tip>
  When executing the command, replace the `topic` and `bootstrap-server` with the actual Kafka server address.
</Tip>

After creating the Topic, you can use the following command to verify whether the Topic has been successfully created.

```bash theme={null}
./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092
```

#### **Generate Test Data**

Generate test data in JSON format that corresponds to the table mentioned earlier.

```json theme={null}
{
  "id": 1,
  "name": "Test User"
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

```

#### **Write Test Data**

Use Kafka's command line tool or programming methods to write the test data into the Topic named example\_topic. Below is an example using the command line tool:

```bash theme={null}
echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic

```

Use the following command to view the data just written to the topic:

```bash theme={null}
sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning
```

<Tip>
  When executing the command, replace the `topic` and `bootstrap-server` with the actual Kafka server address.
</Tip>

## Creating an AutoMQ Source on RisingWave Cloud

1. Go to [RisingWave Cloud](https://cloud.risingwave.com/) \[3] and open **Clusters** to set up a cluster.

2. Open **Source** in RisingWave Cloud \[4] to set up a source.

3. Specify the cluster and database, and log in to the database.

4. AutoMQ is 100% compatible with Apache Kafka®. Click **Create source** and select Kafka.

5. Configure the connector according to the RisingWave Cloud guided interface, setting the source information and schema details.

6. Confirm the generated SQL statement and click **Confirm** to complete the creation of the source.

<Note>
  The default port for AutoMQ is 9092, and SSL is not enabled. If you need SSL, see the [Apache Kafka Documentation](https://kafka.apache.org/documentation/#security_ssl) \[5].

  In this example, you can set the startup mode to earliest and use JSON format to access all data from the topic from the beginning.
</Note>

## Query Data

1. Go to [RisingWave Cloud](https://cloud.risingwave.com/) \[6] and open **Console** to log into your cluster.

2. Run the following SQL statement to access the imported data, replacing the variable your\_source\_name with the custom name specified when creating the source.

```sql theme={null}
SELECT * from {your_source_name} limit 1;

```

## Citation

\[1] RisingWave: [https://risingwave.com/](https://risingwave.com/)

\[2] RisingWave Cloud: [https://cloud.risingwave.com/](https://cloud.risingwave.com/)

\[3] RisingWave Cloud: [https://cloud.risingwave.com/](https://cloud.risingwave.com/)

\[4] RisingWave Cloud Source: [https://cloud.risingwave.com/](https://cloud.risingwave.com/)

\[5] Apache Kafka Documentation: [https://kafka.apache.org/documentation/#security\_ssl](https://kafka.apache.org/documentation/#security_ssl)

\[6] RisingWave Cloud Console: [https://cloud.risingwave.com/](https://cloud.risingwave.com/)


# StarRocks
Source: https://docs.automq.com/automq/integrations/data-analysis/starrocks

AutoMQ enables real-time data import to StarRocks, leveraging Kafka compatibility for integration and cloud-native scalability. Achieve efficient, high-concurrency analysis.

[StarRocks](https://www.starrocks.io/) is a high-performance analytical data warehouse that leverages advanced technologies such as vectorization, MPP architecture, CBO, intelligent materialized views, and a real-time updatable columnar storage engine. It supports multidimensional, real-time, and high-concurrency data analysis.

This article will introduce how to use StarRocks Routine Load to import data from AutoMQ into StarRocks. For a detailed understanding of the basic principles of Routine Load, refer to the [Routine Load Basic Principles](https://docs.starrocks.io/docs/loading/load_concept/strict_mode/#routine-load) documentation.

## Prerequisites

### Prepare StarRocks and Test Data

Ensure that a usable StarRocks cluster is already prepared. For demonstration purposes, you can install a demonstration StarRocks cluster on a Linux machine.

Create test tables for the database and primary key model:

```sql theme={null}
create database automq_db;
create table users (
    id bigint NOT NULL,
    name string NOT NULL,
    timestamp string NULL,
    status string NULL
) PRIMARY KEY (id)
DISTRIBUTED BY HASH(id)
PROPERTIES (
    "replication_num" = "1",
    "enable_persistent_index" = "true"
);

```

### Prepare AutoMQ and Test Data

Refer to [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) to deploy AutoMQ and ensure network connectivity between AutoMQ and StarRocks.

Quickly create a topic named `example_topic` in AutoMQ and write a test JSON data to it following these steps.

#### **Create Topic**

Use the Apache Kafka® command-line tool to create a topic. Ensure you have access to the Kafka environment and that the Kafka service is running. Below is an example command to create a topic:

```bash theme={null}
./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092  --partitions 1 --replication-factor 1

```

<Tip>
  When executing the command, replace `topic` and `bootstrap-server` with the actual Kafka server address.
</Tip>

After creating the topic, you can use the following command to verify whether the topic has been successfully created.

```bash theme={null}
./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092
```

#### **Generate Test Data**

Generate test data in JSON format that corresponds to the table mentioned earlier.

```json theme={null}
{
  "id": 1,
  "name": "test_user",
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

```

#### **Writing Test Data**

Use Kafka's command-line tools or programming methods to write test data into a Topic named `example_topic`. Here is an example using the command-line tool:

```bash theme={null}
echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic

```

Use the following command to view the data just written to the Topic:

```bash theme={null}
sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning
```

<Tip>
  When executing the command, replace `topic` and `bootstrap-server` with the actual Kafka server address.
</Tip>

## Creating Routine Load Import Job

Create a Routine Load job in the StarRocks command line to continuously import data from the AutoMQ Kafka Topic.

```sql theme={null}
CREATE ROUTINE LOAD automq_example_load ON users
COLUMNS(id, name, timestamp, status)
PROPERTIES
(
    "desired_concurrent_number" = "5",
    "format" = "json",
    "jsonpaths" = "[\"$.id\",\"$.name\",\"$.timestamp\",\"$.status\"]"
 )
FROM KAFKA
(
    "kafka_broker_list" = "10.0.96.4:9092",
    "kafka_topic" = "example_topic",
    "kafka_partitions" = "0",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);

```

<Tip>
  When executing the command, replace `kafka_broker_list` with the actual Kafka server address.
</Tip>

### Parameter Description

#### **Data Format**

The data format needs to be specified as JSON in the `PROPERTIES` clause with `"format" = "json"`.

#### **Data Extraction and Transformation**

If you need to specify a mapping and conversion relationship between the source data and the target table columns, you can configure the `COLUMNS` and `jsonpaths` parameters. In `COLUMNS`, the column names correspond to the column names of the **target table**, and the order of columns corresponds to the order of columns in **source data**. The `jsonpaths` parameter is used to extract the necessary field data from the JSON data, similar to newly generated CSV data. Subsequently, the `COLUMNS` parameter will temporarily name the fields in the order specified by `jsonpaths`.

<Tip>
  If each line contains a JSON object where the names and number of keys correspond to the columns in the target table (order does not need to match), the `COLUMNS` configuration is not required.
</Tip>

## Validate Data Import

First, check the status of the Routine Load import job to ensure the task is running.

```sql theme={null}

show routine load\G;

```

And then querying the relevant tables in the StarRocks database, you can see that the data has been successfully imported.

```sql theme={null}
StarRocks > select * from users;
+------+--------------+---------------------+--------+
| id   | name         | timestamp           | status |
+------+--------------+---------------------+--------+
|    1 | Test User     | 2023-11-10T12:00:00 | active |
|    2 | Test User     | 2023-11-10T12:00:00 | active |
+------+--------------+---------------------+--------+
2 rows in set (0.01 sec)

```


# Airbyte
Source: https://docs.automq.com/automq/integrations/data-integration/airbyte

AutoMQ, a cloud-native Kafka-compatible service, delivers cost-efficient, scalable data flow with Airbyte integration for optimal analytics.

## Preface

This guide shows how to integrate AutoMQ \[1] with Airbyte \[2] and a data warehouse to build a real-time data flow and analytics pipeline.

### AutoMQ Overview

AutoMQ is a Kafka-compatible streaming platform. For an overview, see [AutoMQ Overview](/automq/what-is-automq).

### Airbyte Overview

Airbyte is a data integration platform designed to simplify and automate the creation and management of data pipelines. It supports a wide variety of source and target systems, enabling users to easily configure data pipelines through a user-friendly web interface or API. Airbyte offers efficient Extract, Transform, Load (ETL) capabilities with built-in scheduling and monitoring mechanisms to ensure the reliability and performance of data pipelines. Its modular design supports custom connectors to meet diverse data integration demands.

Airbyte's major advantages include high scalability and flexibility, allowing users to swiftly adapt to various data sources and target systems. Built-in data normalization and automated scheduling functionalities enhance the efficiency and consistency of data processing. With containerized deployment, Airbyte streamlines installation and scaling, making it apt for enterprise-level data integration and data warehousing. Additionally, its comprehensive connector library and community support make it an excellent tool for data engineers and analysts to efficiently address complex data integration challenges.

<img alt="Connections" />

## Prerequisites

* Data Source: An available AutoMQ node.

* Data Connector: Available Airbyte Environment.

* Data Endpoint (Data Warehouse): In this example, I've selected a cloud-deployed Databricks \[3] cluster.

## Quick Deployment

### Deploy AutoMQ

Deployment can be achieved by consulting the official AutoMQ documentation: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux). Once the setup is complete, data preparation can be done using either the Kafka SDK or manually, followed by the data synchronization process. I've prepared some data in advance, which can be observed using various visualization tools to monitor AutoMQ node status, such as [Redpanda Console](https://www.redpanda.com/redpanda-console-kafka-ui) \[5], [Kafdrop](https://github.com/obsidiandynamics/kafdrop) \[6], and others. Here, I've chosen Redpanda Console, where you can see that there are currently 50 topics, each containing 1000 initial messages.

<img />

Message Format:

```json theme={null}
[
    {
        "partitionID": 0,
        "offset": 950,
        "timestamp": 1721988652404,
        "compression": "uncompressed",
        "isTransactional": false,
        "headers": [],
        "key": {
            "payload": "key-451",
            "encoding": "text"
        },
        "value": {
            "payload": {
                "userId": 451,
                "action": "visit",
                "timestamp": 1721988652404
            },
            "encoding": "json"
        }
    }
]

```

### Deploying Airbyte

> Refer to the official Airbyte documentation: [Quickstart | Airbyte](https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart) \[7]

Here, I will use the example of deploying Airbyte on a Linux system.

#### Environment Preparation

First, you need to install `abctl`, an official setup tool provided by Airbyte that facilitates quick setup of the required Airbyte environment. Note that this tool requires a Docker environment. If you don't have Docker installed, see Docker's installation instructions: [Docker Install](https://docs.docker.com/desktop/install/linux-install/) \[8]. You can check your Docker version by running the command `docker version`:

```bash theme={null}
Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
 .........
```

#### Preparing the Abctl Tool

To get started with abctl, execute the following commands sequentially. Here, I'm downloading version `version: v0.9.2`:

```bash theme={null}
# Download:
wget https://github.com/airbytehq/abctl/releases/download/v0.9.2/abctl-v0.9.2-linux-amd64.tar.gz
# Unzip:
tar -xvzf abctl-v0.9.2-linux-amd64.tar.gz
# Enter:
cd abctl-v0.9.2-linux-amd64
# Add Execution Permission:
chmod +x abctl
# Global Environment:
sudo mv abctl /usr/local/bin
# Verify Version:
abctl version
# Output
version: v0.9.2
```

#### Deploying the Airbyte Environment

By executing the command `abctl local install`, this will pull Airbyte's images in Docker and deploy the environment using Helm. Some of the logs are as follows:

```bash theme={null}
INFO    Namespace 'airbyte-abctl' already exists
  INFO    Persistent volume 'airbyte-minio-pv' already exists
  INFO    Persistent volume 'airbyte-volume-db' already exists
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' already exists
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' already exists
  INFO    Starting Helm Chart installation of 'airbyte/airbyte' (version: 0.350.0)
 SUCCESS  Installed Helm Chart airbyte/airbyte:
            Name: airbyte-abctl
            Namespace: airbyte-abctl
            Version: 0.350.0
            Release: 2
  INFO    Starting Helm Chart installation of 'nginx/ingress-nginx' (version: 4.11.1)
 SUCCESS  Installed Helm Chart nginx/ingress-nginx:
            Name: ingress-nginx
            Namespace: ingress-nginx
            Version: 4.11.1
            Release: 2
 SUCCESS  Basic-Auth secret created
 SUCCESS  Found existing Ingress
 SUCCESS  Updated existing Ingress
 SUCCESS  Launched web-browser successfully for http://localhost:8000
 SUCCESS  Airbyte installation complete
```

Once the launch is successful, you can log in via your browser at `http://localhost:8000` with the default credentials:

* Username: `airbyte`yaml
* Password: `password`

If you want to set your own username and password, use command line flags or variables. For example, to set the username and password to `zhaoxi` and `ktpro123` respectively, you can run the following command:

```bash theme={null}

abctl local install --username zhaoxi --password ktpro123

```

Or you can set these values using environment variables:

```bash theme={null}
export ABCTL_LOCAL_INSTALL_PASSWORD=airbyte
export ABCTL_LOCAL_INSTALL_USERNAME=password
```

After entering your username and password, you will access the Airbyte workspace. This interface allows you to easily set up and manage all connections and move data!

<img />

### Deploying Databricks

If you do not yet have a Databricks service available, refer to the official documentation for setup: [Google Databricks](https://cloud.google.com/databricks?hl=zh_cn)\[9].

## Data Synchronization

### Add New Data Source

Add AutoMQ as a data source. Thanks to AutoMQ's full compatibility with Kafka, you can set up an AutoMQ data source using Kafka's data source template. Navigate via the Airbyte interface's left sidebar -> Sources -> search Kafka, then fill in basic information such as Bootstrap Servers, Protocol, Topic Pattern, etc.

<img alt="We then need to specify the object of data transfer, which can be topics meeting custom regex criteria, or directly specify particular topics that need data transfer. Here, I choose to use a regex expression " />

We then need to specify the object of data transfer, which can be topics that meet custom regex criteria, or you can directly specify the topics to be transferred. Here, I choose to use the regex expression `Topic-.*` to match all topics with the prefix `Topic-`. This aligns with the format of my prepared data, so you need to ensure your data can be matched as well. After successful addition, we can see the following results, proving that the data source connection was successful:

<img />

### Add Data Destination

We have chosen Databricks as our data destination, although you can select other options if you wish. For a complete list of supported destinations, visit: [Destinations | Airbyte](https://docs.airbyte.com/integrations/destinations/) \[10]. In the Airbyte interface, go to the sidebar -> Destinations -> Search for Databricks:

<img alt="The credential information required needs to be obtained from the information within the Databricks cluster. Detailed steps are:" />

The necessary credential information can be obtained from the Databricks cluster. The specific steps are as follows:

* Go to the created Databricks Cluster -> Select Advanced Options -> JDBC/ODBC, and you will find the values for HTTP PATH and Server Hostname.

<img alt="In the top right corner of the cluster, select the user -> go to Settings -> choose User -> Developer -> AccessToken -> Generate new Token. You will receive something similar to " />

* In the top right corner of the cluster, select the user -> go to Settings -> choose User -> Developer -> AccessToken -> Generate new Token. You will receive a Token similar to `dapi8d336faXXXXXXXXXa6aa18a086c0e`.

Once you have the credential information, proceed to create a data endpoint. If successful, you will see the following interface:

<img />

### Initiate Connection and Transfer Data

With both the data source and data endpoint ready, we can now establish a connection. Select Airbyte's left sidebar -> Connections -> choose the data source and data endpoint -> establish connection.

After successfully connecting, you need to select the mode of data transmission. Here, both incremental sync and full sync options are provided. I opted for the full sync mode:

<img alt="Select the specific Topics data you need to transmit:" />

Select the specific Topics data you need to transmit:

<img alt="Configure sync frequency and target data formats:" />

Configure sync frequency and target data formats:

<img alt="Start Sync:" />

Start Sync:

<img alt="You can check the synchronization status via Job History -> Job -> Logs, where part of the log content is:" />

You can check the synchronization status via Job History -> Job -> Logs, where part of the log content is:

```bash theme={null}
2024-07-29 08:53:33 source > INFO o.a.k.c.c.i.AbstractCoordinator(resetStateAndGeneration):998 [Consumer clientId=consumer-airbyte-consumer-group-1, groupId=airbyte-consumer-group] Resetting generation and member id due to: consumer pro-actively leaving the group
2024-07-29 08:53:33 source > INFO o.a.k.c.c.i.AbstractCoordinator(requestRejoin):1045 [Consumer clientId=consumer-airbyte-consumer-group-1, groupId=airbyte-consumer-group] Request joining group due to: consumer pro-actively leaving the group
2024-07-29 08:53:33 source > INFO o.a.k.c.m.Metrics(close):659 Metrics scheduler closed
2024-07-29 08:53:33 source > INFO o.a.k.c.m.Metrics(close):663 Closing reporter org.apache.kafka.common.metrics.JmxReporter
2024-07-29 08:53:33 source > INFO o.a.k.c.m.Metrics(close):669 Metrics reporters closed
2024-07-29 08:53:33 source > INFO o.a.k.c.u.AppInfoParser(unregisterAppInfo):83 App info kafka.consumer for consumer-airbyte-consumer-group-1 unregistered
2024-07-29 08:53:33 source > INFO i.a.c.i.b.IntegrationRunner(runInternal):231 Completed integration: io.airbyte.integrations.source.kafka.KafkaSource
2024-07-29 08:53:33 source > INFO i.a.i.s.k.KafkaSource(main):62 Completed source: class io.airbyte.integrations.source.kafka.KafkaSource
2024-07-29 08:53:33 replication-orchestrator > (pod: airbyte-abctl / source-kafka-read-2-0-pbvbp) - Closed all resources for pod
2024-07-29 08:53:33 replication-orchestrator > Total records read: 0 (0 bytes)
2024-07-29 08:53:33 replication-orchestrator > Schema validation was performed to a max of 10 records with errors per stream.
2024-07-29 08:53:33 replication-orchestrator > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2024-07-29 08:53:33 replication-orchestrator > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2024-07-29 08:53:33 replication-orchestrator > thread status... heartbeat thread: false , replication thread: true
2024-07-29 08:53:33 replication-orchestrator > writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2024-07-29 08:53:33 replication-orchestrator > thread status... timeout thread: false , replication thread: true
2024-07-29 08:53:35 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-27. schema: default, table name: _airbyte_raw_topic_27
2024-07-29 08:53:40 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-24. schema: default, table name: _airbyte_raw_topic_24
2024-07-29 08:53:45 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-25. schema: default, table name: _airbyte_raw_topic_25
2024-07-29 08:53:50 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-28. schema: default, table name: _airbyte_raw_topic_28
2024-07-29 08:53:55 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-29. schema: default, table name: _airbyte_raw_topic_29
2024-07-29 08:54:01 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-30. schema: default, table name: _airbyte_raw_topic_30
2024-07-29 08:54:06 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-33. schema: default, table name: _airbyte_raw_topic_33
2024-07-29 08:54:10 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-34. schema: default, table name: _airbyte_raw_topic_34
2024-07-29 08:54:15 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-31. schema: default, table name: _airbyte_raw_topic_31
2024-07-29 08:54:19 destination > INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):147 Preparing raw table in destination started for stream Topic-32. schema: default, table name: _airbyte_raw_topic_32
```

Sync successful:

<img />

## Verification Results

After successfully transferring the data, we can access the Databricks cluster to review the transfer results:

<img alt="It can be seen that we have successfully synchronized the selected Topics data from the AutoMQ node to Databricks. Next, data retrieval and processing can be performed via SQL. For specific syntax, please refer to the official documentation:" />

We have successfully synchronized the selected Topics data from the AutoMQ node to Databricks. Next, data retrieval and processing can be performed via SQL. For specific syntax, please refer to the official documentation: [SQL language](https://docs.databricks.com/en/sql/language-manual/index.html)\[11].

## Summary

In this introduction, we show how to integrate AutoMQ, Airbyte, and Databricks to enable efficient real-time data flow and analytics. By leveraging AutoMQ's high-performance stream processing, Airbyte's adaptable data integration, and Databricks' robust data analytics capabilities, enterprises can develop a data processing platform that is both effective and scalable. This integration not only decreases storage and maintenance costs but also boosts data processing efficiency and improves the timeliness of business decisions.

## References

\[1] AutoMQ: [https://www.automq.com/zh](https://www.automq.com/zh)

\[2] Airbyte: httpsyte: [https://airbyte.com/](https://airbyte.com/)

\[3] Databricks: [https://www.databricks.com/](https://www.databricks.com/)

\[4] Quick Start AutoMQ: [https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker](https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

\[5] Redpanda Console: [https://www.redpanda.com/redpanda-console-kafka-ui](https://www.redpanda.com/redpanda-console-kafka-ui)

\[6] Kafdrop: [https://github.com/obsidiandynamics/kafdrop](https://github.com/obsidiandynamics/kafdrop)

\[7] Quickstart Airbyte: [https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart](https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart)

\[8] Docker Install: [https://docs.docker.com/desktop/install/linux-install/](https://docs.docker.com/desktop/install/linux-install/)

\[9] Google databricks: [https://cloud.google.com/databricks?hl=zh\_cn](https://cloud.google.com/databricks?hl=zh_cn)

\[10] Destinations : [https://docs.airbyte.com/integrations/destinations/](https://docs.airbyte.com/integrations/destinations/)

\[11] SQL language: [https://docs.databricks.com/en/sql/language-manual/index.html](https://docs.databricks.com/en/sql/language-manual/index.html)


# Debezium
Source: https://docs.automq.com/automq/integrations/data-integration/debezium

AutoMQ ensures Kafka compatibility with cloud-native scalability for real-time data processing, enhancing system efficiency and reducing costs through shared storage.

## Introduction

Monitoring database changes in real-time helps businesses respond quickly to events like order updates and inventory changes. Whether it involves order processing and inventory management for e-commerce platforms or transaction monitoring in financial systems, capturing and handling real-time database changes is essential. This capability enhances system responsiveness and allows timely business operations in response to data changes.

This article will describe how to use Debezium \[1] to monitor changes in MySQL databases and relay these change events to AutoMQ \[2]. AutoMQ is an efficient and cost-effective stream processing system, offering high elasticity and availability, making it well-suited for real-time data processing. By adopting this approach, businesses can manage order changes, inventory updates, and more, with the ability to set alert rules for automated monitoring. Downstream services can consume these messages to gain a current perspective on database changes, respond swiftly to business needs, optimize system performance, and ensure business continuity and stability.

### Overview of AutoMQ

AutoMQ is a Kafka-compatible streaming platform. See [Overview](/automq/what-is-automq) for details. AutoMQ’s shared storage architecture significantly reduces storage and operational complexity.

<img />

### Debezium Overview

Debezium is an open-source project that provides a low-latency streaming platform for Change Data Capture (CDC). By installing and configuring Debezium, you can monitor database changes and convert these change events into Kafka messages. Debezium supports multiple databases as data sources, including MySQL, PostgreSQL, and MongoDB, ensuring that only committed changes are visible. This means applications need not worry about transactions or rollbacks. Moreover, because Debezium uses persistent, replica-backed logs to record the history of database data changes, your application can stop and restart at any time without missing events that occurred during downtime, ensuring all events are processed correctly and completely.

Debezium leverages Kafka and Kafka Connect's persistence, reliability, and fault tolerance, with each connector monitoring an upstream database server, capturing all database changes, and recording them into Kafka Topics. This allows multiple clients to independently consume the same data change events while minimizing impact on the upstream database. Common use cases for Debezium include cache invalidation, simplifying monolithic applications, sharing databases, and data integration. With Debezium, enterprises can achieve real-time monitoring and processing of database changes, meeting various business scenario needs such as real-time data synchronization and event-driven architecture. Its architecture diagram is presented below:

<img alt="debezium structure [4]" />

## Prerequisites

* Available Docker environment.

* An available AutoMQ node for receiving data change event messages.

* A MySQL database with binlog enabled.

* An available Kafka Connect service that can connect to the AutoMQ node.

* Register the Debezium MySQL plugin with Kafka Connect for monitoring and converting data change operations.

## Quick Deployment

### Deploy AutoMQ

You can refer to the AutoMQ official documentation for setup: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux). You will receive the service access address for AutoMQ, for example: `192.168.123.41:9092`, which will then be used to connect with AutoMQ via Kafka Connect.

### Deploy MySQL

MySQL can be quickly deployed and configured using Docker with the official image provided by Debezium. This image includes some initial database tables, making the deployment process simpler. Use the following command to create a container named "mysql":

```bash theme={null}
docker run -it --rm --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw quay.io/debezium/example-mysql:2.7
```

* `MYSQL_ROOT_PASSWORD`: Set the password for the root user.

* `MYSQL_USER` and `MYSQL_PASSWORD`: Set the username and password for a standard user.

Connect to the MySQL client as the regular user:

```bash theme={null}

docker exec -it mysql mysql -u mysqluser -pmysqlpw

```

Verify the data using the command-line tool to view all existing database tables:

```text theme={null}

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| inventory          |
| performance_schema |
+--------------------+

mysql> use inventory;

mysql> show tables;
+---------------------+
| Tables_in_inventory |
+---------------------+
| addresses           |
| customers           |
| geom                |
| orders              |
| products            |
| products_on_hand    |
+---------------------+

```

### Deploy Kafka Connect

Use the following command to pull the Kafka Connect image and start the container. Make sure to specify the AutoMQ service address:

```bash theme={null}
docker run -it --rm --name connect -p 8083:8083 \
  -e GROUP_ID=1 \
  -e CONFIG_STORAGE_TOPIC=my_connect_configs \
  -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
  -e STATUS_STORAGE_TOPIC=my_connect_statuses \
  -e BOOTSTRAP_SERVERS=192.168.123.41:9092 \
  --link mysql:mysql quay.io/debezium/connect:2.7
```

Parameter Description:

| GROUP\_ID<br />                                 | The group ID of the Kafka Connect cluster.<br />             |
| ----------------------------------------------- | ------------------------------------------------------------ |
| CONFIG\_STORAGE\_TOPIC<br />                    | The AutoMQ topic for storing connector configurations.<br /> |
| OFFSET\_STORAGE\_TOPIC<br />                    | The AutoMQ topic for storing connector offsets.<br />        |
| STATUS\_STORAGE\_TOPIC<br />                    | AutoMQ topic used for storing connector status.<br />        |
| --link mysql:mysql<br />                        | Connect to the container named mysql.<br />                  |
| -e BOOTSTRAP\_SERVERS=192.168.123.41:9092<br /> | Specify the AutoMQ node address.<br />                       |

If the connection fails, check that AutoMQ and MySQL services have started and the address is correct.

### **Create and Register Debezium MySQL Connector**

Debezium MySQL Connector acts as a plugin for Kafka Connect, allowing you to monitor data changes in a MySQL database. Thus, you can register the MySQL connector using curl by executing the following command:

```bash theme={null}
# Create a File Within a Specified Directory.
cd /home
vim mysql-connector.json

```

The JSON file content is:

```json theme={null}
{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "mysql",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": "184054",
    "topic.prefix": "dbserver1",
    "database.include.list": "inventory",
    "schema.history.internal.kafka.bootstrap.servers": "192.168.123.41:9092",
    "schema.history.internal.kafka.topic": "schema-changes.inventory"
  }
}

```

Use the following command to submit the connector configuration file to Kafka Connect:

```bash theme={null}
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" --data @mysql-connector.json http://localhost:8083/connectors/
```

The successful response content is as follows:

```json theme={null}
HTTP/1.1 201 Created
Date: Mon, 05 Aug 2024 01:51:43 GMT
Location: http://localhost:8083/connectors/inventory-connector
Content-Type: application/json
Content-Length: 518
Server: Jetty(9.4.53.v20231009)

{"name":"inventory-connector","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","tasks.max":"1","database.hostname":"mysql","database.port":"3306","database.user":"debezium","database.password":"dbz","database.server.id":"184054","topic.prefix":"dbserver1","database.include.list":"inventory","schema.history.internal.kafka.bootstrap.servers":"192.168.123.41:9092","schema.history.internal.kafka.topic":"schema-changes.inventory","name":"inventory-connector-new"},"tasks":[],"type":"source"}
```

## Verify Data Change Capture

### Insert, Update, and Delete Database Data

By performing data insertions, updates, and deletions in the MySQL console, verify whether the Kafka Connector can capture these changes:

```sql theme={null}
-- insert
INSERT INTO customers (first_name, last_name, email) VALUES ('John', 'Doe', 'john.doe@example.com');

-- update
UPDATE customers SET email='john.doe@newdomain.com' WHERE first_name='John' AND last_name='Doe';

-- delete
DELETE FROM customers WHERE first_name='John' AND last_name='Doe';

```

### Examine AutoMQ Messages

Because the Kafka Connector logs are not distinct enough, a more noticeable method can be used to verify data change capture: check the Topic data in AutoMQ to validate if the capture was successful. This verification of Topic data can be done using scripts or visual monitoring tools; you can refer to the content below for specific operations.

#### Detection Using Kafka Scripts

Download the AutoMQ project binary package from: [Github Releases](https://github.com/AutoMQ/automq/releases) \[6]. After extraction, execute the script command in the root directory of the project. This command will retrieve the data changes from the customers table:

```bash theme={null}
bin/kafka-console-consumer.sh --bootstrap-server 192.168.123.41:9092 --topic dbserver1.inventory.customers --from-beginning
```

> Make sure to replace the AutoMQ service address.

Results are as follows:

<img />

#### Detection Using Visualization Tools

You can monitor the status of AutoMQ nodes through various visualization tools, such as [Redpanda Console](https://www.redpanda.com/redpanda-console-kafka-ui) \[7], [Kafdrop](https://github.com/obsidiandynamics/kafdrop) \[8], etc. Using Redpanda Console as an example, you can view all Topic data and detailed information of the current AutoMQ nodes. For specific deployment instructions for Redpanda Console, you can refer to the AutoMQ official documentation: [Redpanda Console▸](/automq/integrations/kafka-ui/redpanda-console)\[9].

Here, you can see all the database tables monitored by the connector, along with the database and connector configuration files, offsets, with each table corresponding to a Topic.

<img />

You can view detailed data change information, such as updates to customers:

<img />

You can verify data capture by updating SQL with more specific data.

## Clean up the Environment

You can easily clean up the Docker environment by executing the following command:

```bash theme={null}

docker stop mysql connect

```

> Since the `--rm` parameter was included at startup, the container will automatically be removed once it stops.

## Summary

Through the introduction in this article, we explore how to use Debezium to monitor changes in a MySQL database and send these change events to AutoMQ for processing. By deploying MySQL and Kafka Connect, and configuring the Debezium MySQL connector, enterprises can achieve real-time monitoring and processing of database changes to meet business needs such as order modifications and inventory management. The high efficiency and scalability of AutoMQ, along with the low latency and reliability of Debezium, make it an ideal choice for real-time data processing in enterprises. For more scalability options on obtaining event messages from data change monitoring, you can refer to: [Debezium](https://debezium.io/) \[10].

## References

\[1] Debezium: [https://debezium.io/](https://debezium.io/)

\[2] AutoMQ: [https://www.automq.com/](https://www.automq.com/)

\[3] Kafka Connect: [https://docs.confluent.io/platform/current/connect/index.html](https://docs.confluent.io/platform/current/connect/index.html)

\[4] Debezium Structure: [https://docs.redhat.com/zh\_hans/documentation/red\_hat\_integration/2023.q2/html/debezium\_user\_guide/description-of-debezium-architecture](https://docs.redhat.com/zh_hans/documentation/red_hat_integration/2023.q2/html/debezium_user_guide/description-of-debezium-architecture)

\[5] Quick Start with AutoMQ: [https://docs.automq.com/automq/getting-started](https://docs.automq.com/automq/getting-started)

\[6] Github Release: [https://github.com/AutoMQ/automq/releases](https://github.com/AutoMQ/automq/releases)

\[7] Redpanda Console: [https://www.redpanda.com/redpanda-console-kafka-ui](https://www.redpanda.com/redpanda-console-kafka-ui)

\[8] Kafdrop: [https://github.com/obsidiandynamics/kafdrop](https://github.com/obsidiandynamics/kafdrop)

\[9] Redpanda Console | AutoMQ: [Redpanda Console▸](/automq/integrations/kafka-ui/redpanda-console)

\[10] Debezium: [https://debezium.io/](https://debezium.io/)


# Kafdrop
Source: https://docs.automq.com/automq/integrations/kafka-ui/kafdrop

Enable real-time Kafka cluster monitoring with Kafdrop and AutoMQ's integration, scalability, and cost efficiency. Learn to set up and manage your clusters efficiently.

<Tip>
  The relevant code can be found in the [AutoMQ Labs](https://github.com/AutoMQ/automq-labs), including an AutoMQ and Kafdrop [integration example](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/ui/kafdrop) for a straightforward setup experience.
</Tip>

## **Preface**

[Kafdrop](https://github.com/obsidiandynamics/kafdrop) \[1] is a streamlined, intuitive, and powerful Web UI tool designed for Kafka. It enables developers and administrators to view and manage key metadata of Kafka clusters, including topics, partitions, consumer groups, and their offsets. By providing a user-friendly interface, Kafdrop greatly simplifies the monitoring and management processes of Kafka clusters, allowing users to efficiently access cluster status information without relying on complex command-line tools.

Thanks to AutoMQ's full compatibility with Kafka, it integrates with Kafdrop. By leveraging Kafdrop, AutoMQ users also benefit from an intuitive user interface to monitor Kafka cluster status in real-time, covering crucial metadata such as topics, partitions, consumer groups, and their offsets. This monitoring capability not only boosts the efficiency of problem diagnostics but also aids in optimizing cluster performance and resource utilization.

This tutorial will guide you on how to start the Kafdrop service and pair it with an AutoMQ cluster to achieve cluster status monitoring and management.

<img alt="Kafdrop Interface" />

## Prerequisites

* An available Kafdrop environment.

* An available AutoMQ cluster.

## Install and Start the AutoMQ Cluster

Refer to the official AutoMQ documentation for deploying an AutoMQ cluster: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) \[2]

## Starting the Kafdrop Service

In the steps mentioned above, an AutoMQ cluster has already been set up, and you've obtained the addresses and ports on which the broker nodes are listening. The next step is to start the Kafdrop service.

> Note: Ensure that the location where the Kafdrop service is hosted can access the AutoMQ cluster; otherwise, you may encounter connection timeouts or other issues.

Kafdrop can be deployed via JAR package, Docker deployment, or using the protobuf method. Refer to the [official documentation](https://github.com/obsidiandynamics/kafdrop/blob/master/README.md#getting-started) \[3].

In this example, we use the JAR package method to start the Kafdrop service. The steps are as follows:

* Clone the Kafdrop repository source code: [Kafdrop GitHub](https://github.com/obsidiandynamics/kafdrop) \[4]

```bash theme={null}
git clone https://github.com/obsidiandynamics/kafdrop.git
```

* Use Maven to compile and package Kafdrop locally to generate the JAR file. Execute the following in the root directory:

```bash theme={null}
mvn clean compile package
```

* To start the service, you need to specify the addresses and ports of the AutoMQ cluster brokers:

```bash theme={null}
java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
    -jar target/kafdrop-<version>.jar \
    --kafka.brokerConnect=<host:port,host:port>,...
```

1. Replace `kafdrop-<version>.jar` with the specific version, such as `kafdrop-4.0.2-SNAPSHOT.jar`.

2. Use `--kafka.brokerConnect=<host:port,host:port>` to specify the host and port for the specific cluster broker nodes.

Console startup effect is shown below:

<img />

If not specified, `kafka.brokerConnect` defaults to `localhost:9092`.

**Note:** Starting from Kafdrop 3.10.0, a ZooKeeper connection is no longer required. All necessary cluster information is retrieved through the Kafka management API.

Open a browser and navigate to [http://localhost:9000](http://localhost:9000/). The port can be overridden by adding the following configuration:

```text theme={null}

--server.port=<port> --management.server.port=<port>

```

## Final Output

1. Complete Interface

<img />

Displays the number of partitions, number of topics, and other cluster status information.

2. Create New Topic Feature

<img />

3. Details on Broker Node

<img />

4. Details on Topic

<img />

5. Information on Messages Within Topic

<img />

## Summary

This tutorial examines the key features and functionalities of Kafdrop and explains how to integrate it with AutoMQ clusters to easily monitor and manage those clusters. Utilizing Kafdrop not only aids teams in better understanding and controlling their data flows but also enhances development and operational efficiency, ensuring effective and stable data processing. We hope this tutorial offers valuable insights and assistance when using Kafdrop with AutoMQ clusters.

## References

\[1] Kafdrop: [https://github.com/obsidiandynamics/kafdrop](https://github.com/obsidiandynamics/kafdrop)

\[2] Deploy in Cluster Mode on Linux Hosts | AutoMQ: [https://docs.automq.com/automq/getting-started/cluster-deployment-on-linux](https://docs.automq.com/automq/getting-started/cluster-deployment-on-linux)

\[3] Kafdrop Deployment Methods: [https://github.com/obsidiandynamics/kafdrop/blob/master/README.md#getting-started](https://github.com/obsidiandynamics/kafdrop/blob/master/README.md#getting-started)

\[4] Kafdrop project repository: [https://github.com/obsidiandynamics/kafdrop](https://github.com/obsidiandynamics/kafdrop)


# Redpanda Console
Source: https://docs.automq.com/automq/integrations/kafka-ui/redpanda-console

AutoMQ offers cloud-native scalability and Kafka compatibility for cluster management. Integrate with Redpanda Console for real-time monitoring and efficient data handling.

<Tip>
  You can reference the relevant code from [AutoMQ Labs](https://github.com/AutoMQ/automq-labs), which includes integration examples of AutoMQ and Redpanda Console [here](https://github.com/AutoMQ/automq-labs/tree/main/opensource-examples/ui/redpanda-console). Launch with a single click to experience it.
</Tip>

### Manage Kafka/AutoMQ Clusters More Conveniently Using the Kafka Web UI.

With the rapid advancement of big data technologies, Kafka has become a cornerstone for real-time data processing in enterprises as a high-throughput, low-latency distributed messaging system. However, managing and monitoring Kafka clusters is not an easy task. Traditional command-line tools and scripts, while powerful, can be complex and unintuitive for developers and operations personnel. To address these challenges, the Kafka Web UI has emerged, providing users with a more convenient and efficient way to manage Kafka clusters.

After over a decade of development, Apache Kafka has amassed a wealth of ecosystem resources within the industry. AutoMQ, as a successor to Apache Kafka, can leverage products within its ecosystem due to its full compatibility with Kafka. AutoMQ BYOC and AutoMQ Software both offer substantial control capabilities through AutoMQ Console. If you are using AutoMQ, you can also utilize products like Kafdrop, Redpanda Console \[1], and similar tools to manage AutoMQ clusters.

Today's content covers how to monitor the status of AutoMQ clusters using [Redpanda Console](https://www.redpanda.com/redpanda-console-kafka-ui) \[1] to enhance system maintainability and stability.

### Integration of AutoMQ with Redpanda Console

Redpanda Console is a Kafka Web UI provided by [Redpanda](https://www.redpanda.com/), designed for monitoring and managing Redpanda or Kafka clusters. It offers an intuitive user interface through which users can easily view cluster status, monitor performance metrics, and manage topics and partitions. This console is aimed at simplifying the day-to-day operations of data streaming systems, enabling users to maintain and monitor their clusters more effectively.

Thanks to AutoMQ being fully compatible with Kafka, it integrates with Redpanda Console. By leveraging Redpanda Console, AutoMQ users can also benefit from an intuitive user interface to monitor the real-time status of AutoMQ clusters, including vital metadata such as topics, partitions, consumer groups, and their offsets. This monitoring capability not only enhances the efficiency of problem diagnosis but also helps optimize cluster performance and resource usage.

This tutorial will guide you on how to launch the Redpanda Console service and use it alongside AutoMQ clusters to effectively monitor and manage cluster status.

<img alt="Redpanda Console" />

## Prerequisites

* Deploy AutoMQ Cluster

* Prepare Redpanda Console Environment

### Deploy AutoMQ Cluster

Refer to the official AutoMQ documentation: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) \[3].

### Deploy Redpanda Console

There are two ways to deploy the Redpanda Console: Docker deployment and distribution deployment. Docker deployment is simpler, and if you want a quick and easy experience integrating AutoMQ with the Redpanda Console, it is recommended to choose Docker for deployment. If you have specific requirements, such as login authentication, SASL authentication, TLS configuration, and log level settings, you should opt for distribution deployment. Below, I'll introduce the configuration methods for both.

#### Docker Deployment

Redpanda Console can be deployed via Docker, referring to [Quick Start](https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start) \[4]. In the process described above, after setting up the AutoMQ cluster, you will know the addresses and ports that all Broker nodes listen to. Thus, you can establish an association between the Redpanda Console and the AutoMQ cluster by specifying the `KAFKA_BROKERS` parameter in the Docker startup command. The Docker container startup command is as follows:

```bash theme={null}
docker run -p 8080:8080 -e KAFKA_BROKERS=192.168.0.4:9092,192.168.0.5:9092,192.168.0.6:9092 docker.redpanda.com/redpandadata/console:latest
```

* `-p 8080:8080` : Specifies the service access mapping port for the Redpanda Console.

* `KAFKA_BROKERS` : Must be specified as the Broker address of your AutoMQ cluster.

#### Deployment of Distribution

You need to choose an appropriate version to download and extract it into a specified directory, such as `/opt`, from Redpanda Console's GitHub Releases page: [Release Redpanda Console](https://github.com/redpanda-data/console/releases/tag/v2.6.0) \[5]. The command is as follows:

```bash theme={null}
# Ubuntu Linux
cd /opt
sudo curl -L -o redpanda_console.tar.gz https://github.com/redpanda-data/console/releases/download/v2.6.0/redpanda_console_2.6.0_linux_amd64.tar.gz
# Unzip, Get Redpanda_console
sudo tar -xzf redpanda_console.tar.gz
# Config Set
sudo mkdir -p /etc/redpanda
# Write Config
sudo vim /etc/redpanda/redpanda-console-config.yaml
```

An example of the contents for the `redpanda-console-config.yaml` configuration file is shown below:

```yaml theme={null}
kafka:
  #Brokers is a list of bootstrap servers with
  #port (for example "localhost:9092").
  brokers:
    - broker-0.mycompany.com:19092
    - broker-1.mycompany.com:19092
    - broker-2.mycompany.com:19092

```

> Note: Ensure that the server where you are installing Redpanda Console has access to the server where the Broker node mentioned in the configuration file is located.

For more detailed settings, refer to the official documentation: [Redpanda Console Configuration](https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file) \[6]. After completing the configuration, you need to set environment variables so that the Redpanda Console executable can access the configuration file information and start the Redpanda Console:

```bash theme={null}
# Set Env
export CONFIG_FILEPATH="/etc/redpanda/redpanda-console-config.yaml"
# /opt/ Run Console
./redpanda-console
```

You will get the following results:

```json theme={null}
./redpanda-console
{"level":"info","ts":"2024-07-10T09:52:52.958+0800","msg":"started Redpanda Console","version":"2.6.0","built_at":"1717083695"}
{"level":"info","ts":"2024-07-10T09:52:52.963+0800","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2024-07-10T09:52:54.780+0800","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":2,"controller_id":0,"kafka_version":"at least v3.6"}
{"level":"info","ts":"2024-07-10T09:53:05.620+0800","msg":"Server listening on address","address":"[::]:8080","port":8080}
```

## Access the Console Page

After completing the above deployment operations, you can access the console service by entering the address in your browser (e.g., [http://console\_ip:8080](http://localhost:8080)). The display will appear as follows:

### **Cluster Overview**

The Cluster Overview page provides users with a broad perspective, displaying the core information of the AutoMQ cluster, including but not limited to:

* Cluster Operational Status: Displays the current health of the cluster, helping users quickly identify any issues.

* Storage Utilization: Indicates data storage usage within the cluster, aiding users in managing and planning storage effectively.

* Version Information: Shows the version of the running AutoMQ cluster, simplifying the tracking and upgrading process.

* Number of Online Brokers: Displays the real-time count of online brokers, an essential metric.

* Number of Topics and Replicas: Provides information on the number of topics and replicas, assisting users in understanding the cluster's scale and data replication status.

<img />

Monitoring the state of a cluster is essential for maintaining the stability and performance of messaging queue systems. By observing real-time metrics, such as the cluster's operational status, storage utilization, version information, the number of online brokers, and the count of topics and replicas, operations personnel can swiftly detect and address potential issues, preventing system failures from affecting business operations. Furthermore, these metrics support capacity planning and resource management, ensuring the system is prepared to manage future data growth. Having access to cluster version information also enables users to upgrade software promptly to benefit from the latest features and security patches, thereby enhancing overall system reliability and efficiency.

### **Topic Overview**

On the Topic list page, users can see an overview of all Topics in the current AutoMQ cluster, including essential details for each Topic, such as the number of partitions and replica strategy. This page allows users to efficiently browse and manage Topics.

<img />

### **Topic Details**

By clicking on a specific Topic, users navigate to a detailed page for that Topic, where they can further explore understanding and operational functionalities:

* Message List: Browsing and searching messages within a Topic is highly beneficial for tracking and debugging purposes.

* Consumer Information: Displays information on consumers and consumer groups currently subscribed to the Topic, facilitating the monitoring of consumption status.

* Partition Status: Provides detailed information for each partition, including key metrics like Leader and ISR.

* Configuration Information: Lists the configuration settings of the Topic, with support for modifications to optimize performance or behavior.

* ACL (Access Control List): Manages access permissions for the Topic to ensure data security.

Furthermore, Redpanda Console also supports users in manually creating and publishing messages, which is highly valuable for message injection during testing or specific scenarios.

<img />

Monitoring topic details allows us to gain insight into the operational status of the message queue by navigating through the message list for tracking and debugging, checking consumer information to monitor consumption status, understanding partition state to ensure data distribution and high availability, managing configuration information to optimize performance, and setting access controls to safeguard data security. These features aid in the timely identification and resolution of issues, enhancing the overall efficiency and reliability of the system.

## Summary

This article explains the integration process of Redpanda Console with AutoMQ, demonstrating how this powerful tool can simplify and enhance the management of AutoMQ clusters. Hopefully, this article can serve as a practical reference for users aiming to improve the efficiency and functionality of message queue management.

## References

\[1] Redpanda Console: [https://www.redpanda.com/redpanda-console-kafka-ui](https://www.redpanda.com/redpanda-console-kafka-ui)

\[2] Redpanda: [https://www.redpanda.com/](https://www.redpanda.com/)

\[3] Cluster Deployment for AutoMQ: [https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker](https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

\[4] Quick Start: [https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start](https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start)

\[5] Release Redpanda Console: [https://github.com/redpanda-data/console/releases/tag/v2.6.0](https://github.com/redpanda-data/console/releases/tag/v2.6.0)

\[6] Redpanda Console Configuration: [https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file](https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file)

\[7] Kafdrop Github:  [https://github.com/obsidiandynamics/kafdrop](https://github.com/obsidiandynamics/kafdrop)


# Flashcat
Source: https://docs.automq.com/automq/integrations/observability/flashcat

AutoMQ offers cloud-native scalability, Kafka compatibility, and cost efficiency by integrating with Nightingale for robust, real-time monitoring and performance optimization.

## Introduction

Monitoring your AutoMQ cluster helps you maintain stability and optimize performance. This guide shows how to integrate AutoMQ with the Nightingale monitoring system.

### Overview of Nightingale

The Nightingale Monitoring System (Nightingale) is an open-source, cloud-native observability and analysis tool that adopts an All-in-One design philosophy, integrating data collection, visualization, monitoring alerts, and data analysis into one platform. Its main advantages include efficient data collection capabilities, flexible alert strategies, and rich visualization features. Nightingale is closely integrated with various cloud-native ecosystems, supporting multiple data sources and storage backends, and providing low-latency, high-reliability monitoring services. By using Nightingale, enterprises can achieve comprehensive monitoring and management of complex distributed systems, quickly identify and resolve issues, thereby optimizing system performance and enhancing business continuity.

<img alt="Nightingale Architecture [3]" />

## Prerequisites

To monitor the status of the cluster, you will need the following setup:

* Deploy an available AutoMQ node/cluster and open the Metrics collection port

* Deploy Nightingale monitoring and its dependencies

* Deploy [Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) to collect Metrics data

## Deploy AutoMQ, Prometheus, and Nightingale Monitoring

### Deploy AutoMQ

Refer to the AutoMQ documentation: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux). Before deployment, add the following configuration parameters to enable the Prometheus pull interface. After starting the AutoMQ cluster with these parameters, each node will additionally open an HTTP interface to fetch AutoMQ monitoring metrics. These metrics follow the format of Prometheus Metrics.

```bash theme={null}
bin/kafka-server-start.sh ...\
--override  s3.telemetry.metrics.exporter.type=prometheus \
--override  s3.metrics.exporter.prom.host=0.0.0.0 \
--override  s3.metrics.exporter.prom.port=8890 \
....
```

Once AutoMQ monitoring metrics are enabled, metrics in Prometheus format can be pulled via HTTP from any node using the address: `http://{node_ip}:8890`. An example of the response results is as follows:

```text theme={null}

....
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="DescribeDelegationToken"} 0.0 1720520709290
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="CreatePartitions"} 0.0 1720520709290
...

```

For more information about the metrics, refer to the AutoMQ official documentation: [Prometheus Metrics▸](/automq/observability/prometheus-metrics).

### Deploying Prometheus

Prometheus can be deployed by downloading the binary package or using Docker. Below is an introduction to these two deployment methods.

#### Binary Deployment

For ease of use, you can create a new script and modify the Prometheus download version as needed. Then, execute the script to complete the deployment. First, create a new script:

```bash theme={null}
cd /home
vim install_prometheus.sh
# !!! Paste the Following Script Content and Save and Exit
# Grant Permissions
chmod +x install_prometheus.sh
# Execute the Script
./install_prometheus.sh
```

```bash theme={null}
version=2.45.3
filename=prometheus-${version}.linux-amd64
mkdir -p /opt/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v${version}/${filename}.tar.gz
tar xf ${filename}.tar.gz
cp -far ${filename}/*  /opt/prometheus/

# Config as a Service
cat <<EOF >/etc/systemd/system/prometheus.service
[Unit]
Description="prometheus"
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple

ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --web.enable-lifecycle --web.enable-remote-write-receiver

Restart=on-failure
SuccessExitStatus=0
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=prometheus


[Install]
WantedBy=multi-user.target
EOF

systemctl enable prometheus
systemctl restart prometheus
systemctl status prometheus

```

Subsequently, modify the Prometheus configuration file to **add a task to collect observability data from AutoMQ** and **restart** Prometheus by executing the following command:

```bash theme={null}
# Add the Following Content to the Configuration File:
vim /opt/prometheus/prometheus.yml
# Restart Prometheus
systemctl restart prometheus
```

**Refer to the following configuration file content**, please modify the `client_ip` to the AutoMQ observable data exposure address:

```yaml theme={null}
# My Global Config
global:
  scrape_interval: 15s # Set the Scrape Interval to Every 15 Seconds. Default Is Every 1 Minute.
  evaluation_interval: 15s # Evaluate Rules Every 15 Seconds. the Default Is Every 1 Minute.

scrape_configs:
  # The Job Name Is Added as a Label `job=<job_name>` to Any Timeseries Scraped from This Config.
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
      
  - job_name: "automq"
    static_configs:
      - targets: ["{client_ip}:8890"]

```

Once the deployment is completed, you can access Prometheus through a browser to determine if AutoMQ's Metrics data has been successfully collected by visiting `http://{client_ip}:9090/targets`:

<img />

#### Docker Deployment

If you already have a running Prometheus Docker container, please execute the following command to remove the container:

```bash theme={null}
docker stop prometheus
docker rm prometheus
```

Create a new configuration file and mount it during Docker startup:

```bash theme={null}
mkdir -p /opt/prometheus
vim /opt/prometheus/prometheus.yml
# Refer to the Configuration Mentioned in the "Binary Deployment" Section Above for the Content
```

Start the Docker container:

```bash theme={null}
docker run -d \
  --name=prometheus \
  -p 9090:9090 \
  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -m 500m \
  prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=otlp-write-receiver \
  --web.enable-remote-write-receiver
```

This will direct you to a Prometheus service that collects AutoMQ Metrics. For more information on integrating AutoMQ Metrics with Prometheus, refer to: [Integrating Metrics with Prometheus▸](/automq/observability/integrating-metrics-with-prometheus).

### Deploy Nightingale Monitoring

Nightingale monitoring can be deployed in the following three ways. For more detailed deployment instructions, refer to the [official documentation](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/install/intro/) \[5]:

* Deploy using Docker compose

* Deploy using binary

* Helm Deployment

Next, I will proceed with the deployment using a binary method.

#### Download Nightingale

Select the appropriate version to download from the Nightingale [GitHub releases](https://github.com/ccfos/nightingale) \[6] page. The version used here is `v7.0.0-beta.14`. If you are on an amd architecture machine, you can directly execute the following command:

```bash theme={null}
cd /home
# Download
wget https://github.com/ccfos/nightingale/releases/download/v7.0.0-beta.14/n9e-v7.0.0-beta.14-linux-amd64.tar.gz
mkdir -p /home/flashcat
# Extract the Files to the /home/flashcat Folder
tar -xzf /home/n9e-v7.0.0-beta.14-linux-amd64.tar.gz -C /home/flashcat
# Navigate to the Home Directory
cd /home/flashcat
```

#### Configure the Dependency Environment

Nightingale depends on MySQL and Redis, so you need to install these environments beforehand. You can deploy them via Docker or by executing the commands as follows:

```bash theme={null}
# Install Mysql
yum -y install mariadb*
systemctl enable mariadb
systemctl restart mariadb
mysql -e "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('1234');"

# Install Redis
yum install -y redis
systemctl enable redis
systemctl restart redis
```

Here, Redis is set to no password. Additionally, the MySQL database password is specified as `1234`. **If you need to change to another password**, you need to configure it in the Nightingale configuration file to ensure Nightingale can connect to your database. Modify the Nightingale configuration file:

```json theme={null}
vim /home/flashcat/etc/config.toml

Modify the username and password under [DB]:
[DB]
# Postgres: Host=%s Port=%s User=%s Dbname=%s Password=%s Sslmode=%s
# Postgres: DSN="host=127.0.0.1 Port=5432 User=root Dbname=n9e_v6 Password=1234 Sslmode=disable"
# Sqlite: DSN="/path/to/filename.db"
DSN = "{username}:{password}@tcp(127.0.0.1:3306)/n9e_v6?charset=utf8mb4&parseTime=True&loc=Local&allowNativePasswords=true"
# Enable Debug Mode or Not
```

#### Import Database Tables

Execute the following command:

```bash theme={null}

mysql -uroot -p1234 < n9e.sql

```

Use a database tool to check if the database tables were successfully imported:

```text theme={null}

> show databases;
+--------------------+
| Database           |
+--------------------+
| n9e_v6             |
+--------------------+

> show tables;
+-----------------------+
| Tables_in_n9e_v6      |
+-----------------------+
| alert_aggr_view       |
| alert_cur_event       |
| alert_his_event       |
| alert_mute            |
| alert_rule            |
| alert_subscribe       |
| alerting_engines      |
| board                 |
| board_busigroup       |
| board_payload         |
| builtin_cate          |
| builtin_components    |
| builtin_metrics       |
······

```

#### Modify the Nightingale Configuration File

You need to modify the Nightingale configuration file to set up the Prometheus data source:

```json theme={null}
vim /home/flashcat/etc/config.toml

# Modify the [[Pushgw.Writers]] Section to
[[Pushgw.Writers]]
# Url = "http://127.0.0.1:8480/insert/0/prometheus/api/v1/write"
Url = "http://{client_ip}:9090/api/v1/write"
```

#### Start Nightingale

In the root directory of Nightingale `/home/flashcat`, execute: `./n9e`. After a successful start, you can access it in your browser at `http://{client_ip}:17000`, with the default login username and password being:

* Username: `root`yaml
* Password: `root.2020`bash

<img />

#### Integrate Prometheus Data Source

Left sidebar Integration -> Data Source -> Prometheus.

<img alt="At this point, Nightingale monitoring has been deployed." />

At this point, the Nightingale monitoring deployment is complete.

## Nightingale Monitoring AutoMQ Cluster Status

Next, we will introduce some of the features provided by Nightingale monitoring to help you better understand the available features integrated with AutoMQ.

### Instant Query

Select built-in AutoMQ metrics:

<img alt="You can try querying some data, like the average processing time for Fetch requests." />

You can try querying some data, such as the average fetch request processing time
`kafka_request_time_50p_milliseconds`:

<img />

<img alt="Furthermore, you can customize some metrics and aggregate them with expressions:" />

Additionally, you can customize some metrics and use expressions to aggregate these metrics:

<img />

### Alerting Feature

Select from the left sidebar Alerts -> Alert Rules -> Create New Rule. For example, you can set an alert for `kafka_network_io_bytes_total`, which measures the total number of bytes sent or received by Kafka Broker nodes over the network. By setting an expression for this metric, you can calculate the inbound network I/O rate for Kafka Broker nodes. The expression is:

```text theme={null}

sum by(job, instance) (rate(kafka_network_io_bytes_total{direction="in"}[1m]))

```

Setting Alert Rules:

<img alt="Data preview: " />

Data preview:

<img alt="You can also configure groups to receive notifications when an alert is triggered." />

You can also configure groups to be notified when an alert occurs:

<img alt="After setting up an alert," />

After creating the alert, you can simulate a high-concurrency message processing scenario: a total of `2500000` messages are sent to AutoMQ nodes within a short period. The method used is sending messages through the Kafka SDK, with a total of 50 Topics, each Topic receiving 500 messages, repeated 100 times. An example is as follows:

```bash theme={null}
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class KafkaTest {

    private static final String BOOTSTRAP_SERVERS = "http://{}:9092"; // your automq broker ip
    private static final int NUM_TOPICS = 50;
    private static final int NUM_MESSAGES = 500;

    public static void main(String[] args) throws Exception {
        KafkaTest test = new KafkaTest();
        // test.createTopics();    // create 50 topics
        for(int i = 0; i < 100; i++){
            test.sendMessages();   // 25,000 messages will be sent each time, and 500 messages will be sent to each of 50 topics.
        }
    }
    
    public void createTopics() {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);

        try (AdminClient adminClient = AdminClient.create(props)) {
            List<NewTopic> topics = new ArrayList<>();
            for (int i = 1; i <= NUM_TOPICS; i++) {
                topics.add(new NewTopic("Topic-" + i, 1, (short) 1));
            }
            adminClient.createTopics(topics).all().get();
            System.out.println("Topics created successfully");
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        }
    }

    public void sendMessages() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 1; i <= NUM_TOPICS; i++) {
                String topic = "Topic-" + i;
                for (int j = 1; j <= NUM_MESSAGES; j++) {
                    String key = "key-" + j;
                    String value = "{\"userId\": " + j + ", \"action\": \"visit\", \"timestamp\": " + System.currentTimeMillis() + "}";
                    ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
                    producer.send(record, (RecordMetadata metadata, Exception exception) -> {
                        if (exception == null) {
                            System.out.printf("Sent message to topic %s partition %d with offset %d%n", metadata.topic(), metadata.partition(), metadata.offset());
                        } else {
                            exception.printStackTrace();
                        }
                    });
                }
            }
            System.out.println("Messages sent successfully");
        }
    }
}
```

You can then see the alarm information in the Nightingale console:

<img alt="Alert Details:" />

Alert Details:

<img />

### Dashboard

First, you can use the known metrics to create your own dashboard. Below is an example of a statistical dashboard for AutoMQ message request processing time, total message count, and network IO bits:

<img alt="Additionally, you can use the built-in official dashboard for monitoring. Left sidebar -> Aggregation -> Template Center:" />

Additionally, you can use the built-in dashboards provided by the official source. Left sidebar -> Aggregate -> Template Center:

<img alt="Select AutoMQ, and you'll see several Dashboard options available:" />

Choosing AutoMQ, you will see several DashBoards available:

<img alt="Select the Topic Metrics dashboard, and the displayed content is shown below:" />

Select the Topic Metrics dashboard, and the content is displayed as follows:

<img alt="This showcases the message input and output utilization, input and request rates, message size, and additional metrics for the AutoMQ cluster over a recent period. These metrics are crucial for monitoring and optimizing the performance and stability of the AutoMQ cluster: By analyzing message input and output utilization, you can assess the load on producers and consumers, ensuring the cluster can handle message traffic effectively; the input rate helps monitor the rate at which producers send messages in real-time, thus identifying any potential bottlenecks or traffic surges; request rates help understand the frequency of client requests, optimizing resource allocation and processing capacity; the message size metric allows for analysis of the average message size, enabling configuration adjustments to enhance storage and network transmission efficiency. By monitoring these metrics, you can quickly detect and resolve performance issues, ensuring the efficient and stable operation of the AutoMQ cluster." />

This dashboard shows the message input and output utilization, message input and request rates, and message sizes of the AutoMQ cluster over a recent period. These metrics are used to monitor and optimize the performance and stability of the AutoMQ cluster: By assessing the message input and output utilization, you can evaluate the load on producers and consumers, ensuring the cluster can handle the message flow properly. The message input rate is used for real-time monitoring of the rate at which producers send messages, identifying potential bottlenecks or sudden spikes in traffic. The request rate helps understand the frequency of client requests, optimizing resource allocation and processing capabilities. The message size metric analyzes the average size of messages to adjust configurations for optimizing storage and network transmission efficiency. Monitoring these metrics allows you to detect and resolve performance issues promptly, ensuring the efficient and stable operation of the AutoMQ cluster.

The integration process of FlashCat is now complete. For additional usage options, please refer to Nightingale's [official documentation](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/overview/) \[7] to explore and experience further.

## Summary

This article elaborates on how to comprehensively monitor an AutoMQ cluster using the Nightingale monitoring system. Starting with the basic concepts of AutoMQ and Nightingale, it gradually explains how to deploy AutoMQ, Prometheus, and Nightingale, and configure monitoring and alarm rules. Through this integration, enterprises can monitor the operational status of the AutoMQ cluster in real-time, promptly identify and resolve potential issues, optimize system performance, and ensure business continuity and stability. The Nightingale monitoring system, with its powerful data collection capabilities, flexible alerting mechanisms, and rich visualization features, becomes an ideal choice for enterprises to monitor complex distributed systems. We hope this article provides valuable reference for your practical application, helping to make your system operations more efficient and stable.

## References

\[1] AutoMQ: [https://www.automq.com/](https://www.automq.com/)

\[2] Nightingale Monitoring: [https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/introduction/](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/introduction/)

\[3] Nightingale Architecture: [https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/introduction/](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/introduction/)

\[4] Prometheus: [https://prometheus.io/docs/prometheus/latest/getting\_started/](https://prometheus.io/docs/prometheus/latest/getting_started/)

\[5] Deployment Instructions: [https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/install/intro/](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/install/intro/)

\[6] Nightingale GitHub releases: [https://github.com/ccfos/nightingale](https://github.com/ccfos/nightingale)

\[7] Nightingale Official Documentation: [https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/overview/](https://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v7/overview/)


# Guance Cloud
Source: https://docs.automq.com/automq/integrations/observability/guance-cloud

AutoMQ offers Kafka compatibility, cloud-native scalability, and cost efficiency, enabling automatic fault recovery and high availability without human intervention.

## Preface

### Guance Cloud

[Observing Cloud](https://docs.guance.com/product-introduction/) \[1] is a unified real-time monitoring application designed for cloud platforms, cloud-native environments, applications, and business-related needs. It integrates three major signals: metrics, logs, and tracing, covering testing, prerelease, and production environments to achieve observability throughout the software development lifecycle. Through Observing Cloud, enterprises can build complete application full-link observability, enhancing the overall transparency and controllability of the IT architecture.

As a powerful data analytics platform, Guance Cloud includes several core modules, such as DataKit \[2], a unified data collector, and the DataFlux Func data processing development platform.

<img alt="Data Flow Observability" />

### AutoMQ

[AutoMQ](https://www.automq.com/zh) \[3] is a new generation of Apache Kafka® distribution redesigned based on cloud-native concepts. It offers cloud-native scalability and cost-efficient resource usage while remaining 100% compatible with the Apache Kafka protocol. AutoMQ stores data entirely on S3, allowing for rapid response to sudden traffic spikes without the need for data replication during cluster expansion. In contrast, Apache Kafka requires significant bandwidth for partition data replication after scaling, making it difficult to handle sudden traffic. AutoMQ achieves high system autonomy through features like automatic scaling, self-balancing, and automatic fault recovery, achieving higher availability without human intervention. AutoMQ's shared storage architecture:

<img />

## Observability Interfaces of AutoMQ

Due to AutoMQ's full compatibility with Kafka and support for open Prometheus-based metrics collection ports, the data collection tool DataKit provided by Observing Cloud can be integrated with the Observing Cloud platform. This facilitates users to monitor and manage the state of AutoMQ clusters. Additionally, the Observing Cloud platform supports user-defined aggregation and querying of metrics data. By using predefined dashboard templates or custom dashboards, various statistics of the AutoMQ cluster, such as common topics, brokers, partitions, and group statistics, can be effectively summarized.

Furthermore, based on metrics observability data, it is also possible to query error conditions encountered during the operation of the AutoMQ cluster and various utilization metrics of the current system, such as JVM CPU usage, JVM heap usage, and cache size. These metrics can help quickly identify and resolve issues when anomalies occur in the cluster, which is very beneficial for high availability and rapid recovery of the system. Next, this article will introduce how to use the Observing Cloud platform to monitor the state of the AutoMQ cluster.

## Steps to Integrate with Observation Cloud

### Enable Metric Fetching Interface in AutoMQ

Refer to the AutoMQ documentation: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux) Before starting the deployment, add the following configuration parameters to enable Prometheus pull interfaces. After starting the AutoMQ cluster with these parameters, each node will further expose an HTTP interface for pulling AutoMQ monitoring metrics. The format of these metrics follows the Prometheus Metrics format.

```bash theme={null}
bin/kafka-server-start.sh ...\
--override  s3.telemetry.metrics.exporter.type=prometheus \
--override  s3.metrics.exporter.prom.host=0.0.0.0 \
--override  s3.metrics.exporter.prom.port=8890 \
....
```

Once AutoMQ monitoring metrics are enabled, the Prometheus format monitoring metrics can be pulled via the HTTP protocol on any node. The address is: `http://{node_ip}:8890`. An example of the response is as follows:

```text theme={null}

....
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="DescribeDelegationToken"} 0.0 1720520709290
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="CreatePartitions"} 0.0 1720520709290
...

```

For an introduction to metrics, refer to the AutoMQ official documentation: [Prometheus Metrics▸](/automq/observability/integrating-metrics-with-prometheus/prometheus-metrics).

### Install and Configure DataKit Collection Tool

DataKit is an open-source monitoring collection tool provided by Observing Cloud that supports Prometheus metrics scraping. You can use DataKit to pull monitoring data from AutoMQ and aggregate it to the Observing Cloud platform.

#### Install the DataKit Tool

> Tips: For more details on DataKit installation, refer to the documentation: [Host Installation - Guance Documentation](https://docs.guance.com/datakit/datakit-install/) \[6].

First, register for a Guance account and log in. Then, in the main interface, click "Integration" on the left side and select "DataKit" at the top to see the DataKit installation command.

```bash theme={null}
DK_DATAWAY="https://openway.guance.com?token=<TOKEN>" bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"
```

Copy the above command and run it on all nodes in the cluster to complete the DataKit installation.

> Tips: DataKit needs to be installed on all Brokers in the cluster that need to be monitored.

After successfully executing the installation command, use the command `datakit monitor` to verify if DataKit is successfully installed.

<img />

#### AutoMQ Collector Configuration and Activation

Here, you need to configure the DataKit AutoMQ collector on the server of each node where data is to be collected. Enter the directory `/usr/local/datakit/conf.d/prom` and create a collector configuration file `prom.conf`. The collector configuration will include the open observable data interfaces, collector name, prom instance name, and important collection interval, etc. You can make some adjustments to the configuration on each server as needed:

```bash theme={null}
[[inputs.prom]]
  
  urls = ["http://clientIP:8890/metrics"]   # ClientIP Should Be Your Own Server Address.
  source = "AutoMQ"

  ## Keep Exist Metric Name
  ## If the Keep_exist_metric_name Is True, Keep the Raw Value for Field Names.
  keep_exist_metric_name = true

  [inputs.prom.tags_rename]
    overwrite_exist_tags = true

  [inputs.prom.tags_rename.mapping]
    service_name = "job"
    service_instance_id = "instance"

  [inputs.prom.tags]
    component="AutoMQ"
  interval = "10s"
```

Parameter Adjustment Instructions:

| urls<br />     | AutoMQ metrics address, provide the corresponding metrics URL exposed by the component here<br /> |
| -------------- | ------------------------------------------------------------------------------------------------- |
| source<br />   | Collector alias, it is recommended to distinguish it<br />                                        |
| interval<br /> | Collection interval, specifies the interval at which the collector collects data<br />            |

Execute `datakit service -R` to restart the DataKit on that node to activate the collector configuration.

## Monitor AutoMQ Clusters Via GUI-Based Management

The monitoring platform has integrated AutoMQ and provides multiple default dashboards. You can view the [Dashboard Examples](https://console.guance.com/scene/dashboard/createDashboard?w=wksp_63b96920660e4962a07429b65ef163e7\&lak=Scene)\[7]. Below are examples of several commonly used templates, and we will introduce their functions:

### Cluster Monitoring

The main display shows the number of currently active Brokers, the total number of Topics, Partitions, etc. Additionally, you can specify which node to query specifically by selecting in the Cluster\_id.

<img alt="By monitoring the status of the Kafka cluster, potential issues such as node failures, insufficient disk space, and network latency can be quickly identified and resolved to ensure a more controllable and stable system." />

By monitoring the status of the Kafka cluster, potential issues such as node failures, insufficient disk space, network delays, etc., can be detected and resolved promptly to ensure a more controllable and stable system.

### Broker Monitoring

The cloud observability `AutoMQ Broker` dashboard describes several metrics of all Brokers, such as the number of connections, partitions, the number of messages received per second (ops), and the amount of data input and output per second, measured in Bytes.

<img />

### Topic Monitoring

This section provides an overview of all the Topics contained within all nodes. As previously mentioned, you can also specify a node to query the Topic information. These metrics primarily include the space occupied by each Topic, the number of messages received, and Request Throughput, which indicates the ability to process requests per unit of time.

<img alt="Currently, the state of the AutoMQ cluster is successfully monitored through Observability Cloud, with dashboard data obtained via aggregation or query of metrics indicators." />

At this point, the cloud observability has successfully monitored the AutoMQ cluster status, and the data in the dashboard is obtained by aggregating or querying the Metrics indicators.

## Summary

This article introduced how to integrate the cloud observability platform with AutoMQ to monitor the status information of the AutoMQ cluster. There are actually more advanced operations, such as custom alert functions, custom data queries, etc., which can be done DIY according to the rules provided officially. You can experiment with these operations manually, and you will find the ones that suit your needs.

## References

\[1] Cloud Observability: [https://docs.guance.com/product-introduction/](https://docs.guance.com/product-introduction/)

\[2] DataKit: [https://docs.guance.com/datakit/](https://docs.guance.com/datakit/)

\[3] AutoMQ: [https://www.automq.com](https://www.automq.com)

\[4] Host Installation - Observability Cloud Documentation: [https://docs.guance.com/datakit/datakit-install/](https://docs.guance.com/datakit/datakit-install/)

\[5] Dashboard Example: [https://console.guance.com/scene/dashboard/createDashboard?w=wksp\_63b96920660e4962a07429b65ef163e7\&lak=Scene](https://console.guance.com/scene/dashboard/createDashboard?w=wksp_63b96920660e4962a07429b65ef163e7\&lak=Scene)


# Prometheus/VictoriaMetrics
Source: https://docs.automq.com/automq/integrations/observability/prometheus

Optimize AutoMQ clusters with Prometheus and VictoriaMetrics for cloud-native scalability, Kafka compatibility, and reduced storage costs.

## Introduction

Monitoring your AutoMQ cluster helps you detect issues early and optimize performance. This guide shows how to integrate Prometheus \[2] and VictoriaMetrics \[3] to collect, store, and query AutoMQ metrics.

### AutoMQ Overview

AutoMQ is a Kafka-compatible streaming platform \[1]. See [Overview](/automq/what-is-automq) for details.

<img />

### Prometheus Overview

Prometheus is an open-source system monitoring and alerting toolkit initially developed by SoundCloud and released as open source in 2012. It offers robust data collection, storage, and querying capabilities, enabling real-time monitoring of system and application performance metrics. Prometheus utilizes a multidimensional data model that uses labels to tag and query monitoring data, and it supports the flexible query language PromQL. The primary advantages of Prometheus include efficient data collection, strong query functionality, and a rich alerting mechanism, making it widely used in the cloud-native ecosystem.

However, Prometheus has certain limitations in large-scale data storage, particularly when it comes to long-term storage and high-concurrency querying. Its storage performance and query efficiency may be affected. Consequently, integrating an efficient storage backend becomes essential to address these challenges.

### VictoriaMetrics Overview

VictoriaMetrics is a high-performance, open-source time-series database designed for high-concurrency data writing and querying. It features efficient data compression and storage capabilities, offering high throughput for data writes and queries with minimal hardware resource consumption. VictoriaMetrics supports Prometheus' remote storage interface, allowing seamless integration into existing Prometheus monitoring systems. Below is an architecture diagram of VictoriaMetrics:

<img alt="VictoriaMetrics Structure [4]" />

By utilizing VictoriaMetrics as the storage backend for Prometheus, organizations can significantly improve the storage and query performance of their monitoring systems, fulfilling the requirements for large-scale data storage and high-concurrency queries. The primary advantages of VictoriaMetrics include efficient data compression, rapid data querying, and low resource consumption, making it an excellent option as a storage backend for Prometheus. Below, I will proceed with the deployment of AutoMQ, Prometheus, and VictoriaMetrics.

## Prerequisites

* Deploy a functional AutoMQ node/cluster and open the Metrics collection port

* Deploy VictoriaMetrics as the storage backend for Prometheus

* Deploy Prometheus to retrieve Metrics data

## Deploying AutoMQ, VictoriaMetrics, Prometheus

### Deploying AutoMQ

Refer to the AutoMQ documentation: [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux). Before starting deployment, add the following configuration parameters to enable Prometheus's pull interface. Once the AutoMQ cluster is launched with these parameters, each node will additionally open an HTTP interface for retrieving AutoMQ monitoring metrics. These metrics comply with the Prometheus Metrics format.

```bash theme={null}
bin/kafka-server-start.sh ...\
--override  s3.telemetry.metrics.exporter.type=prometheus \
--override  s3.metrics.exporter.prom.host=0.0.0.0 \
--override  s3.metrics.exporter.prom.port=8890 \
....
```

With AutoMQ monitoring metrics enabled, you can access Prometheus format monitoring metrics from any node via HTTP at the address: `http://{node_ip}:8890`, with the response example as follows:

```text theme={null}

....
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="DescribeDelegationToken"} 0.0 1720520709290
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="CreatePartitions"} 0.0 1720520709290
...

```

For a detailed introduction to the metrics, refer to the AutoMQ official documentation: [Prometheus Metrics▸](/automq/observability/prometheus-metrics).

### Deploying VictoriaMetrics

Reference documentation: [VictoriaMetrics](https://docs.victoriametrics.com/?_gl=1*196n44l*_ga*NjU0MDAxODA2LjE3MjE3OTA1NTM.*_ga_N9SVT8S3HK*MTcyMTc5MDU1Mi4xLjAuMTcyMTc5MDU1Mi42MC4wLjA.) \[7]. VictoriaMetrics supports deployments via binary versions, Docker images, and source code. Here, we choose to deploy via Docker. Execute the following command to start the VictoriaMetrics container and map the data files to the folder `/home/VictoriaMetrics/data` for easy monitoring of data import changes:

```bash theme={null}
# Create a New Data Folder
sudo mkdir -p /home/VictoriaMetrics/data
# Docker Start Command, Mount the Data Folder
docker run -d --name victoria-metrics \
  -p 8428:8428 \
  -v /home/VictoriaMetrics/data:/victoria-metrics-data \
  victoriametrics/victoria-metrics

```

Verify a successful installation by accessing: `http://{client_ip}:8428` in a browser.

### Deploying Prometheus

Prometheus can be deployed by downloading the binary package or using Docker. Below is an introduction to both deployment methods.

#### **Binary Deployment**

For ease of use, you can create a new script and modify the Prometheus download version as needed. After creating the script, execute it to complete the deployment. First, create a new script:

```bash theme={null}
cd /home
vim install_prometheus.sh
# !!! Paste the Script Content Below, Save, and Exit
# Grant Permissions
chmod +x install_prometheus.sh
# Execute Script
./install_prometheus.sh
```

The content of the script is as follows:

```bash theme={null}
version=2.45.3
filename=prometheus-${version}.linux-amd64
mkdir -p /opt/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v${version}/${filename}.tar.gz
tar xf ${filename}.tar.gz
cp -far ${filename}/*  /opt/prometheus/

# Config as a Service
cat <<EOF >/etc/systemd/system/prometheus.service
[Unit]
Description="prometheus"
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple

ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --web.enable-lifecycle --web.enable-remote-write-receiver

Restart=on-failure
SuccessExitStatus=0
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=prometheus


[Install]
WantedBy=multi-user.target
EOF

systemctl enable prometheus
systemctl restart prometheus
systemctl status prometheus

```

Next, modify Prometheus's configuration file, **add the task to collect AutoMQ observable data**, and **restart** Prometheus. Execute the command:

```bash theme={null}
# Fill in the Following for the Configuration File Content:
vim /opt/prometheus/prometheus.yml
# Restart Prometheus
systemctl restart prometheus
```

**The configuration file reference is as follows:** Please change the `client_ip` to the address exposed by AutoMQ for observable data.

```yaml theme={null}
# My Global Config
global:
  scrape_interval: 15s # Set the Scrape Interval to Every 15 Seconds. Default Is Every 1 Minute.
  evaluation_interval: 15s # Evaluate Rules Every 15 Seconds. the Default Is Every 1 Minute.

scrape_configs:
  # The Job Name Is Added as a Label `job=<job_name>` to Any Timeseries Scraped from This Config.
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
      
  - job_name: "automq"
    static_configs:
      - targets: ["{client_ip}:8890"]

# Config VictoriaMetrics IO Port
remote_write:
  - url: "http://{client_ip}:8428/api/v1/write"

remote_read:
  - url: "http://{client_ip}:8428/api/v1/read"

```

After deployment, you can access Prometheus via a browser to verify if the Metrics data from AutoMQ has been successfully collected. Visit `http://{client_ip}:9090/targets`.

<img />

#### Docker Deployment

If you have a running Prometheus Docker container, execute the command to remove the container first:

```bash theme={null}
docker stop prometheus
docker rm prometheus
```

Create a new configuration file and attach it when launching Docker:

```bash theme={null}
mkdir -p /opt/prometheus
vim /opt/prometheus/prometheus.yml
# Use the Configuration Details Provided in the Previous "Binary Deployment" Section.
```

Launch the Docker container:

```bash theme={null}
docker run -d \
  --name=prometheus \
  -p 9090:9090 \
  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -m 500m \
  prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=otlp-write-receiver \
  --web.enable-remote-write-receiver
```

This sets up a Prometheus service to collect AutoMQ Metrics. For further details on integrating AutoMQ Metrics with Prometheus, please visit: [Integrating Metrics into Prometheus | AutoMQ](/automq/observability/overview) \[8].

## Check the Integration Results.

### VictoriaMetrics Data Verification

VictoriaMetrics offers a Dashboard accessible via `http://{client_ip}:8428/vmui`, where the initial display is:

<img alt="Data file verification. Existing data files are identified in the data directory:" />

Data file verification. Existing data files are identified in the data directory:

```bash theme={null}
root@debian: ls /home/VictoriaMetrics/data/small/2024_07/
17E5088F4DC9C068  17E5088F4DC9C073  17E5088F4DC9C0B3  17E509EE0189D399  17E509EE0189D39C  17E509EE0189D39F  17E509EE0189D3A2  17E509EE0189D3A5
17E5088F4DC9C06A  17E5088F4DC9C075  17E509EE0189D38D  17E509EE0189D39A  17E509EE0189D39D  17E509EE0189D3A0  17E509EE0189D3A3  17E509EE0189D3A6
17E5088F4DC9C071  17E5088F4DC9C07B  17E509EE0189D398  17E509EE0189D39B  17E509EE0189D39E  17E509EE0189D3A1  17E509EE0189D3A4  parts.json
```

By accessing the status at `http://{client_ip}:8428/api/v1/status/tsdb`, you can obtain a JSON file:

```json theme={null}
 "status": "success",
    "data": {
        "totalSeries": 3125,
        "totalLabelValuePairs": 15564,
        "seriesCountByMetricName": [
            {
                "name": "prometheus_http_request_duration_seconds_bucket",
                "value": 150
            },
            {
                "name": "prometheus_http_response_size_bytes_bucket",
                "value": 135
            },
            {
                "name": "kafka_request_queue_time_50p_milliseconds",
                "value": 90
            },
...............

```

### Grafana Dashboard (Optional)

Since we have used VictoriaMetrics as Prometheus storage, we can configure it as a data source for Grafana at `http://{client_ip}:8428`, to access the status information of the AutoMQ cluster. For Grafana template files, you can refer to the official AutoMQ templates: [grafana](https://github.com/AutoMQ/automq/tree/main/docker/telemetry/grafana/provisioning/dashboards) \[9]. The final demonstration output is as follows:

**Cluster Overview:** Provides cluster-level monitoring information, including metrics like the number of nodes, data size, and cluster traffic. Additionally, it offers an overview of dimensions such as Topic, Group, and Broker, with support for drill-down functionality to examine detailed monitoring information.

<img alt="With this, our integration process concludes; more features to explore can be referenced at VictoriaMetrics " />

With this, our integration process concludes; further features to explore can be found in the VictoriaMetrics [official documentation](https://docs.victoriametrics.com/?_gl=1*196n44l*_ga*NjU0MDAxODA2LjE3MjE3OTA1NTM.*_ga_N9SVT8S3HK*MTcyMTc5MDU1Mi4xLjAuMTcyMTc5MDU1Mi42MC4wLjA.) \[10].

## **Summary**

In this article, we have thoroughly examined how to integrate AutoMQ, Prometheus, and VictoriaMetrics to enable comprehensive monitoring of AutoMQ clusters. We began by introducing the functions and advantages of each component, explaining why Prometheus was chosen as the monitoring system and why VictoriaMetrics is essential as the storage backend for Prometheus. Next, we offered step-by-step guidance on deploying and configuring AutoMQ, VictoriaMetrics, and Prometheus, and confirmed the effectiveness of this integration.

With this integration approach, enterprises can gain the following benefits:

1. Efficient data collection and storage: Prometheus handles efficient data collection, while VictoriaMetrics provides efficient data storage and compression capabilities, ensuring that the system can store large-scale time-series data with minimal hardware resource consumption.

2. Fast data querying: VictoriaMetrics offers rapid data query performance to meet high concurrent query demands, ensuring efficient queries in scenarios involving large-scale data storage.

3. Prometheus Integration: VictoriaMetrics supports Prometheus' remote storage interface, enabling smooth integration with existing Prometheus monitoring systems and enhancing the overall performance and stability of such systems.

Through this integration, enterprises can achieve comprehensive monitoring of AutoMQ clusters, ensuring high availability and performance optimization, promptly detecting and addressing potential issues, optimizing system performance, and ensuring business continuity and stability. Looking ahead, as business demands evolve and technology advances, we can further explore and refine monitoring solutions to address more complex application scenarios and higher performance requirements efficiently.

## References

\[1] AutoMQ: [https://www.automq.com/zh](https://www.automq.com/zh)

\[2] Prometheus: [https://prometheus.io/docs/prometheus/latest/getting\_started/](https://prometheus.io/docs/prometheus/latest/getting_started/)

\[3] VictoriaMetrics: [https://docs.victoriametrics.com/](https://docs.victoriametrics.com/)

\[4] VictoriaMetrics Structure:  [https://docs.victoriametrics.com/cluster-victoriametrics/](https://docs.victoriametrics.com/cluster-victoriametrics/)

\[5] Cluster Deployment | AutoMQ: [https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker](https://docs.automq.com/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker)

\[6] Metrics | AutoMQ: [https://docs.automq.com/automq/observability/overview](https://docs.automq.com/automq/observability/overview)

\[7] VictoriaMetrics Installation Documentation: [https://docs.victoriametrics.com/](https://docs.victoriametrics.com/)

\[8] Integrating Metrics with Prometheus: [https://docs.automq.com/automq/observability/overview](https://docs.automq.com/automq/observability/overview)

\[9] grafana: [https://github.com/AutoMQ/automq/tree/main/docker/telemetry/grafana/provisioning/dashboards](https://github.com/AutoMQ/automq/tree/main/docker/telemetry/grafana/provisioning/dashboards)

\[10] VictoriaMetrics Official Documentation: [https://docs.victoriametrics.com/](https://docs.victoriametrics.com/)


# RustFS
Source: https://docs.automq.com/automq/integrations/storage/rustfs

AutoMQ partners with RustFS to provide a scalable, cost-efficient Kafka-compatible cloud service. Leverage cloud-native benefits with superior performance and licensing ease.

## Preface

Enterprises' thirst for real-time data is driving significant shifts in data architecture. However, building efficient data streaming platforms faces two major bottlenecks: the severe cost and operational challenges of traditional Apache Kafka in cloud environments, and the architectural complexity or restrictive AGPL licensing of existing object storage solutions. These hurdles create significant barriers to cost-effective, scalable, and commercially viable real-time data processing.

To address these challenges, AutoMQ and RustFS have formed a strategic partnership. This collaboration deeply integrates AutoMQ's cloud-native, 100% Kafka-compatible stream processing with RustFS, a high-performance, Apache 2.0 licensed distributed object storage. By combining these two technologies, enterprises gain access to a next-generation Diskless Kafka platform that delivers a superior architecture, lower TCO, and complete avoidance of licensing risks.

## **Hands-On Guide: Deploying AutoMQ with RustFS in 4 Steps**

### Precondition

Before starting, make sure you have Docker and Docker Compose installed and running on your system.

This guide uses Docker Compose to deploy a decoupled AutoMQ and RustFS stack. We'll solve the two main challenges: service startup dependency and automatic S3 bucket initialization.

### Deployment Instructions

With the precondition met, we'll now build the stack. This process is broken down into the key phases of the deployment: first, defining all services in the complete docker-compose.yaml, then analyzing how they're orchestrated, and finally, validating the end-to-end data flow.

1. **The S3 Backend-RustFS:**  This service acts as our S3-compatible storage.

   * `healthcheck` **:**  This block is crucial. It tells Docker not to just assume the service is "on," but to actively check its /health endpoint. Only when this check passes is the service considered "ready" to handle S3 requests.

2. **The Bucket Initializer-mc-init:** This is a small utility container whose only job is to create our S3 buckets.

   * `depends_on: service_healthy` **:**  This line is the key to our automation. It explicitly tells Docker Compose: "Do not *even start* this mc-init container until the rustfs service's healthcheck  is passing." This prevents errors where the script tries to create buckets on a service that isn't ready.

3. **The AutoMQ Service-server1:** This is the main Kafka broker service.

   * `depends_on` **:**  This service depends on *both* rustfs and mc-init being Up. This guarantees that when AutoMQ starts, the S3 backend is ready *and* the mc-init script has already finished creating the required buckets.

   * `Endpoint Override` **:**  The command block's --override flags (e.g., endpoint=[http://rustfs:9000](http://rustfs:9000)) are critical. They redirect AutoMQ's S3 requests away from the public internet and toward our own rustfs container within the Docker network.

### Configuration

This entire stack—all three services and their dependencies—is defined in a single docker-compose.yaml file. Create this file now and paste the complete configuration block below into it.

```yaml theme={null}

services:
  rustfs:
    image: rustfs/rustfs:latest
    container_name: rustfs-server
    ports:
      - "9000:9000"
      - "9001:9001"  
    environment:
      - RUSTFS_VOLUMES=/data
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
      - RUSTFS_CONSOLE_ENABLE=true
      - RUSTFS_ACCESS_KEY=rustfsadmin
      - RUSTFS_SECRET_KEY=rustfsadmin
    volumes:
      - rustfs_data:/data
    networks:
      - automq_net
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "sh", "-c", "curl -f http://localhost:9000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  # 1.5. MC (MinIO Client)
  mc-init:
    container_name: mc-init-rustfs
    image: minio/mc:latest
    depends_on:
      rustfs:
        condition: service_healthy
    networks:
      - automq_net
    entrypoint: >
      /bin/sh -c "
      echo 'Waiting for RustFS service...';
      /usr/bin/mc alias set myrustfs http://rustfs:9000 rustfsadmin rustfsadmin;
      echo 'RustFS is ready. Creating buckets...';
      /usr/bin/mc mb myrustfs/automq-data;
      /usr/bin/mc mb myrustfs/automq-ops;
      echo 'Buckets created successfully. Keeping container alive.';
      tail -f /dev/null
      "


  server1:
    container_name: "automq-server1"
    image: automqinc/automq:1.6.0
    stop_grace_period: 1m
    environment:
      - KAFKA_S3_ACCESS_KEY=rustfsadmin
      - KAFKA_S3_SECRET_KEY=rustfsadmin
      - AWS_ACCESS_KEY_ID=rustfsadmin
      - AWS_SECRET_ACCESS_KEY=rustfsadmin
      - KAFKA_HEAP_OPTS=-Xms1g -Xmx4g -XX:MetaspaceSize=96m -XX:MaxDirectMemorySize=1G
      - CLUSTER_ID=3D4fXN-yS1-vsQ8aJ_q4Mg
    command:
      - bash
      - -c
      - |
        /opt/automq/kafka/bin/kafka-server-start.sh \
        /opt/automq/kafka/config/kraft/server.properties \
        --override cluster.id=$$CLUSTER_ID \
        --override node.id=0 \
        --override controller.quorum.voters=0@server1:9093 \
        --override controller.quorum.bootstrap.servers=server1:9093 \
        --override advertised.listeners=PLAINTEXT://server1:9092 \
        --override s3.data.buckets='0@s3://automq-data?region=us-east-1&endpoint=http://rustfs:9000&pathStyle=true' \
        --override s3.ops.buckets='1@s3://automq-ops?region=us-east-1&endpoint=http://rustfs:9000&pathStyle=true' \
        --override s3.wal.path='0@s3://automq-data?region=us-east-1&endpoint=http://rustfs:9000&pathStyle=true'
    networks:
      - automq_net
    depends_on:
      - rustfs
      - mc-init

networks:
  automq_net:
    driver: bridge

volumes:
  rustfs_data:

```

### Validate Deployment

Time to see it all work. With your docker-compose.yaml file saved, launch the stack. We will then verify that the three services defined in your YAML have orchestrated correctly. The test will use the standard Kafka bin scripts, found inside the automq-server1 container, to create a topic and send messages, proving that AutoMQ is successfully reading and writing data to the RustFS backend.

#### 4.1 Launch and Check Status

```bash theme={null}

# Start all services
docker compose up -d

# Check the status  to see if everything is running
docker compose ps

```

**Expected Result:**  The STATUS for rustfs-server should include (healthy), and all three services should show as Up.

```text theme={null}

NAME             IMAGE                    COMMAND                  SERVICE   CREATED         STATUS                   PORTS
automq-server1   automqinc/automq:1.6.0   "bash -c '/opt/autom…"   server1   ...             Up ...
mc-init-rustfs   minio/mc:latest          "/bin/sh -c ' echo '…"   mc-init   ...             Up ...
rustfs-server    rustfs/rustfs:latest     "/entrypoint.sh rust…"   rustfs    ...             Up ... (healthy)         ...

```

#### 4.2 Create a Topic

This step proves AutoMQ can successfully write to the RustFS backend by creating a Kafka topic.

```bash theme={null}

docker exec -it automq-server1 bash
/opt/automq/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server server1:9092 --partitions 1 --replication-factor 1

```

**Expected Result:**  You will see the message Created topic test-topic.

#### 4.3 Produce and Consume

Finally, let's confirm the full data flow by sending and receiving a message. This requires two terminals.

Terminal 1 (Consumer):

```bash theme={null}

/opt/automq/kafka/bin/kafka-console-consumer.sh --bootstrap-server server1:9092 --topic test-topic

```

Terminal 2 (Producer):

```bash theme={null}

docker exec -it automq-server1 bash
/opt/automq/kafka/bin/kafka-console-producer.sh --broker-list server1:9092 --topic test-topic
> Hello, AutoMQ and RustFS!

```

Expected Result:

The message Hello, AutoMQ and RustFS! will instantly appear in Terminal 1. This confirms the entire data flow—from producer to S3 storage and back to the consumer—is fully operational.

## **Future Outlook**

The integration of AutoMQ with RustFS demonstrates significant advantages over using standard object storage or complex traditional systems. RustFS provides a high-performance, stable storage layer, delivering high read/write throughput with stable memory usage, which allows AutoMQ to maintain its P99 low latency targets. Additionally, its lightweight, metadata-free architecture and Apache-2.0 license drastically reduce operational complexity and eliminate commercial compliance risks.

Ultimately, this partnership highlights a compelling synergy. [AutoMQ](https://www.automq.com/?utm_source=automq_rustfs_partner) provides a modern, elastic, and cost-effective architecture for Kafka by making brokers stateless and leveraging compute-storage separation. When paired with [RustFS](https://rustfs.com/?utm_source=automq_rustfs_partner), which provides a high-performance, license-friendly, and easy-to-operate S3-compatible backend, the result is a highly reliable, scalable, and resource-efficient streaming platform. This combination is engineered to meet modern data demands, delivering both architectural elasticity (at the compute layer) and high-performance, compliant persistence (at the storage layer).


# Tigris
Source: https://docs.automq.com/automq/integrations/storage/tigris

Deploy AutoMQ statelessly with Tigris. Achieve Kafka compatibility, cloud-native scalability, and cost efficiency with S3 integration for global storage.

[Tigris](https://www.tigrisdata.com/) is a globally distributed, multi-cloud object storage service with built-in support for the S3 API. It uses Dynamic Data Placement and Access-Based Rebalancing to deliver low-latency access worldwide — without the need to manage replication or caching.

When paired with Tigris, AutoMQ  can run as fully stateless brokers, leveraging Tigris's unique architecture to eliminate attached disks and replication overhead.

## Quick Start with Docker Compose:

The easiest way to run AutoMQ with Tigris is using Docker Compose. This guide will walk you through setting up a single-node AutoMQ cluster backed by Tigris storage.

### Prerequisites

* **Docker**  and **Docker Compose**  installed

* A **Tigris account**  - create one at [https://storage.new](https://storage.new/)

* **Tigris credentials**  - create Access Key and Secret Key from your Tigris dashboard at [https://console.storage.dev](https://console.storage.dev)

### Create Buckets in Tigris

AutoMQ requires two buckets: one for data storage and one for cluster's metrics and logs. You can create them via the Tigris console or using the AWS CLI:

```bash theme={null}

# Configure credentials
export AWS_ACCESS_KEY_ID=YOUR_TIGRIS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_TIGRIS_SECRET_KEY
export AWS_ENDPOINT_URL_S3=https://t3.storage.dev

# Create buckets for AutoMQ data and operations storage
aws s3api create-bucket --bucket your-automq-data --endpoint-url https://t3.storage.dev
aws s3api create-bucket --bucket your-automq-ops --endpoint-url https://t3.storage.dev

```

**Note** : Bucket names must be globally unique across all Tigris users.

### Configure Docker Compose

Edit the  `docker-compose.yaml`  file and update the Tigris credentials and bucket names:

```yaml theme={null}

services:
  server1:
    container_name: "automq-server1"
    image: automqinc/automq:1.6.0-rc0
    stop_grace_period: 1m
    environment:
      # Replace with your Tigris credentials
      - KAFKA_S3_ACCESS_KEY=tid_YOUR_ACCESS_KEY_HERE
      - KAFKA_S3_SECRET_KEY=tsec_YOUR_SECRET_KEY_HERE
      - KAFKA_HEAP_OPTS=-Xms1g -Xmx4g -XX:MetaspaceSize=96m
        -XX:MaxDirectMemorySize=1G
      - CLUSTER_ID=3D4fXN-yS1-vsQ8aJ_q4Mg
    command:
      - bash
      - -c
      - |
        /opt/automq/kafka/bin/kafka-server-start.sh \
        /opt/automq/kafka/config/kraft/server.properties \
        --override cluster.id=$$CLUSTER_ID \
        --override node.id=0 \
        --override controller.quorum.voters=0@server1:9093 \
        --override controller.quorum.bootstrap.servers=server1:9093 \
        --override advertised.listeners=PLAINTEXT://server1:9092 \
        --override s3.data.buckets='0@s3://your-automq-data?region=auto&endpoint=https://t3.storage.dev' \
        --override s3.ops.buckets='1@s3://your-automq-ops?region=auto&endpoint=https://t3.storage.dev' \
        --override s3.wal.path='0@s3://your-automq-data?region=auto&endpoint=https://t3.storage.dev'
    networks:
      - automq_net

networks:
  automq_net:
    driver: bridge

```

**Key Configuration Parameters:**

* `KAFKA_S3_ACCESS_KEY` - Your Tigris Access Key (starts with `tid_`)
* `KAFKA_S3_SECRET_KEY` - Your Tigris Secret Key (starts with `tsec_`)
* `s3.data.buckets` - Your data bucket name in the S3 URL (stores Kafka data)
* `s3.ops.buckets` - Your ops bucket name in the S3 URL (stores operational
  metadata)
* `s3.wal.path` - Write-Ahead Log path (typically same as data bucket)
* `endpoint=https://t3.storage.dev` - Tigris S3-compatible endpoint
* `region=auto` - Tigris automatically routes to the nearest region

For detailed information on these Tigris and S3 configuration parameters, refer
to the
[AutoMQ Broker and Controller Configuration guide](https://docs.automq.com/automq/configuration/broker-and-controller-configuration#s3-data-buckets).

### 4. Start AutoMQ

Start the AutoMQ cluster with Docker Compose:

```bash theme={null}
docker-compose up -d

```

Check the logs to verify AutoMQ is running:

```bash theme={null}

docker-compose logs -f

```

You should see messages indicating:

* `Readiness check pass! (ObjectStorageReadinessCheck)`  - Connected to Tigris

* `The broker has been unfenced`  - Broker is ready

* `Kafka Server started`  - AutoMQ is running

### Create a Topic

Create a Kafka topic using the AutoMQ CLI:

```bash theme={null}

docker run --network automq_net automqinc/automq:1.6.0-rc0 \
  /bin/bash -c "/opt/automq/kafka/bin/kafka-topics.sh \
  --create \
  --topic my-test-topic \
  --bootstrap-server server1:9092 \
  --partitions 3 \
  --replication-factor 1"

```

List all topics to verify:

```bash theme={null}

docker run --network automq_net automqinc/automq:1.6.0-rc0 \
  /bin/bash -c "/opt/automq/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server server1:9092"

```

Describe the topic:

```bash theme={null}

docker run --network automq_net automqinc/automq:1.6.0-rc0 \
  /bin/bash -c "/opt/automq/kafka/bin/kafka-topics.sh \
  --describe \
  --topic my-test-topic \
  --bootstrap-server server1:9092"

```

### Produce and Consume Messages

**Produce test messages:**

```bash theme={null}

docker run --network automq_net automqinc/automq:1.6.0-rc0 \
  /bin/bash -c "/opt/automq/kafka/bin/kafka-producer-perf-test.sh \
  --topic my-test-topic \
  --num-records=10000 \
  --throughput 1000 \
  --record-size 1024 \
  --producer-props bootstrap.servers=server1:9092"

```

**Consume messages:**

```bash theme={null}

docker run --network automq_net automqinc/automq:1.6.0-rc0 \
  /bin/bash -c "/opt/automq/kafka/bin/kafka-console-consumer.sh \
  --topic my-test-topic \
  --bootstrap-server server1:9092 \
  --from-beginning \
  --max-messages 10"

```

## Congratulations! 🎉

You've successfully deployed AutoMQ with Tigris as the storage backend! In this guide, you:

* Created Tigris buckets for data and operational storage

* Configured and launched a single-node AutoMQ cluster using Docker Compose

* Connected AutoMQ to Tigris using S3-compatible endpoints

* Created a Kafka topic with multiple partitions

* Produced and consumed messages through AutoMQ

Your AutoMQ cluster is now running entirely stateless with all data durably stored in Tigris object storage. You can scale brokers up or down without worrying about data migration, and benefit from Tigris' global distribution and zero egress fees.

## Learn More

### Tigris Resources

* [Tigris Documentation](https://www.tigrisdata.com/docs/)

* [Tigris Overview](https://www.tigrisdata.com/docs/overview/)

* [Tigris Get Started](https://www.tigrisdata.com/docs/get-started/)

* [Tigris Buckets](https://www.tigrisdata.com/docs/buckets/create-bucket/)

* [Tigris Blog](https://www.tigrisdata.com/blog/)


# Migrating from Apache Kafka to AutoMQ
Source: https://docs.automq.com/automq/migration/migrating-from-apache-kafka-to-automq

Migrate from Apache Kafka to AutoMQ with cloud-native scalability, full Kafka compatibility, and cost efficiency. Follow detailed steps for seamless transition and minimize disruption.

This article details the plan and implementation process for migrating from Apache Kafka® to AutoMQ.

<Tip>
  AutoMQ commercial editions offer the [Kafka Linking Tool](https://docs.automq.com/automq-cloud/migrate-to-automq/overview#overview-of-kafka-linking) that supports byte-for-byte message synchronization, ensuring that message offsets remain identical before and after synchronization, and enables zero-downtime migrations. If your business scenario requires a smoother migration approach, you can reach out to the AutoMQ team for necessary assistance and best practices through [this form](https://www.automq.com/contact).
</Tip>

## Prerequisites

### Feature List

Currently, AutoMQ only provides the Kafka Server component and does not offer other ecosystem components. Therefore, before migrating to AutoMQ, you need to confirm whether the following features are used and handle them according to the following guidelines:

* **Kafka Server:** AutoMQ provides a more cost-effective and flexible Kafka Server solution, **compatible with Apache Kafka versions 0.9.x to 3.9.x**. If your current Apache Kafka Server version falls outside the supported range, please contact us for support and updates.

* **Kafka Client:** AutoMQ is fully compatible with the original Apache Kafka Client SDK; ensure your SDK version is between **0.9.x and 3.9.x** for full compatibility.

* **Kafka Connector:** AutoMQ is compatible with the Apache Kafka Connector. If you are using Kafka Connector, it is recommended to maintain the existing service and replace the Kafka Server endpoint in the Connector configuration with the AutoMQ instance endpoint.

### Business Scope

Migrating from Apache Kafka® to AutoMQ in a production environment is generally recommended to be done in batches, gradually migrating Topics and Consumer Groups from the source cluster based on business chains and scopes.

Therefore, it is advisable to follow the process below for business scope assessment:

1. **Inventory Source Cluster Resources:** It is recommended to use Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster, categorizing them by business systems. Clean up any invalid or unclaimed Topics and Consumer Groups in advance.

<Tip>
  It is advisable to check the Topics in the source cluster and clean up those that have not been written to or subscribed to for a long time, thereby narrowing the scope of migration.
</Tip>

2. **Batch Division by Business Systems:** In a production environment, it is recommended to plan migration batches based on business systems. Start with offline or non-core business systems and gradually expand the scope of migration to avoid unexpected risks.

3. **Coordinate with Business Personnel for Migration:** The process of migrating from Apache Kafka to AutoMQ requires cooperation from business personnel to change access points and restart applications. Therefore, before executing the migration plan, coordination with business personnel is necessary.

<Info>
  Once a connection is established between the Apache Kafka Client and the server, the connection to the source cluster will not be disconnected. Therefore, even if the server is accessed via a domain name, it is still necessary to restart the client application to connect to the new AutoMQ cluster.
</Info>

## Migration Process

From migrating from Apache Kafka® to AutoMQ, the primary considerations include the following tasks:

* **Message Data Migration:** Kafka stores both consumed and unconsumed historical message data. The migration process must ensure that the message data is replicated to the new cluster as needed, with no message loss.

* **Consumer Progress Metadata Migration:** The metadata of consumer progress that Kafka consumers have already processed in the source cluster needs to be reconstructed in the new cluster to avoid losing progress and re-consuming messages, which can result in a large amount of duplication.

* **Producer Switch:** Besides data synchronization, the migration work also involves switching producer applications at the appropriate time so that producers connect to the target cluster to produce new messages.

* **Consumer Switch:** Besides data synchronization, the migration work also involves switching consumer applications at the appropriate time so that consumers connect to the target cluster and continue consuming messages from the previous progress.

The overall migration plan can be referenced in the process diagram below:

<img />

### Step 1: **Create a Migration Task and Synchronize Data**

Refer to [Overview▸](/automq-cloud/migrate-to-automq/overview). AutoMQ recommends using a separate MirrorMaker2 Connector for each batch task to optimize data synchronization and enhance isolation. Fill in the following parameters according to your specific requirements:

* **Source Cluster:** Specify the source cluster that needs migration.

* **Target Cluster:** Specify the target cluster for the migration. Here, select the AutoMQ cluster.

* **List of Topics to Sync:** Set the list of Topics covered by the migration task. This list supports wildcard inputs, allowing you to configure multiple Topics at once.

<Tip>
  The AutoMQ migration tool supports automatic syncing of dynamic Topic updates. Once a migration task is created, any new Topics or changes in the number of partitions of existing Topics that meet the current list criteria will be automatically synced to the target cluster.
</Tip>

* **Sync Consumer Progress:** It is recommended to enable this. When enabled, the migration task will map the consumption progress of the ConsumerGroup from the source cluster to the target cluster, ensuring that consumers can continue from where they left off after switching, thus avoiding a large amount of duplicate processing.

<Warning>
  It is important to note that the offsets of messages in the clusters synced by MirrorMaker2 are inconsistent. Therefore, the consumption progress of the source cluster will be remapped to the target cluster. This process may result in a slight rollback of offsets.
</Warning>

* **List of Topics to Exclude:** If wildcards are used in the **List of Topics to Sync** but some Topics need to be excluded from the migration, you can set the list of Topics to exclude, removing the relevant Topics from the migration.

### Step 2: **Monitor Sync Progress and Wait for the Migration to Approach Completion**

MirrorMaker2 supports viewing synchronization latency via JMX metrics. Users should monitor the synchronization task's latency metrics and wait until the metrics approach zero.

<Tip>
  Monitoring synchronization completion helps reduce the observation wait time for subsequent application switchovers.
</Tip>

In MirrorMaker2, synchronization latency refers to the time taken for messages from the migrated Topic to be produced in the source cluster and synchronized. When multiple Topics are involved in a single migration task, the maximum latency among all Topics is considered.

According to the migration plan description, it is necessary to regularly monitor the synchronization latency and the synchronization backlog metrics of the migration task. It is recommended to prepare for the next step of application switchover when the **synchronization latency is less than 1 minute**.

### Step 3: **Stop the Source Cluster Consumers**

Application switchover requires switching the consumers first. It is advisable to stop the consumer applications initially and wait for the consumption progress data from the source cluster to be synchronized to the target cluster before switching to the target cluster.

<Tip>
  It is recommended to wait for a period, allowing the consumption progress to be fully synchronized. The default time is 60 seconds, but it can be customized in the advanced configuration of the migration task.

  Alternatively, the application can first start new consumers in the target cluster with the same Consumer Group, and then stop the consumers in the source cluster. This will result in both ends consuming simultaneously for a period.
</Tip>

### Step 4: Switch Consumers to the Target Cluster

After completing Step 3, update the consumer configuration to replace the endpoint with the one provided by the AutoMQ instance, then restart the application.

<Info>
  According to MirrorMaker2's technical implementation, there may be some consumer duplication during Steps 3 and 4. If you want to reduce the likelihood of duplication, you can customize the configuration in the advanced parameters of the migration task.
</Info>

### Step 5: Stop Producers on the Source Cluster

For each Topic, ensure that Steps 3 and 4 have been repeated for all consumers of the current Topic. Once all consumers have been switched, you can stop the producers on the source cluster and wait for all message data to synchronize to the target cluster.

<Warning>
  Stopping the producers is necessary to prevent the target cluster from receiving messages from both producers and MirrorMaker2 simultaneously, which could lead to message disorder.
</Warning>

<Tip>
  The switch to the producer process must be completely shut down to ensure messages are not processed out of order. AutoMQ commercial editions offer the [Kafka Linking Tool](https://docs.automq.com/automq-cloud/migrate-to-automq/overview#overview-of-kafka-linking) that supports byte-for-byte message synchronization, ensuring that message offsets remain identical before and after synchronization, and enables zero-downtime migrations. If your business scenario requires a smoother migration approach, you can reach out to the AutoMQ team for necessary assistance and best practices through [this form](https://www.automq.com/contact).
</Tip>

### Step 6: Switch Producers to the Target Cluster

Monitor the synchronization delay of the migration task. Once the synchronization delay is confirmed to be zero, restart the producers with the new endpoint address to connect to the AutoMQ target cluster, completing the entire migration task.

<Warning>
  Monitoring the upstream and downstream applications of the Topic continuously is necessary after switching producers to ensure they meet expectations.
</Warning>

### Step 7: Repeat Steps 3 Through 6 Until the Migration Is Complete, Then Delete the Migration Task.

1. **Repeatedly check if all Topic migrations are completed.** Steps 3 to 6 are performed at the granularity of the Topic and the associated producer and consumer applications. Therefore, continuous checks are needed until all Topics in the current migration task are completed.

2. **Delete the migration task.** Once all Topic migrations are completed, click confirm to delete the task. AutoMQ will clean up the underlying machine resources and configuration information.

<Warning>
  After deleting the migration task, it cannot be recovered. Even if a migration task for the same Topic is recreated, it will be treated as a new task and will not continue from the previous migration state. Therefore, ensure that the current migration task has successfully completed its work before deletion.
</Warning>

## Rollback Process

During various stages of the migration process, if unexpected exceptions occur and cannot be quickly diagnosed, users can attempt to rollback using the following process and be aware of the impacts caused by the rollback.

<table>
  <thead>
    <tr>
      <th>
        Migration Stage

        <br />
      </th>

      <th>
        Rollback Plan

        <br />
      </th>

      <th>
        Rollback Impact

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Step 1: Create Task

        <br />
      </td>

      <td>
        <ul>
          <li>Delete Task</li>

          <li>Delete the Topic and Consumer Group of the target instance, and clean up synchronized data</li>
        </ul>
      </td>

      <td>
        At this stage, since the application has not yet switched, there is no impact on the application

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Step 2: Monitor Synchronization Progress

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Step 3: Stop Source Cluster Consumers

        <br />
      </td>

      <td>
        <ul>
          <li>Resume Source Cluster Consumers, roll back to Step 2 stage</li>
        </ul>
      </td>

      <td>
        Data remains unaffected, only causing consumption pauses and delays

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Step 4: The consumer has been switched to the target cluster

        <br />
      </td>

      <td>
        <ul>
          <li>Stop the target cluster consumer, switch back to the source cluster access point, and roll back to Step 2</li>
        </ul>
      </td>

      <td>
        Rolling back to the source cluster will reconsume data during the switching period

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Step 5: The source cluster producer has been stopped

        <br />
      </td>

      <td>
        <ul>
          <li>Resume the source cluster producer, and roll back to Step 4</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        Step 6: The producer has been switched to the target cluster

        <br />
      </td>

      <td>
        <ul>
          <li>Stop the target cluster producer, switch back to the source cluster access point, and roll back to Step 4</li>
        </ul>
      </td>

      <td>
        During Step 6, messages produced to the target cluster cannot be reverse-synchronized back to the source cluster and require application processing.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        In Step 7, the migration task has been deleted.

        <br />
      </td>

      <td>
        <ul>
          <li>
            After deleting the task
            rollback is not possible
          </li>
        </ul>
      </td>

      <td>
        Not applicable

        <br />
      </td>
    </tr>
  </tbody>
</table>


# Overview
Source: https://docs.automq.com/automq/migration/overview

Migrate to AutoMQ, a next-gen Kafka-compatible cloud-native service, with ease. Enjoy seamless cluster migration and cost-efficient scalability using MirrorMaker2.

# Migration

AutoMQ is a next-generation Kafka release redesigned based on cloud-native concepts, and it is compatible with Apache Kafka®. This article will detail how to migrate from Apache Kafka® or other Kafka distributions to AutoMQ.

## Migration Tool

The AutoMQ Open Source recommends using the MirrorMaker2 tool for cluster migration, supporting transitions from Apache Kafka or other Kafka distributions to AutoMQ.

<Tip>
  AutoMQ commercial editions provide a [Kafka Linking tool](https://docs.automq.com/automq-cloud/migrate-to-automq/overview#overview-of-kafka-linking), which supports Byte-to-Byte message synchronization (ensuring that message offsets remain completely consistent before and after synchronization) and enables zero-downtime migration. If your business scenario requires a smoother migration solution, you can reach out to the AutoMQ team via [this form](https://www.automq.com/contact) for assistance and best practices.
</Tip>

MirrorMaker2 migration tool has the following capabilities:

* **Supports batch migration of clusters:** MirrorMaker2 facilitates batch migration of Topics and Groups from the source cluster to the target cluster, enabling phased rollout and validation.

* **Supports synchronization of consumption progress:** MirrorMaker2 ensures real-time synchronization of consumption progress from the source cluster, allowing consumers to continue consuming from the target cluster after migration.

* **Supports dynamic updates of Topics and Groups:** MirrorMaker2 enables real-time monitoring of dynamic updates of Topics and Groups, ensuring that no updates are missed during the migration process.

The technical architecture of MirrorMaker2 is as follows:

<img />

## Core Concepts

MirrorMaker2 defines several Connectors to implement data synchronization tasks between two Apache Kafka® clusters, including necessary configurations and real-time running states.

* **MirrorSourceConnector**: Used for synchronizing message data from the source cluster.

* **MirrorCheckpointConnector**: Used to synchronize the consumption progress data of the source cluster ConsumerGroup.

* **MirrorHeartbeatConnector**: Used for system detection of the synchronization link's operational status.

## Deploying MirrorMaker2

For the installation and deployment of MirrorMaker2, refer to the relevant links from the Apache Kafka community:

* [MirrorMaker2 Design Document.](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-ConnectorConfigurationProperties)

* [MirrorMaker2 Configuration Documentation.](https://kafka.apache.org/documentation/#mirrormakerconfigs)

## Constraints and Limitations

MirrorMaker2 uses Connectors to synchronize messages between the source and target clusters. This mechanism cannot guarantee complete data consistency between the source and target clusters. Therefore, when using the migration tool, please note the following constraints and limitations:

<Warning>
  **Minor Message Duplication**

  Consumer progress data is asynchronously and intermittently synchronized, which may cause offset rollbacks and result in duplicate consumption when switching consumers.

  Message synchronization is an asynchronous operation, and due to machine anomalies or fault recovery, partial data may be duplicated.

  **Minor Message Reordering**

  If offset rollbacks occur when switching consumers, a small number of messages will be reconsumed, leading to disorder in the message sequence.
</Warning>

## Next Steps

After understanding the background of cluster migration, it is recommended to choose the corresponding solution based on the current cluster situation:

* **Migrating from Apache Kafka to AutoMQ:** Follow the migration plan and execute the migration step by step in batches. [Migrating from Apache Kafka to AutoMQ▸](/automq/migration/migrating-from-apache-kafka-to-automq)


# Alert Rule Configuration
Source: https://docs.automq.com/automq/observability/alert-rule-configuration

AutoMQ enhances Kafka compatibility with cloud scalability. Configure alerts using Prometheus for real-time monitoring of your data infrastructure.

AutoMQ integrates with Prometheus, enabling the transfer of Metrics data to Prometheus. This allows users to configure custom alert rules using Prometheus for the monitoring of unusual conditions like business pressure levels.

### Alert Templates

The AutoMQ team has crafted a series of alert templates based on Metrics commonly utilized in production environments. These templates allow users to choose and set up alert rules according to their specific requirements.

<Tip>
  To get the Prometheus alert rule templates mentioned below, you can reach out to the AutoMQ team by [filling out this form](https://www.automq.com/contact).
</Tip>

The list of alert rule templates is as follows:

| **Alert Rule Item**<br />                   | **Function and Scenario**<br />                                                                                                                                                                                                                                                                                           |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| HighTopicBytesInPerSec<br />                | <ul><li>Alert Rule: High alert for topic data write throughput</li><li>Function and Scenario: Monitor if the data write volume per second for a specified topic exceeds the threshold. Typically used for monitoring sudden bursts in write traffic.</li></ul>                                                            |
| HighTopicBytesInPerSecDayToDayChange<br />  | <ul><li>Alert Rule: High alert for day-to-day increase in topic data write throughput</li><li>Function and Scenario: Monitor if the day-to-day increase rate in data write volume per second for a specified topic exceeds the threshold. Generally used for monitoring sudden bursts in write traffic.</li></ul>         |
| LowTopicBytesInPerSec<br />                 | <ul><li>Alert Rule: Low alert for topic data write throughput</li><li>Function and Scenario: Monitor if the data write volume per second for a specified topic falls below the threshold. Typically used to detect write link damage or drop to zero.</li></ul>                                                           |
| LowTopicBytesInPerSecDayToDayChange<br />   | <ul><li>Alert Rule: Alert for day-to-day decrease in topic data write throughput</li><li>Function and Scenario: Monitor if the day-to-day decrease rate in data write volume per second for a specified topic exceeds the threshold. Usually employed to monitor write link damage or business fluctuations.</li></ul>    |
| HighTopicBytesOutPerSec<br />               | <ul><li>Alarm Rule: High topic data read throughput alert</li><li>Purpose and Scenario: Monitor if the data read per second on a specified topic exceeds the threshold, typically used for monitoring burst reads and fanout traffic.</li></ul>                                                                           |
| HighTopicBytesOutPerSecDayToDayChange<br /> | <ul><li>Alarm Rule: High day-to-day growth in topic data read throughput alert</li><li>Purpose and Scenario: Monitor if the day-to-day growth rate of data read per second on a specified topic exceeds the threshold, generally used for monitoring burst read traffic.</li></ul>                                        |
| LowTopicBytesOutPerSec<br />                | <ul><li>Alarm Rule: Low topic data read throughput alert</li><li>Purpose and Scenario: Monitor if the data read per second on a specified topic falls below the threshold, typically used for monitoring potential link degradation or drops to zero.</li></ul>                                                           |
| LowTopicBytesOutPerSecDayToDayChange<br />  | <ul><li>Alarm Rule: Day-to-day drop in topic data read throughput exceeding threshold alert</li><li>Purpose and Scenario: Monitor if the day-to-day drop rate of data read per second on a specified topic exceeds the threshold, generally used for monitoring data flow degradation or business fluctuations.</li></ul> |
| HighGroupConsumeRatePerTopic<br />          | <ul><li>Alarm Rule: Consumer group high consumption rate alert</li><li>Purpose and Scenario: Monitor if the consumption rate per second in a specified consumer group exceeds the threshold, generally used for monitoring potential read anomalies.</li></ul>                                                            |
| LowGroupConsumeRatePerTopic<br />           | <ul><li>Alert Rule: Consumer Group Low Consumption Rate Alert</li><li>Function and Scenario: Monitors the specified Consumer Group to check if the number of messages consumed per second falls below a threshold, typically used to detect reading anomalies.</li></ul>                                                  |
| HighGroupConsumerLag<br />                  | <ul><li>Alert Rule: Consumer Group Consumption Lag Alert</li><li>Function and Scenario: Monitors the specified Consumer Group to determine if the consumption lag exceeds a threshold, usually applied to detect reading anomalies.</li></ul>                                                                             |


# Dashboard Configuration
Source: https://docs.automq.com/automq/observability/dashboard-configuration

Monitor and optimize your AutoMQ cloud-native Kafka environment with customizable Grafana dashboards, enhancing scalability and cost-efficiency.

The AutoMQ team has set up a series of Grafana dashboard templates tailored to production environment scenarios. Once users export metrics to Prometheus, they can import these Grafana dashboard templates, configure the Grafana data source to link to the respective Prometheus, and begin monitoring AutoMQ.

<Tip>
  To obtain the Grafana dashboard template collection, you can reach out to the AutoMQ team by [filling out the form](https://www.automq.com/contact).
</Tip>

### Cluster Overview

**Cluster Overview** provides monitoring information at the cluster level, including metrics like the number of nodes, data size, and cluster traffic. It also includes an overview with dimensions such as Topic, Group, and Broker, allowing users to delve into detailed monitoring information.

<img />

### Broker Dashboard

**Broker Metrics** provides monitoring data for brokers, including metrics like connection count, partition count, node traffic, and node requests.

<img />

### Topic Dashboard

**Topic Metrics** offers tailored monitoring for topics, covering metrics such as message throughput, total data volume, partition count, and consumption lag.

<img />

### Group Dashboard

**Group Metrics** delivers monitoring indicators for groups, encompassing consumption rate and consumption latency.

<img />

### Detail Dashboard

**Detailed Metrics** provide a robust set of metrics at the implementation layer, allowing developers to effectively monitor, troubleshoot, and gain deeper insights into the operational mechanisms of AutoMQ.

<img />


# Integrating Metrics with Prometheus
Source: https://docs.automq.com/automq/observability/integrating-metrics-with-prometheus

Integrate AutoMQ with Prometheus for metrics collection and visualization. Leverage cloud-native scalability, Kafka compatibility, and cost efficiency.

Prometheus is a widely adopted service for monitoring, alerting, and time series data storage. It facilitates the collection, storage, and querying of metrics. This article outlines how to integrate AutoMQ with Prometheus to achieve metrics data collection and querying.

## Metrics Integration

AutoMQ Open Source uses the OpenTelemetry SDK to implement metrics collection and export, enabling the exposure of Apache Kafka business metrics and underlying storage-related metrics. Both types of metrics are unified and transformed using the OTLP format for exposure.

<img alt="Based on the diagram above, first choose an appropriate metrics export solution to export metrics data to Prometheus. There are three ways to export metrics data:" />

For the above figure, first select an appropriate metrics export solution to export metrics data to Prometheus. There are three methods to export metrics data:

* Using OpenTelemetry Collector

* Using Prometheus OTLP Receiver

* Directly Pulling Metrics from the Server

## Prometheus Metrics Definition

After integrating the metrics into Prometheus through either the OT Collector or Prometheus OTLP Receiver, you can refer to [Prometheus Metrics▸](/automq/observability/prometheus-metrics) to view the metrics definitions.

## Option 1: Using OpenTelemetry Collector

1. Start the OpenTelemetry Collector and configure the Prometheus Exporter to collect and export metrics data.

```yaml theme={null}
exporters:
  prometheus:
    endpoint: $endpoint_for_promehteus
```

2. When starting the Prometheus service, make sure to add the following configuration.

```yaml theme={null}
scrape_configs:
  - job_name: $job_name
    scrape_interval: 5s
    honor_labels: true # DO NOT CHANGE THIS
    static_configs:
      - targets: ["$endpoint_for_prometheus"]
```

3. Configure AutoMQ to push metrics data to the Collector.

```properties theme={null}
s3.telemetry.metrics.exporter.type=otlp
s3.telemetry.exporter.otlp.protocol=grpc
s3.telemetry.exporter.otlp.endpoint=http://${endpoint}:4317

# Or with Http Protocol
# s3.telemetry.exporter.otlp.protocol=http
# s3.telemetry.exporter.otlp.endpoint=http://${endpoint}:4318/v1/metrics
```

## Option 2: Using Prometheus OTLP Receiver

### **Integration into Self-Managed Open Source Prometheus**

According to Prometheus's [version requirements](https://prometheus.io/docs/prometheus/latest/querying/api/#otlp-receiver), the user's Prometheus service needs to meet the following conditions:

* Prometheus version must be at least 2.47.

* Prometheus needs to enable feature flags: --enable-feature=otlp-write-receiver. Refer to the command below for enabling it.

```bash theme={null}

./prometheus --config.file=prometheus.yml --enable-feature=otlp-write-receiver

```

<Info>
  If your current self-hosted Prometheus service is based on the [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) Enterprise edition, ensure that the version of [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) is at least 1.92.0. Configure the access point according to the relevant documentation on [Sending Data via OpenTelemetry](https://docs.victoriametrics.com/single-server-victoriametrics/#sending-data-via-opentelemetry).
</Info>

### **Integrate into Cloud Provider's Prometheus Service**

If users use managed services provided by public cloud vendors, it is advisable to consult the technical staff from the cloud vendor. Taking Alibaba Cloud as an example, Alibaba Cloud's Prometheus service provides ready-to-use support for the OT protocol. Refer to the [OT Protocol Data Reporting Documentation](https://help.aliyun.com/arms/prometheus-monitoring/instructions-for-using-the-reporting-address-of-opentelemetry-indicator?spm=a2c4g.11186623.0.i1).

After starting the Prometheus service, configure AutoMQ to push metrics to Prometheus:

```properties theme={null}
s3.telemetry.metrics.exporter.type=otlp
s3.telemetry.exporter.otlp.protocol=http
s3.telemetry.exporter.otlp.endpoint=http://${endpoint}:9090/api/v1/otlp/v1/metrics

# Or Use with Aliyun Prometheus OpenTelemetry Endpoint
# s3.telemetry.exporter.otlp.endpoint=https://${endpoint}/api/v1/metrics
```

## Solution Three: Pull Metrics Directly from the Server

In this mode, AutoMQ acts as a Prometheus HTTP Server, and users can configure scrape tasks on the Prometheus backend service to fetch Prometheus metrics from specific endpoints. The configuration method is as follows:

```properties theme={null}
s3.telemetry.metrics.exporter.type=prometheus
s3.metrics.exporter.prom.host=127.0.0.1
s3.metrics.exporter.prom.port=9090
```


# Overview
Source: https://docs.automq.com/automq/observability/overview

Discover AutoMQ's Kafka compatibility and cloud-native scalability. Elevate monitoring with Grafana dashboards and Prometheus integration for optimized alerts.

This document explains how to monitor, set up alerts, and visualize dashboards when using AutoMQ.

### Metrics Collection

AutoMQ is fully compatible with Apache Kafka. Along with supporting native Kafka metrics, it also introduces new system metrics such as S3 Stream. AutoMQ system metrics are exposed in the following ways:

* **Native JMX Metrics Query:** AutoMQ Open Source supports Apache Kafka's native JMX metrics query. All metrics can be accessed through the JMX API.

* [Integrating Metrics with Prometheus▸](/automq/observability/integrating-metrics-with-prometheus)**:** AutoMQ Open Source uses the OpenTelemetry SDK for metrics collection and export. Users can deploy an OTLP Collector or use components like Prometheus OTLP Receiver to report and transfer metrics to Prometheus.

### Visualization & Monitoring Alerts

In real-world production environments, the AutoMQ team has set up a collection of Grafana dashboard templates and monitoring alert templates. Once users export metrics to Prometheus, they can import these Grafana dashboard templates and configure the Grafana data source to align with Prometheus, thereby starting the monitoring of AutoMQ.

<Tip>
  To obtain the Grafana dashboard templates and monitoring alert templates, you can [contact the AutoMQ team](https://www.automq.com/contact) by filling out the form.
</Tip>

### Operation Guide

* [Integrating Metrics with Prometheus▸](/automq/observability/integrating-metrics-with-prometheus)

* [Dashboard Configuration▸](/automq/observability/dashboard-configuration)

* [Alert Rule Configuration▸](/automq/observability/alert-rule-configuration)


# Prometheus Metrics
Source: https://docs.automq.com/automq/observability/prometheus-metrics

AutoMQ offers Kafka-compatible metrics with Prometheus format, enhancing cloud-native scalability and cost efficiency. Understand performance deeply with detailed insights.

This document provides detailed observability metrics for AutoMQ, enabling you to better understand its performance and operational status.

<Info>
  AutoMQ metrics are defined and presented in the Prometheus format. If other protocol formats are needed, independent conversion is required.
</Info>

## **General Metrics**

### Kafka\_server\_connection\_count

The current number of connections established by the node.

* Type: Gauge

### Kafka\_network\_threads\_idle\_rate

The idle rate of Kafka SocketServer network threads, ranging from \[0, 1.0].

* Type: Gauge

### Kafka\_io\_threads\_idle\_time\_nanoseconds\_total

The idle time of Kafka request handler threads is measured by the Apache Kafka native metric RequestHandlerAvgIdlePercent and is expressed as a percentage. By differentiating over time, you can determine the thread idle rate. Note that when a node functions as both a Controller and a Broker, each having its own request handler, this metric represents the combined value of both the Controller and Broker. The maximum idle rate derived from differentiation in this scenario is 2.0.

* Type: Counter

## Controller Metrics

### Kafka\_controller\_active\_count

This indicates whether the current Controller node is active. A metric value of 1 signifies it is active, while 0 indicates it is inactive.

* Type: Gauge

### Kafka\_broker\_active\_count

The number of active Brokers in the current cluster.

* Type: Gauge

### Kafka\_broker\_fenced\_count

The number of Brokers that are fenced in the current cluster.

* Type: Gauge

### Kafka\_topic\_count

Total number of topics in the current cluster.

* Type: Gauge

### Kafka\_partition\_total\_count

Total number of partitions in the current cluster.

* Type: Gauge

### Kafka\_partition\_offline\_count

Total number of partitions without leaders in the current cluster.

* Type: Gauge

### Kafka\_stream\_auto\_balancer\_metrics\_time\_delay\_milliseconds

The delay time for each broker node in the cluster to report AutoBalancer monitoring metrics. When this delay time exceeds a certain threshold, the broker node is deemed out-of-sync by the AutoBalancer and excluded from partition reassignment by the AutoBalancer.

* Type: Gauge

* Labels:

  * node\_id: The ID of the node reporting AutoBalancer monitoring metrics.

### Kafka\_stream\_s3\_object\_count

The current total number of objects uploaded to object storage by the cluster, categorized by object state.

* Type: Gauge

* Labels:

  * State: Object state, divided into the following three categories:

    * Prepared: Objects that have not yet completed writing and have not been committed

    * Committed: Objects that have completed writing and have been committed

    * Mark\_destroyed: Objects marked for deletion, which will be removed from object storage after a certain delay

### Kafka\_stream\_s3\_object\_size\_bytes

The total size of objects uploaded to object storage by the current cluster.

* Type: Gauge

### Kafka\_stream\_stream\_object\_num

The number of StreamObjects uploaded to object storage by the current cluster.

* Type: Gauge

### Kafka\_stream\_stream\_set\_object\_num

The number of StreamSetObjects uploaded to object storage by each Broker within the current cluster.

* Type: Gauge

* Labels:

  * node\_id: The corresponding Broker node ID

## Broker Metrics

### Kafka\_message\_count\_total

The derivative of the total number of messages received by the Broker node with respect to time gives the message count throughput.

* Type: Counter

* Labels:

  * topic

### Kafka\_network\_io\_bytes\_total

The derivative of the total size of messages received and sent by the Broker node with respect to time gives the message size throughput.

* Type: Counter

* Labels:

  * topic

  * partition

  * direction:

    * "in": indicates incoming messages

    * "out": indicates outgoing messages

### Kafka\_topic\_request\_count\_total

The total number of requests received for each Topic on the Broker node includes only the produce and fetch types of requests.

* Type: Counter

* Labels:

  * topic

  * type: Request Type

    * produce

    * fetch

### Kafka\_topic\_request\_failed\_total

The total number of request failures for each topic on the Broker node, including only produce and fetch request types.

* Type: Counter

* Labels:

  * topic

  * type: Request Type

    * produce

    * fetch

### Kafka\_request\_count\_total

The total number of requests received by the Broker node.

* Type: Counter

* Labels:

  * type: Request Type

  * version: The API version for the request of this type

### Kafka\_request\_error\_count\_total

The total number of failed requests on the Broker node. Note that this metric also accounts for successful requests, where the error code for a successful request is NONE.

* Type: Counter

* Labels:

  * type: Request Type

  * error: Error code, with NONE indicating a successful request

### Kafka\_request\_size\_bytes\_total

The total size of requests received by the Broker node.

* Type: Counter

* Labels:

  * type: Request Type

### Kafka\_request\_size\_50p(99p/mean/max)\_bytes

The size of requests received by Broker nodes, represented by different percentiles.

* Type: Gauge

* Labels:

  * type: Request Type

### Kafka\_request\_time\_milliseconds\_total

The total time taken by Broker nodes to process requests.

* Type: Counter

* Labels:

  * type: Request Type

### Kafka\_request\_time\_50p(99p/mean/max)\_milliseconds

Processing time for broker node requests, shown in various percentiles.

* Type: Gauge

* Labels:

  * type: Request Type

### Kafka\_request\_queue\_time\_milliseconds\_total

Total request queue time for broker nodes. When Kafka IO threads are busy, it results in longer request queue times.

* Type: Counter

* Labels:

  * type: Request Type

### Kafka\_request\_queue\_time\_50p(99p/mean/max)\_milliseconds

Request queue time for broker nodes, illustrated in different percentiles.

* Type: Gauge

* Labels:

  * type: Request Type

### Kafka\_response\_queue\_time\_milliseconds\_total

The response queue time on Broker nodes increases when Kafka Network threads are busy.

* Type: Counter

* Labels:

  * type: Request Type

### Kafka\_response\_queue\_time\_50p(99p/mean/max)\_milliseconds

Broker node response queue time is represented by different percentiles.

* Type: Gauge

* Labels:

  * type: Request Type

### Kafka\_request\_queue\_size

The request queue size for the broker node.

* Type: Gauge

### Kafka\_response\_queue\_size

The response queue size for the broker node.

* Type: Gauge

### Kafka\_purgatory\_size

The number of requests in the producer or fetch purgatory on the broker node.

* Type: Gauge

* Labels:

  * type:

    * Produce

    * Fetch

### Kafka\_partition\_count

The number of partitions currently assigned to the broker node.

* Type: Gauge

### Kafka\_logs\_flush\_time\_50p(99p/mean/max)\_milliseconds

The log flush time of the broker node; in AutoMQ, this indicates the flush time of Delta WAL, expressed by different percentiles.

* Type: Gauge

### Kafka\_log\_end\_offset

The maximum logical offset for each partition on the broker node.

* Type: Gauge

* Labels:

  * topic

  * partition

### Kafka\_log\_size

The message size for each partition on the broker node.

* Type: Gauge

* Labels:

  * topic

  * partition

### Kafka\_group\_commit\_offset

The consumption offset for each Consumer Group on the corresponding partition; note that this metric is reported by the Broker where the Group Coordinator for each Consumer Group resides.

* Type: Gauge

* Labels:

  * consumer\_group

  * topic

  * partition

### Kafka\_group\_count

The number of Consumer Groups managed by the Broker node where each Group Coordinator is located.

* Type: Gauge

### Kafka\_group\_preparing\_rebalance\_count

The number of Consumer Groups that are preparing to rebalance.

* Type: Gauge

### Kafka\_group\_completing\_rebalance\_count

Number of Consumer Groups waiting for state assignment from the Leader.

* Type: Gauge

### Kafka\_group\_stable\_count

Number of Consumer Groups in a Stable state.

* Type: Gauge

### Kafka\_group\_empty\_count

Number of Consumer Groups with no members but not yet expired.

* Type: Gauge

### Kafka\_group\_dead\_count

Number of Consumer Groups with no members and metadata already removed.

* Type: Gauge

### Kafka\_stream\_upload\_size\_bytes\_total

Total size of data uploaded by Broker nodes to object storage.

* Type: Counter

### Kafka\_stream\_download\_size\_bytes\_total

The total size of data downloaded from object storage by the Broker node.

* Type: Counter

### Kafka\_stream\_network\_inbound\_usage\_bytes\_total

The total inbound bandwidth usage of the Broker node, including received messages and data downloaded from object storage, can be analyzed over time to determine inbound throughput.

* Type: Counter

### Kafka\_stream\_network\_outbound\_usage\_bytes\_total

The total outbound bandwidth usage of the Broker node, including consumed messages and data uploaded to object storage, can be analyzed over time to determine outbound throughput.

* Type: Counter

### Kafka\_stream\_network\_inbound\_available\_bandwidth\_bytes

Inbound throughput reserved for cold reads and Compaction on the Broker node—when this value is less than what is required for cold reads and Compaction inbound traffic, the respective requests are queued in the rate limiting queue, thus not affecting the normal message sending and receiving traffic. Note that this metric only represents the instantaneous value at the time of sampling and is for reference only due to constraints from the sampling interval and rate limiting strategy implementation.

* Type: Gauge

### Kafka\_stream\_network\_outbound\_available\_bandwidth\_bytes

Outbound throughput reserved for cold reads and Compaction on the Broker node—when this value is less than what is required for cold reads and Compaction outbound traffic, the respective requests are queued in the rate limiting queue, thus not affecting the normal message sending and receiving traffic. Note that this metric only represents the instantaneous value at the time of sampling and is for reference only due to constraints from the sampling interval and rate limiting strategy implementation.

* Type: Gauge

### Kafka\_stream\_network\_inbound\_limiter\_queue\_time\_50p(99p/mean/max)\_nanoseconds

The queuing time for incoming requests for cold reads and Compaction in the rate limiting queue during execution.

* Type: Gauge

### Kafka\_stream\_network\_outbound\_limiter\_queue\_time\_50p(99p/mean/max)\_nanoseconds

The queuing time for outgoing requests for cold reads and Compaction in the rate limiting queue during execution.

* Type: Gauge

### Kafka\_stream\_operation\_latency\_50p(99p/mean/max)\_nanoseconds

The operational duration of each stage in the AutoMQ S3Stream module.

* Type: Gauge

* Labels:

  * operation\_type

  * operation\_name

### Kafka\_stream\_cert\_expiry\_timestamp\_milliseconds

This metric shows the expiration UNIX timestamp of the TLS certificate, measured in milliseconds.

* Type: gauge

* Labels:

  * instance: Instance ID.

  * job: Task identifier.

  * host\_name: System hostname.

  * cert\_subject: Certificate subject.

  * cert\_type: Certificate type, where `server_cert` represents a server certificate, and `truststore_cert` represents a CA certificate.

### Kafka\_stream\_cert\_days\_remaining

This metric indicates the number of days remaining until the expiration of the TLS certificate from the current moment.

* Type: gauge

* Labels:

  * instance: Instance ID.

  * job: Task identifier.

  * host\_name: System hostname.

  * cert\_subject: Certificate subject.

  * cert\_type: Certificate type, where `server_cert` represents a server certificate, and `truststore_cert` represents a CA certificate.


# Glossary
Source: https://docs.automq.com/automq/reference/glossary

AutoMQ offers cloud-native scalability, Kafka compatibility, and cost efficiency with unique modules like S3Stream and WAL. Explore our glossary for detailed insights.

This article lists the technical terms and provides a brief overview of various modules involved in AutoMQ.

## Terminology Classification

* **Cloud Service Concepts:** Covers the cloud services and product components used by AutoMQ. Users can refer to the documentation of each cloud provider for more detailed information.

* **Apache Kafka Concepts:** Covers some existing concepts of Apache Kafka that may vary due to AutoMQ's implementation.

* **AutoMQ Concepts:** Covers the new concepts defined within the various modules of AutoMQ.

## Cloud Service Terminology

### EBS

EBS (Elastic Block Store) is a high-performance, scalable, durable, and low-latency block storage service. In AutoMQ's system design, EBS is used to temporarily store some message data that has not yet been uploaded to object storage, aiming to achieve lower message sending and receiving latency. Different cloud service providers may have different product names for EBS.

### S3

S3 (Simple Storage Service) is a secure, durable, and highly scalable object storage service. In AutoMQ's system design, object storage is used as the primary storage medium for messages, allowing for on-demand usage and pay-as-you-go pricing, reducing storage costs by up to 90% compared to Apache Kafka. S3 also refers to object storage in subsequent documentation, and different cloud service providers may have different product names for their object storage services.

### Bucket

Bucket is the fundamental container for object storage services, used for efficient data management. When deploying AutoMQ, some Buckets need to be pre-allocated as message storage configuration.

### Auto Scaling Group (ASG)

Auto Scaling Group (ASG) is a service that can automatically adjust computing resources to meet application load demands. ASG can automatically increase or decrease the number of instances in a group of virtual hosts, ensuring high availability of applications and optimizing costs. AutoMQ uses ASG to implement automatic elasticity and scaling features. Different cloud providers may use different product names for ASG.

## Apache Kafka Terminology

### Broker

Broker is the logical role in the Apache Kafka system responsible for processing, storing, and transmitting messages. Multiple Broker nodes together form a Kafka cluster. In AutoMQ's system design, Broker specifically refers to the logical role that handles routine message sending and receiving, excluding the Controller role used for scheduling and allocation.

### Controller

Controller is the logical role in the Apache Kafka system responsible for scheduling and coordinating task allocation among multiple nodes. Depending on the version, the Controller might have different implementation schemes. In AutoMQ's system design, the Controller is built on the KRaft mode, no longer relying on ZooKeeper services. Among multiple Controller nodes, there will be one Active Controller node serving as the primary decision-making node.

### Partition

Partition is the logical shard of an Apache Kafka Topic, used to achieve parallel data processing and increase throughput. Each Partition is an ordered, immutable sequence of messages. In AutoMQ's system design, Partition retains its original functional definition but no longer stores data on local disks. Instead, it leverages object storage to achieve infinite capacity and on-demand scalability.

## AutoMQ Terminology

### AutoMQ

AutoMQ is a next-generation Apache Kafka release redesigned based on cloud-native concepts, offering up to 10x lower TCO (see [Cost Benchmark](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)) and seconds-level scaling while being 100% compatible with the Apache Kafka protocol.

### S3Stream

S3Stream is a low-latency, high-throughput, elastic, and cost-effective streaming repository built on cloud storage services, integrating externally through the Stream operation interface. AutoMQ replaces Apache Kafka's Log storage with S3Stream, ensuring 100% compatibility with Apache Kafka's upper-layer functionalities while offering up to 10x lower TCO (see [Cost Benchmark](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)) and seconds-level scaling.

### S3Url

S3Url is a unified configuration item used by AutoMQ for rapid cluster deployment, containing information such as object storage access points and identity credentials. It is recommended to use the installation tool to generate the S3Url configuration to pre-validate parameter legality and resource compatibility, avoiding the cumbersome cluster ID generation and storage formatting operations in Apache Kafka.

### WAL

WAL (Write-Ahead Log) is a high-throughput persistent write buffer in the S3Stream library. It temporarily caches data not yet committed to object storage. WAL supports multiple storage backends, including S3 WAL, EBS WAL, Regional EBS WAL, and NFS WAL. Latency characteristics depend on the backend: EBS WAL and Regional EBS WAL are designed for lower-latency workloads, while S3 WAL is the default option for AutoMQ Open Source and is better suited to latency-insensitive scenarios. In AutoMQ, WAL is allocated at the Broker level. When a Broker receives a message, it first writes the message sequentially to the WAL and immediately returns a client response, then asynchronously uploads the WAL data to object storage.

### Stream Object

Stream Object is the smallest unit for storing Stream data in S3Stream. Data from each Stream is distributed across multiple Stream Objects, which collectively simulate an infinite Stream.

### Stream Set Object

Stream Set Object is a temporary data structure in S3Stream used to merge scattered Stream write requests. When uploading the temporary data from WAL to object storage, data from multiple scattered Streams is merged into a single Stream Set Object before uploading. Subsequently, the Stream Set Object is asynchronously classified and organized into regular Stream Objects.


# Release notes
Source: https://docs.automq.com/automq/releases/release-notes





# Software Artifact
Source: https://docs.automq.com/automq/releases/software-artifact

AutoMQ delivers cloud-native Kafka compatibility with full scalability, cost efficiency, and Java support. Deploy on Linux for optimal performance.

AutoMQ is a cloud-native Kafka distribution developed entirely in the Java language, supporting most Java platforms. For optimal performance, deploy on the Linux operating system, preferably on amd64 or arm64 architecture. If you need further guidance on deployment, refer to [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux).

## Versioning

AutoMQ adheres to the semantic versioning specification, formatted as `{major version}.{minor version}.{patch version}`, where:

* major version: Incremented for significant upgrades or incompatible changes.

* minor version: Incremented when new features are added while maintaining backward compatibility.

* patch version: Used for bug fixes.

Each AutoMQ version is compatible with specific versions of Apache Kafka. Refer to the [release notes](https://github.com/AutoMQ/automq/releases) for historical versions and detailed version mappings to choose the appropriate AutoMQ version.

## Software Artifacts

<Tabs>
  <Tab title="Binaries">
    AutoMQ provides binary packages as pure Java installations, requiring Java 17 or later to run. You can use the following download script to obtain and deploy the latest version. Java 17 can be downloaded via the following link: [Java 17 Download Link](https://www.oracle.com/java/technologies/downloads/#java17).

    ```bash theme={null}
    curl -O https://download.automq.com/community_edition/artifacts/automq-kafka-latest.tgz
    ```
  </Tab>

  <Tab title="Docker">
    You can use the following command to get the latest release version Docker image.

    ```bash theme={null}

    docker pull automqinc/automq:latest

    ```

    For more historical versions, visit [this link](https://hub.docker.com/r/automqinc/automq/tags).
  </Tab>
</Tabs>

## AutoMQ Open Source Code

Refer to [Licensing and Enterprise Features▸](/automq/what-is-automq/licensing-and-enterprise-features). You can independently deploy clusters using AutoMQ Open Source, as its source code is publicly available. You can find the AutoMQ Open Source source code in the [GitHub project](https://github.com/AutoMQ/automq).


# Overview
Source: https://docs.automq.com/automq/table-topic/overview

AutoMQ's Table Topic offers seamless Iceberg integration, auto-scaling, and ETL-free data lake ingestion, enhancing real-time analytics with S3 integration. Ideal for AWS.

AutoMQ Table Topic provides built-in Iceberg integration, allowing streaming data to flow into the data lake for analysis and querying. This article introduces the technical architecture, principles, and core concepts related to the Table Topic feature.

## **Architecture and Benefits**

The AutoMQ Table Topic feature enables one-stop real-time data lake entry and query analysis through its built-in stream table architecture. The technical architecture is outlined as follows:

<img />

Compared to traditional ETL data lake solutions, Table Topic offers the following advantages:

* **Out-of-the-box:** With just one click, AutoMQ Table Topic can be activated, streaming data into Iceberg tables for continuous and real-time analysis.

* **ETL-Free (Extract, Transform, Load):** Traditional data lake ingestion methods often require tools like Kafka Connect or Flink. Table Topic eliminates the need for such ETL pipelines, significantly cutting costs and reducing operational complexity.

* **Auto-Scaling:** AutoMQ features a stateless and elastic architecture, enabling brokers to scale up or down with dynamic partition reassignment. Table Topic uses this framework to efficiently manage data ingestion rates ranging from hundreds of MiB/s to several GiB/s.

* **Seamless Integration with AWS S3 Table:** Table Topic integrates with S3 Table, leveraging its Data Catalog and maintenance functions, such as compression, snapshot management, and unreferenced file deletion. This integration also enables large-scale data analytics via AWS Athena.

## Constraints and Limitations

To utilize AutoMQ's Table Topic functionality, the following conditions must be met:

* **Version Constraint:** Requires AutoMQ version >= 1.5.

* **Feature Constraint:** The Table Topic feature must be configured at the time of deploying the AutoMQ cluster for subsequent use. Once the cluster is deployed, the Table Topic feature cannot be enabled.

* **Catalog Requirements:** To use Table Topic, users must provide an externally accessible Data Catalog service. Currently, AutoMQ supports the following Catalog types:

  * **AWS S3 Table Catalog:** AWS S3 offers a new Table Bucket with integrated Catalog management and data lake storage.

  * **AWS Glue Catalog:** AWS Glue provides cloud-based unified Catalog management, supporting integration with query tools such as Athena.

  * **Hive Catalog:** Users can either set up their own Hive Metastore Catalog within the Hadoop ecosystem or choose a managed EMR HMS service offered by a cloud provider.

## Procedure

To use the AutoMQ Table Topic feature, users should follow these configuration steps:

<img />


# Table Topic Configuration
Source: https://docs.automq.com/automq/table-topic/table-topic-configuration

Optimize your Kafka data management with AutoMQ's Table Topic feature. Enjoy seamless Kafka compatibility, cloud-native scalability, and cost-efficient Iceberg integration.

This document outlines the configuration of the AutoMQ Table Topic feature, which includes the Schema Registry, Catalog, and the setup of the AutoMQ server.

## Schema Registry Configuration

The Schema Registry manages the Kafka Topic message Schema, ensuring data consistency and compatibility between producers and consumers. AutoMQ Table Topic uses the Schema Registry to parse Topic data and synchronize with the Iceberg table structure, supporting Schema evolution.

### Constraints

Table Topic supports only the following Schema Registry implementations.

* **Confluent Schema Registry**: Facilitates schema management for Kafka and offers a REST API.

* **Aiven Karapace**: An open-source Schema Registry that is compatible with the Confluent Schema Registry's REST API.

<Info>
  For Schema Registry client options, use the `automq.table.topic.schema.registry.config.*` prefix. AutoMQ removes this prefix and passes the remaining keys to the Schema Registry client as `schema.registry.*` options.
</Info>

### Configure Schema Registry

Users specify the Schema Registry service address via the following parameters (cluster-level configuration).

**Configuration Parameters**:

* `automq.table.topic.schema.registry.url`: The URL of the Schema Registry (e.g., `http://schema-registry.example.com:8081`). AutoMQ uses this URL to retrieve and synchronize the Schema.

* `automq.table.topic.schema.registry.config.*`: Optional Schema Registry client configuration. For example, `automq.table.topic.schema.registry.config.ssl.truststore.type=PEM` becomes `schema.registry.ssl.truststore.type=PEM`.

**Note**:

* Configure this when using the deprecated `automq.table.topic.schema.type=schema` value or when either `automq.table.topic.convert.value.type` or `automq.table.topic.convert.key.type` is set to `by_schema_id` or `by_latest_schema`.

**Example**:

```properties theme={null}
automq.table.topic.schema.registry.url=https://schema-registry.example.com:8081
automq.table.topic.schema.registry.config.ssl.truststore.type=PEM
automq.table.topic.schema.registry.config.ssl.truststore.certificates=${CA_PEM}
```

## Catalog Configuration

In Apache Iceberg, the **Catalog** plays a crucial role in managing table metadata. Its primary responsibilities include:

1. **Tracking the current metadata pointer for each Iceberg table.** This pointer indicates the location of the latest metadata file (e.g., `vN.metadata.json`).

2. **Providing atomic operations to update this metadata pointer.** This is critical for ensuring data consistency during commit processes such as writing new data or evolving schemas.

3. **Organizing tables into namespaces** and providing methods to list, create, delete, and rename tables.

Essentially, a Catalog serves as the entry point for any Iceberg operation, guiding the query engine on where to find authoritative information regarding a table's schema, partitions, snapshots, and data files.

### Configuring the Catalog

Table Topic uses an external Iceberg Catalog to manage the metadata of its "table topics."

* **General Configuration Prefix**: All Iceberg Catalog settings related to the AutoMQ Table Topic use the prefix `automq.table.topic.catalog.*`.

* **Primary Configuration Key**: Specifies the type of Iceberg Catalog, as defined by the `automq.table.topic.catalog.type` attribute.

* **Catalog option pass-through**: AutoMQ removes the `automq.table.topic.catalog.` prefix and passes the remaining key-value pairs to the Iceberg Catalog implementation. For example, `automq.table.topic.catalog.header.x-custom=my-x` becomes `header.x-custom=my-x`. Use this for Catalog-specific options such as REST headers, custom `io-impl`, or authentication properties.

* **Hadoop option pass-through**: AutoMQ removes the `automq.table.topic.hadoop.` prefix and adds the remaining key-value pairs to the Hadoop configuration used by the Catalog.

AutoMQ Table Topic supports the following catalog types: `rest`, `glue`, `tablebucket`, `nessie`, `hive`.

### Supported Catalog Types and Configuration

#### 1. REST Catalog

This method uses the standard Iceberg REST Catalog service.

* **Type Setting**: `automq.table.topic.catalog.type=rest`

* **Configuration Property**:

  * `automq.table.topic.catalog.uri` : The URI for the REST Catalog service (e.g., `http://rest:8181`).

  * `automq.table.topic.catalog.warehouse` : Defines the S3 path for the Iceberg data warehouse. If not set, it will default to the Iceberg directory of the DataBucket.

  * `automq.table.topic.catalog.oauth2-server-uri` : The OAuth2 token endpoint used by the REST Catalog client when client credentials authentication is enabled.

  * `automq.table.topic.catalog.credential` : The client credential used to fetch an OAuth2 token from the REST Catalog token endpoint.

  * `automq.table.topic.catalog.token` : A static bearer token. If this is set, REST Catalog requests use it in the HTTP Authorization header.

  * `automq.table.topic.catalog.scope` : The OAuth2 scope used when requesting a token through the client credentials flow.

  * `automq.table.topic.catalog.header.*` : Additional HTTP headers for REST Catalog requests. For example, `automq.table.topic.catalog.header.x-custom=my-x` sends `x-custom: my-x`.

* **Required Permissions**:

  * Credentials (if authentication is needed) that allow listing, creating, reading, and updating table metadata.

  * AutoMQ requires read/write/delete permissions for the S3 warehouse path.

**Example**:

```properties theme={null}
automq.table.topic.catalog.type=rest
automq.table.topic.catalog.uri=http://rest:8181
automq.table.topic.catalog.warehouse=s3://automq-bucket/wh/
```

**Authentication examples**:

```properties theme={null}
# Static bearer token
automq.table.topic.catalog.type=rest
automq.table.topic.catalog.uri=http://rest:8181
automq.table.topic.catalog.token=${REST_CATALOG_TOKEN}
```

```properties theme={null}
# OAuth2 client credentials
automq.table.topic.catalog.type=rest
automq.table.topic.catalog.uri=http://rest:8181
automq.table.topic.catalog.oauth2-server-uri=http://rest:8181/oauth/tokens
automq.table.topic.catalog.credential=${CLIENT_ID}:${CLIENT_SECRET}
automq.table.topic.catalog.scope=catalog
```

#### 2. AWS Glue Data Catalog

Using AWS Glue as the Iceberg Catalog.

* **Type Setting**: `automq.table.topic.catalog.type=glue`

* **Configuration Property**:

  * `automq.table.topic.catalog.warehouse`: Specifies the S3 path for the Iceberg data warehouse.

* **Required Permissions**:

* AutoMQ requires database and table management permissions for AWS Glue.

* AutoMQ needs read/write/delete permissions on the S3 warehouse path.

**Example:**

```properties theme={null}
automq.table.topic.catalog.type=glue
automq.table.topic.catalog.warehouse=s3://automq-bucket/glue/
```

#### 3. TableBucket (S3Table)

* **Type Setting**: `automq.table.topic.catalog.type=tablebucket`

* **Configuration Property**:

  * `automq.table.topic.catalog.warehouse`: Specifies the TableBucket ARN (e.g., `arn:aws:s3tables:::bucket-name`).

* **Required Permissions**:

  * AutoMQ requires read and write access to the S3Table.

**Example**:

```properties theme={null}
automq.table.topic.catalog.type=tablebucket
automq.table.topic.catalog.warehouse=arn:aws:s3tables:us-east-1:xxxxxx:bucket/xxxxx
```

#### 4. Nessie Catalog

Use Project Nessie, a transactional data lake catalog with Git-like semantics.

* **Type Setting**: `automq.table.topic.catalog.type=nessie`

* **Configuration Property**:

  * `automq.table.topic.catalog.uri` : Specifies the URI of the Nessie server (e.g., `http://nessie-server:19120/api/v2` ).

  * `automq.table.topic.catalog.warehouse` : Defines the S3 path for the Iceberg data warehouse. If not set, it will default to the Iceberg directory of the DataBucket.

**Example**:

```properties theme={null}
automq.table.topic.catalog.type=nessie
automq.table.topic.catalog.uri=http://nessie-server:19120/api/v2
automq.table.topic.catalog.warehouse=s3://automq-bucket/nessie/
```

#### 5. Hive Metastore Catalog

Use the existing Hive Metastore as an Iceberg Catalog. When configuring Hive Metastore as the Catalog, refer to [documentation](https://iceberg.apache.org/docs/latest/hive/#enabling-iceberg-support-in-hive) for enabling Iceberg in Hive.

* **Type Setting**: `automq.table.topic.catalog.type=hive`

* **Configuration Property**:

  * `automq.table.topic.catalog.uri`: Specifies the Hive Metastore URI (e.g., `thrift://hostname:9083`).

  * `automq.table.topic.catalog.warehouse` : Specifies the HDFS or S3 path for the Iceberg data warehouse. If not configured, the Iceberg directory of DataBucket will be used.

* **Authentication Configuration**:

  * **Simple Authentication**:

    * `automq.table.topic.catalog.auth=simple://?username=xxx` .

  * **Kerberos Authentication**:

```properties theme={null}
automq.table.topic.catalog.auth=kerberos://?principal=base64(clientPrincipal)&keytab=base64(keytabFile)&krb5conf=base64(krb5confFile)
```

* `clientPrincipal` : Base64 encoded client Kerberos principal.

* `keytabFile`: Base64 encoded content of the client keytab file.

* `krb5confFile`: Base64 encoded content of the `krb5.conf` file.

* `automq.table.topic.hadoop.metastore.kerberos.principal`: Kerberos principal for the Hive Metastore server (e.g., `hive/_HOST@REALM`).

* **Required Permissions**:

  * Create, alter, and drop table permissions for the Hive Metastore.

  * Kerberos authentication requires a valid principal and keytab.

  * AutoMQ needs read, write, and delete permissions in the repository path (HDFS or S3).

**Example**:

```properties theme={null}
automq.table.topic.catalog.type=hive
automq.table.topic.catalog.uri=thrift://hostname:9083
automq.table.topic.catalog.warehouse=s3://automq-bucket/hive/
automq.table.topic.catalog.auth=simple://?username=user&password=pass
```

# Table Topic Configuration

The following configurations are all topic-level settings that allow for enabling Table Topic features, adjusting the Iceberg commit interval, and configuring features such as partitioning and upsert.

## Enabling the Table Topic Feature

The AutoMQ Table Topic feature allows for the conversion of data from a specified Kafka Topic into an Apache Iceberg table to support structured queries and analysis.

**Configuration Parameters**:

* `automq.table.topic.enable` : Determines whether the Table Topic is enabled. Setting this to `true` will result in the creation of an Iceberg table for data storage.

* `automq.table.topic.namespace` : Specifies the namespace for the Iceberg table under the Catalog, adhering to the naming conventions of the Catalog (such as a combination of letters, numbers, and underscores). This should be configured at the time of Topic creation and cannot be modified later.

* `automq.table.topic.schema.type` **(deprecated)**: \[DEPRECATED] The table topic schema type configuration. This setting will be removed in a future release. Please migrate to the converter and transform configurations.

  * `schemaless`: Maps to `automq.table.topic.convert.value.type=raw` and `automq.table.topic.transform.value.type=none`.

  * `schema`: Maps to `automq.table.topic.convert.value.type=by_schema_id` and `automq.table.topic.transform.value.type=flatten`.

```properties theme={null}
automq.table.topic.enable=true
automq.table.topic.namespace=default
automq.table.topic.convert.value.type=by_schema_id
automq.table.topic.transform.value.type=flatten
```

**Compatibility Example (deprecated)**:

```properties theme={null}
automq.table.topic.schema.type=schema
```

### Record Conversion Settings

Decide how Table Topic parses Kafka records before they are written to Iceberg.

**Value converter (`automq.table.topic.convert.value.type`)**

* `raw`: Pass the Kafka payload through without interpretation. Use this when the downstream query only needs the opaque bytes (for example, storing binary blobs). Schema Registry is not consulted.
* `string`: Decode the payload as UTF-8 text and expose it as a string column. Useful for topics that store plain text or JSON strings but you still want Iceberg queries to treat them as text.
* `by_schema_id`: Resolve the schema ID encoded in the Confluent wire format (leading magic byte plus 4-byte schema ID) and deserialize Avro/Protobuf data using Schema Registry. Choose this for producers that already register schemas and include the schema ID with every record.
* `by_latest_schema`: **Only supports Protobuf format**. Fetch the most recent schema from Schema Registry and use it to decode raw Protobuf payloads that do not carry a schema ID. Configure the subject and the fully qualified message name as described below.

**Key converter (`automq.table.topic.convert.key.type`)**

The key converter accepts the same options (`raw`, `string`, `by_schema_id`, `by_latest_schema`). Configure it when the Kafka record key is meaningful for your table:

* Use `raw` when the key should stay opaque (for example, binary hash keys).
* Use `string` when the key is textual.
* Use `by_schema_id` or `by_latest_schema` when keys are structured Avro/Protobuf objects and you need individual fields for partitioning or equality deletes.

**Configuring latest-schema lookup**

When either converter uses `by_latest_schema`, the service needs hints to select the right schema version.

* `automq.table.topic.convert.value.by_latest_schema.subject` (or the key variant) overrides the Schema Registry subject. If unset, it defaults to `<topic>-value` or `<topic>-key`.
* `automq.table.topic.convert.value.by_latest_schema.message.full.name` (and the key equivalent) specifies the fully qualified Protobuf message name. If not configured, the system defaults to using the first message definition in the schema file; explicit specification is required when a single schema file contains multiple messages to avoid ambiguity.
* The latest version of the subject is used, so align your Schema Registry retention and compatibility rules accordingly.

```properties theme={null}
automq.table.topic.convert.value.type=by_latest_schema
automq.table.topic.convert.value.by_latest_schema.subject=product-value
automq.table.topic.convert.value.by_latest_schema.message.full.name=examples.clients.proto.ProductData
```

### Value Transformation

After conversion, you can apply an optional transformation to reshape the data before writing to Iceberg.

* `none`: Persist the converted value as-is. Choose this when you only need the raw payload or when downstream consumers will parse it later.
* `flatten`: Expand fields from structured Avro/Protobuf objects into top-level Iceberg columns. This is the typical choice for analytics-friendly tables.
* `flatten_debezium`: A specialized transformer that extracts the row state from the Debezium envelope. It adds a `_cdc.op` field to indicate the operation type (e.g., 'c' for create, 'u' for update, 'd' for delete). This requires a schema-aware converter (`by_schema_id` or `by_latest_schema`).

```properties theme={null}
automq.table.topic.transform.value.type=flatten_debezium
automq.table.topic.convert.value.type=by_schema_id
automq.table.topic.cdc.field=_cdc.op
automq.table.topic.id.columns=[id]
```

#### Example: none vs flatten

The following shows a single Kafka record and how query results differ between `none` and `flatten` value transformations.

Kafka record (value payload):

```json theme={null}
{
    "order_id": 1,
    "product_name": "str_1_xaji0y"
}
```

* With `none`:

```properties theme={null}
automq.table.topic.convert.value.type=by_schema_id
automq.table.topic.transform.value.type=none
```

Query result:

```text theme={null}
 _kafka_value                            | _kafka_key |                 _kafka_metadata                  | _kafka_header 
--------------------------------------------------------+------------+--------------------------------------------------+---------------
 {order_id=1, product_name=str_1_xaji0y} |            | {partition=0, offset=0, timestamp=1762138970865} | {}            
```

* With `flatten` (expanded business columns + system columns):

```properties theme={null}
automq.table.topic.convert.value.type=by_schema_id
automq.table.topic.transform.value.type=flatten
```

Query result (one row, illustrative):

```text theme={null}
 order_id | product_name  | _kafka_header | _kafka_key |                 _kafka_metadata                  
----------+---------------+---------------+------------+--------------------------------------------------
    1     | str_1_xaji0y  | {}            |            | {partition=0, offset=0, timestamp=1762138970865} 
```

### Error Handling

The `automq.table.topic.errors.tolerance` property controls how the Table Topic pipeline behaves when it encounters errors during record conversion or transformation.

* **`none`**: In this mode, any error encountered during processing will halt the pipeline for the affected topic partition. The system will continuously retry to process the problematic record, effectively blocking any further records from that partition until the issue is resolved. This mode guarantees that no data is skipped, but it can cause data processing to stall.

* **`invalid_data` (default)**: This mode skips records with content issues, such as a missing magic code or an unresolvable schema ID (`InvalidDataException`). It continuously retries on `RestClientException` / `SerializationException` and exceptions in later processing stages to prevent data loss. Other data exceptions are skipped. This is the recommended setting for most production environments.

* **`all`**: This mode skips any record that causes an exception.

```properties theme={null}
automq.table.topic.errors.tolerance=invalid_data
```

## Configuration Commit Interval

To enhance real-time analytics, you can modify the commit frequency of an Iceberg table to improve data freshness.

**Configuration Parameters**:

* `automq.table.topic.commit.interval.ms`: Specifies the data commit interval in milliseconds. The default is 60,000 (1 minute). Valid values are from 1 to 900,000 (15 minutes). Shorter intervals improve real-time capability but increase processing overhead.

**Note**:

* Frequent commits may result in commit conflicts and MetadataFile bloat, which can raise storage and query costs.

* AutoMQ automatically deletes snapshots that are older than one hour to prevent the `metadata.json` file from becoming excessively large over time.

* Regular table maintenance is necessary through compaction to merge small files, optimizing the size of the `ManifestFile` and improving query performance.

**Example**:

```properties theme={null}
automq.table.topic.commit.interval.ms=60000
```

## Optimize Query Performance: Partition Configuration

To enhance the query performance of Iceberg tables, especially in scenarios involving selective filtering, partitioning rules can be configured.

**Configuration Parameters**:

* `automq.table.topic.partition.by`: Specifies partitioning rules, supporting partitioning by fields or functions. For example, `[bucket(name, 3), month(timestamp)]` means partitioning by hashing the `name` field into 3 buckets and partitioning by the month of the `timestamp`.

**Supported Partition Strategies**:

* Bucket Partitioning: `bucket(field, N)`, partitions by the hash value of the field.

* Truncate Partitioning: `truncate(field, N)`, partitions by the truncated value of the field.

* Temporal Partitioning: `year(timestamp)`, `month(timestamp)`, `day(timestamp)`, `hour(timestamp)`.

For more details, refer to [Iceberg Partitioning Documentation](https://iceberg.apache.org/docs/latest/partitioning/).

**Note**:

* Having too many partitions may increase metadata management overhead, leading to query planning delays and higher storage costs. It is recommended to set a reasonable number of partitions based on data volume and query patterns, and to regularly optimize small files through Compaction.

**Example**:

```properties theme={null}
automq.table.topic.partition.by=[bucket(name, 3), month(timestamp)]
```

## Upsert and CDC Mode Support

To facilitate Upsert or Change Data Capture (CDC) operations, you can enable the following features for dynamic data management. A primary key must be configured to support row-level operations.

**Configuration Parameters**:

* `automq.table.topic.id.columns`: Specify the primary key columns (composite keys are supported), such as `[region, name]`.

* `automq.table.topic.upsert.enable`: Indicate whether to enable Upsert mode. Set to `true` to have the system insert or update records based on the primary key.

* `automq.table.topic.cdc.field`: Indicates the type of CDC operation, which can be either `I` (insert), `U` (update), or `D` (delete).

**Note**:

* To enable Upsert or CDC mode, it is necessary to configure `automq.table.topic.id.columns`; otherwise, only append-only writes will be supported.

* Iceberg V2 tables support row-level operations:

  * **Creation Logic**: Records inserted or updated are kept in the datafile, while deleted records are marked via a deletefile (which includes the primary key and deletion marker). An Upsert operation might create a new datafile, and the CDC's delete operation results in a deletefile.

  * **Query Logic**: Use the Merge-on-Read (MOR) mechanism to combine data files and delete files during queries, filtering out records marked for deletion using equality delete.

* Enhance query performance consistently by merging data files and delete files through Compaction.

**Example**:

1. Enable Upsert Mode:

```properties theme={null}
automq.table.topic.upsert.enable=true
automq.table.topic.id.columns=[id, name]
```

2. Enable CDC Mode

```properties theme={null}
automq.table.topic.cdc.field=op_type
automq.table.topic.id.columns=[id, name]
```


# Table Topic Quick Start
Source: https://docs.automq.com/automq/table-topic/table-topic-quick-start

Discover seamless data integration with AutoMQ's Table Topic, enhancing data lakes through cloud-native scalability and Kafka compatibility. Perfect for swift trials.

Table Topic is a core feature of AutoMQ, designed for modern data lake architectures. Its primary purpose is to create direct integration between streaming data and static data lakes through architectural innovation, addressing traditional issues like stream-batch separation, complex ETL processes, and high costs. This document is aimed at new users experiencing AutoMQ for the first time, focusing on how to deploy an AutoMQ cluster with Table Topic support in a local development environment (by deploying MinIO to provide object storage services), enabling users to quickly experience the features of Table Topic.

<Tip>
  This document is intended solely for testing and learning scenarios. For production environments, please use the Table Topic feature and refer to deployment documentation such as [Deploy Multi-Nodes Cluster on Linux▸](/automq/deployment/deploy-multi-nodes-cluster-on-linux).
</Tip>

## Prerequisites

* Linux/Mac/Windows Subsystem for Linux

* Docker

* Docker Compose version > 2.22.0

* At least 8GB of free memory

## Install and Start AutoMQ Cluster

Run the following command to download the Docker Compose configuration file for a single-node deployment and start the AutoMQ cluster using Docker Compose.

```bash theme={null}
curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/table_topic/docker-compose.yml && docker-compose up -d
```

Once started successfully, we will send Table Topic messages and perform queries using Spark SQL to experience automatic table creation, table querying and updating, schema changes, data partitioning, and CDC capabilities. Users can open [http://localhost:8888](http://localhost:8888) in their browser and interactively execute commands using Jupyter Notebook.

<img />

## Create Table

The first three steps in the Notebook involve the creation of a Table Topic, writing a single message, and triggering the creation of an Iceberg table.

<img />

## Query Data

<img />

<Tip>
  **Note:**

  There may be a delay between writing the message and the data being entered into the table. If an error occurs during the query, we recommend retrying after a short wait.
</Tip>

## Update Records

<img />

When querying updated results, the data `ops` will change from `I` to `U`.

<img />

## Delete Data

<img />

After querying the deletion results, records are empty.

<img />

## Delete Table

<img />

## Stop and Uninstall the AutoMQ Cluster

Once you have completed the functional tests, run the following commands to stop and uninstall the current AutoMQ cluster.

```bash theme={null}
curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.0/docker/table_topic/docker-compose.yml && docker-compose down -v
```

<Tip>
  For detailed parameter configurations, please refer to the [Table Topic Configuration▸](/automq/table-topic/table-topic-configuration) section.
</Tip>


# Compatibility with Apache Kafka
Source: https://docs.automq.com/automq/what-is-automq/compatibility-with-apache-kafka

AutoMQ is a cloud-native, Kafka-compatible service offering seamless integration with Apache Kafka. Enjoy scalable, cost-efficient performance while maintaining API compatibility.

AutoMQ is a next-generation Apache Kafka distribution reimagined with cloud-native principles, fully compatible with Apache Kafka protocols and features. This document provides a comprehensive overview of AutoMQ's compatibility across different versions.

## Apache Kafka Compatibility Overview

AutoMQ strategically leverages the Apache Kafka computing layer code, making only minimal modifications in the storage layer. This approach ensures full compatibility with relevant versions of Apache Kafka. Applications built on Apache Kafka can transition to AutoMQ seamlessly.

During the compatibility verification phase, AutoMQ employed Apache Kafka's test case projects and successfully passed the tests for relevant versions.

AutoMQ corresponds directly to Apache Kafka version numbers. For details on the compatibility between AutoMQ and Apache Kafka versions, please refer to the [Release Notes▸](https://github.com/AutoMQ/automq/releases/). Each AutoMQ version ensures compatibility with Kafka Client, Connector, Proxy, and other components within the Apache Kafka ecosystem.

<Info>
  AutoMQ achieves adaptation to Apache Kafka by replacing micro facets at the storage layer, allowing for swift adaptation to new community versions. It supports the latest updates to Apache Kafka versions within T+1 month.
</Info>

## Compatibility with Products in the Apache Kafka Ecosystem

AutoMQ's complete compatibility with Apache Kafka enables direct integration with all ecosystem products. The following Apache Kafka ecosystem products have passed AutoMQ's compatibility tests and are available for use.

### Data Analysis

* [RisingWave▸](/automq/integrations/data-analysis/risingwave)

* [Databend▸](/automq/integrations/data-analysis/databend)

* [Apache Doris▸](/automq/integrations/data-analysis/apache-doris)

* [Flink▸](/automq/integrations/data-analysis/flink)

* [StarRocks▸](/automq/integrations/data-analysis/starrocks)

### Kafka UI

* [Kafdrop▸](/automq/integrations/kafka-ui/kafdrop)

* [Redpanda Console▸](/automq/integrations/kafka-ui/redpanda-console)

### Observability

* [Prometheus/VictoriaMetrics▸](/automq/integrations/observability/prometheus)

* [Guance Cloud▸](/automq/integrations/observability/guance-cloud)

* [Flashcat▸](/automq/integrations/observability/flashcat)

### Data Integration

* [Debezium▸](/automq/integrations/data-integration/debezium)

* [Airbyte▸](/automq/integrations/data-integration/airbyte)


# Difference with Apache Kafka
Source: https://docs.automq.com/automq/what-is-automq/difference-with-apache-kafka

AutoMQ delivers cloud-native scalability and cost efficiency while ensuring 100% API compatibility with Apache Kafka. Experience up to 10x cost savings and enhanced flexibility.

AutoMQ is a next-generation Apache Kafka® distribution redesigned with cloud-native principles. It offers up to 10x lower TCO ([see benchmark](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)) and elasticity advantages while maintaining 100% compatibility with the Apache Kafka® protocol. This article primarily introduces the main differences and connections between AutoMQ and Apache Kafka®.

## Differences from Apache Kafka®

### Architecture: Shared Nothing Vs Shared Storage

Apache Kafka® uses local disk storage and constructs a highly reliable storage system through software-level high availability replica replication logic (ISR mechanism). This provides a "limitless" stream storage abstraction to the business logic side. All Kafka® data is stored on local disk media according to specific logic, a method generally referred to as a Shared Nothing architecture.

AutoMQ differs from Apache Kafka® by adopting a compute-storage separation approach. Instead of using local disks, it uses shared object storage services to store data. AutoMQ abstracts an S3Stream storage repository (software library) to replace Apache Kafka®'s local log storage. This transparent use of object storage to store Kafka® data while ensuring the upper layer Apache Kafka® functionalities and semantics remain unchanged is known as a Shared Storage architecture.

<img alt="Architecture comparison: Apache Kafka Shared Nothing architecture vs AutoMQ Shared Storage architecture" />

A comparison of the two different architectures of Apache Kafka® and AutoMQ is as follows:

| **Apache Kafka** <br />Utilizes Shared Nothing architecture<br />                                         | **AutoMQ** <br />Utilizes Shared Storage architecture<br />                                                                             |
| --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Data is stored on local disks, requiring cross-node multiple replica replication<br />                    | Data is stored in S3 shared storage (high reliability with three replicas), eliminating the need for multiple replica replication<br /> |
| Data is isolated across nodes, and data access is bound to specific nodes. <br />                         | Data is shared across nodes, allowing cross-node access. <br />                                                                         |
| When adding nodes for horizontal scaling or replacing faulty nodes, shard data must be reassigned. <br /> | When adding nodes or replacing faulty nodes, data reassignment is not needed for quick switching. <br />                                |

<Info>
  Since version 3.6, Apache Kafka has been planning tiered storage capabilities (not yet production-ready), which supports the offloading of historical data to object storage services. This architecture shares certain similarities and differences with AutoMQ, which fully relies on object storage to construct its storage layer. For a detailed comparison, see [Difference with Tiered Storage▸](/automq/what-is-automq/difference-with-tiered-storage).
</Info>

### Flexibility: Second-level Partition Reassignment vs. Hour-level Partition Reassignment

Partition reassignment is a frequent and unavoidable issue in Apache Kafka production environments, required in scenarios like **partial node failures, cluster scaling, and partial hotspot handling**.

Apache Kafka uses a Shared Nothing architecture, where each partition's data is stored entirely on specific storage nodes. When partition reassignment is involved, the full data of the partition must be transferred to the new target node before service can be provided. This process can be time-consuming and unpredictable.

<Info>
  **Example:**

  For a Kafka partition with a write throughput of 100MiB/s, the data generated in one day is approximately 8.2TiB. If this partition needs to be reassigned, all the data must be moved to another node. Even with a network bandwidth of 1Gbps, it would take hours to complete the reassignment.
</Info>

AutoMQ adopts a storage-compute separation architecture, where the entire data of each partition is stored in S3 object storage. During partition reassignment, only a small amount of metadata needs to be synchronized to complete the switch. AutoMQ can ensure the switch is completed in seconds for partitions with any write throughput scale.

AutoMQ supports second-level partition reassignment, providing AutoMQ with faster and more predictable flexibility advantages over Apache Kafka in scenarios such as cluster scaling and fault recovery.

### Cost: 10x Cost Difference

Referencing the above architectural differences, AutoMQ and Apache Kafka also have significant cost structure differences in terms of computing and storage. AutoMQ does not require cross-node multiple replica replication when writing messages, saving most of the cross-node replication traffic and pressure. Additionally, AutoMQ uses S3 object storage as the storage medium, which is far less costly than mounting EBS block storage to each node in typical Public Cloud environments.

The specific comparison items are as follows:

| **Cost Comparison**<br />             | **Apache Kafka**<br />                                                                                                                 | **AutoMQ**<br />                                                                                                                             |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Storage Unit Price<br />              | <ul><li>Scenario: 1GB data requires 3GB EBS (three replicas)</li><li>Cost: 0.288 USD/month</li></ul>                                   | <ul><li>Scenario: 1GB business data requires 1GB S3</li><li>Cost: 0.023 USD/month</li></ul>                                                  |
| Cross-Node Replication Traffic <br /> | <ul><li>Scenario: Writing 1GB of data requires cross-node replication of 2GB traffic (three replicas)</li><li>Cost: 0.04 USD</li></ul> | <ul><li>Scenario: Writing 1GB of data, directly uploading to S3, no cross-node traffic needed (three replicas)</li><li>Cost: 0 USD</li></ul> |

<Info>
  The storage unit prices above are based on a comparison between AWS S3 US East EBS GP3 instances and S3 Standard storage. For detailed information, refer to [AWS S3 Pricing](https://aws.amazon.com/cn/s3/pricing/?did=ap_card\&trk=ap_card).

  The cross-node replication traffic costs above are based on the cost of data transfer between AWS AZs.

  For a detailed cost comparison between AutoMQ and Apache Kafka, refer to [AutoMQ vs. Apache Kafka Benchmarks and Cost▸](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost).
</Info>

### Capacity: Reserved Vs Pay-as-you-go

Capacity planning is another challenge when Kafka is applied on a large scale in production environments. Due to the architectural differences and storage media differences between AutoMQ and Apache Kafka, there are differences in capacity planning:

| **Apache Kafka** <br /> Uses local disks, integrated storage and compute <br /> | **AutoMQ** <br /> Uses S3 object storage, separated storage and compute <br />   |
| ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Disk space must be reserved in advance<br />                                    | Storage space is on-demand, pay-as-you-go<br />                                  |
| Limited storage on a single node, poor storage scalability<br />                | S3 object storage space is nearly unlimited, with good storage scalability<br /> |

## 100% Compatible with Apache Kafka®

As a redesigned, next-generation Kafka release, AutoMQ provides a cost-effective and elastic alternative while ensuring 100% compatibility with Apache Kafka®. Applications based on Apache Kafka can migrate to AutoMQ without any modifications or adaptations.

Referencing the architectural comparison above, AutoMQ abstracts the S3Stream storage repository to replace Apache Kafka's local log storage at the storage layer. The storage layer exposes the same Partition abstraction to the upper layers, allowing modules such as KRaft metadata management, Coordinator, ReplicaManager, and KafkaApis to reuse the original code logic. The storage layer reuses Apache Kafka's logic. AutoMQ supports 100% compatibility with Apache Kafka protocols and semantics and can continuously follow the latest features and bug fixes of Apache Kafka.

For compatibility details between AutoMQ and Apache Kafka, see [Compatibility with Apache Kafka▸](/automq/what-is-automq/compatibility-with-apache-kafka).


# Difference with Tiered Storage
Source: https://docs.automq.com/automq/what-is-automq/difference-with-tiered-storage

AutoMQ offers seamless cloud-native scalability, cost-efficient storage, and Kafka compatibility by leveraging object storage and treating cloud storage as a service, ensuring stateless and rapid partition reassignments.

AutoMQ uses object storage as its primary storage service, while Apache Kafka® introduced tiered storage with [KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage) starting from version 3.6.0, leveraging object storage to offload historical data.

The [Overview▸](/automq/architecture/s3stream-shared-streaming-storage/overview) of AutoMQ consists of WAL (Write-Ahead Logging) storage and data main memory, whereas Kafka's tiered storage includes primary EBS storage and secondary object storage. Developers often assume AutoMQ's WAL storage is similar to Kafka's primary EBS storage, but they fundamentally differ. This article will highlight the advantages and differences of AutoMQ in comparison to Apache Kafka's tiered storage.

## Architectural Differences

According to the design outlined in [KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage), Apache Kafka's tiered storage version adopts a two-tier storage approach, relying on both local disk and object storage. Message data is initially written to the local disk and then asynchronously uploaded to object storage based on a cooling-off strategy. Since local disks are susceptible to failure, each message needs to be replicated across multiple disks on different nodes via the ISR mechanism to ensure durability.

Currently, when deploying the tiered storage version in a Public Cloud environment, Apache Kafka's architecture remains unchanged, still using EBS as a replacement for local disks, requiring messages to be replicated across multiple EBS instances. In summary, **Apache Kafka still treats EBS as a standard block storage device**, with no fundamental difference from a physical hard drive in a local data center.

AutoMQ employs object storage as its primary storage method, without the concept of storage tiers. However, to optimize storage efficiency, such as reducing latency for writing to object storage and improving write efficiency for large partitions, AutoMQ introduces a WAL storage mechanism. The architectural comparison is as follows:

<img alt="Storage architecture comparison: AutoMQ WAL storage and object storage vs Apache Kafka Tiered Storage with EBS primary and object storage secondary" />

Since WAL storage can use EBS as storage media, developers might think it has similarities with Kafka's primary storage. However, WAL storage fundamentally differs from Kafka in terms of design philosophy and implementation, including storage responsibilities, storage efficiency, storage space, storage media, durability, and multi-AZ disaster recovery design. Please refer to the table below for detailed differences:

| -<br />                          | WAL Storage<br />                                                                                                                                                                          | Kafka Tier 1 Storage<br />                                                                                                                                              |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Responsibilities<br />           | Similar to a database Binlog, used for quick data persistence writes and data recovery during failures.<br />                                                                              | Core data storage, providing read, write, and replay functionalities.<br />                                                                                             |
| Storage Efficiency<br />         | Centralized storage, mixing all partitions' data into one WAL file or object, offering high write efficiency and low IOPS consumption.<br />                                               | Decentralized storage, where the system uses independent file lists to store data of each partition, resulting in low write efficiency and high IOPS consumption.<br /> |
| Storage Space<br />              | Occupies less space, around 10GiB, with deterministic storage needs.<br />                                                                                                                 | Occupies large and uncertain space, requiring capacity assessment, with single nodes typically needing several hundred GiB.<br />                                       |
| Storage Medium<br />             | Can choose block storage, object storage, or file storage provided by cloud providers based on latency level and durability requirements.<br />                                            | Usually recommended to use local hard drives or block storage services provided by cloud vendors.<br />                                                                 |
| Durability Guarantee<br />       | Utilizing multiple replicas or EC (Erasure Coding) technologies in cloud storage services, data durability can generally reach reliability levels between 99.999% and 99.999999999%.<br /> | Apache Kafka's ISR replica mechanism provides reliability but does not guarantee durability metrics.<br />                                                              |
| Multi-AZ Disaster Recovery<br /> | Cloud providers offer regional EBS, object storage, and file storage (such as AWS EFS and FSx) with multi-AZ data durability, and they waive Inter-Zone replication traffic fees.<br />    | Using ISR to implement Inter-Zone data replication will incur Inter-Zone traffic replication fees.<br />                                                                |

## Cost Advantage

In Apache Kafka's tiered storage architecture, the first tier of EBS storage is still used as the primary storage for read and write operations. Each Kafka partition must retain at least the latest active segment on the first tier storage. This leads to the following phenomenon:

* EBS space is uncertain and directly related to the number of partitions in the cluster.

* Reserving a large EBS space in the production environment is necessary to reduce risks.

* EBS reservation costs are high, and the cost reduction potential through tiered storage is limited.

<Info>
  **Example:**

  Taking the default configuration of Apache Kafka® as an example, with each segment size set to 1GB, if the number of active partitions is 1000, it still requires reserving 1TB of EBS.
</Info>

In AutoMQ's architecture, object storage is used as the primary storage. WAL storage provides a lightweight buffer for fault recovery, with multiple backend options available. For example, with EBS WAL, each AutoMQ Broker node only needs a 2GB EBS volume and can guarantee the temporary storage of approximately 500MB of data (the aforementioned space sizes are customizable). With S3 WAL (the default for AutoMQ Open Source), no additional block storage is needed at all.

This design ensures that AutoMQ's WAL space consumption is **predictable**. When using EBS WAL, the additional storage cost remains low because the required EBS volume is small. When using S3 WAL (the default for AutoMQ Open Source), no additional block storage cost is incurred at all.

## Operations Advantage

Due to the non-fixed primary storage space in Apache Kafka's multi-tiered storage architecture, the data left on EBS for each partition is also non-fixed. Therefore, during operations like elastic scaling and fault reassignment, the time required is also uncertain, making quick scaling unachievable.

While AutoMQ's buffer only contains up to 500MB of data that needs to be uploaded to object storage, the upload can be completed within seconds, thereby supporting second-level partition reassignment.

<Info>
  In the case of Confluent, an expansion operation on a high-traffic cluster takes 43 hours in a non-tiered storage architecture and still requires 1.4 hours in a tiered storage architecture.
</Info>

## Summary

Compared to Apache Kafka's tiered storage solution, AutoMQ represents a qualitative leap driven by quantitative changes. Through architectural optimization, AutoMQ achieves a "stateless" state, allowing for arbitrary scaling and second-level partition reassignments. In contrast, Apache Kafka's tiered storage architecture remains an optimized yet stateful solution, making it challenging to achieve lightweight scaling and partition reassignment.


# Licensing and Enterprise Features
Source: https://docs.automq.com/automq/what-is-automq/licensing-and-enterprise-features

AutoMQ offers cloud-native scalability and cost-efficient Kafka compatibility, reducing deployment hassles with its fully managed cloud services and open-source options.

Users can follow the deployment methods listed below to install and use AutoMQ:

* **Install AutoMQ Open Source:** Deploy and operate AutoMQ Open Source within your private network environment or data center.

* **AutoMQ Cloud Service (Recommended):** Simplify the process by directly enabling the AutoMQ Cloud service in a public cloud environment, thereby avoiding installation and maintenance hassles. AutoMQ Cloud is a fully managed, maintenance-free cloud service provided by the AutoMQ team. For more details, please refer to the [reference documentation](https://docs.automq.com/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws).

<Tip>
  AutoMQ Cloud provides developers with a 30-day free trial credit worth \$1,400, without the need for credit card registration or payment information.
</Tip>

## AutoMQ Open Source

AutoMQ Open Source is covered under the [Apache License 2.0](https://github.com/AutoMQ/automq/blob/main/LICENSE), and the source code for all related functionalities is available for review in the [GitHub project](https://github.com/AutoMQ/automq). AutoMQ Open Source is fully production-ready and widely deployed across numerous industry-leading enterprises; however, significant challenges remain with self-hosting distributed storage software. For best practices in the open-source community regarding deployment, operations, monitoring, and performance tuning, please reach out to [Community Support](https://www.automq.com/contact).

## Product Comparison

Below is a comparison between AutoMQ Open Source and AutoMQ BYOC:

| **Product Name**<br />                | **AutoMQ Open Source**<br />                                                                                                                                                                                                                                                               | **AutoMQ BYOC**<br />                                                                                                                                                                                                                                                                                         |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Service Model**<br />               | Open-source software (Apache License 2)<br />                                                                                                                                                                                                                                              | Fully managed service, AutoMQ BYOC deployment \[1]<br />                                                                                                                                                                                                                                                      |
| **Deployment Environment**<br />      | Supported cloud providers:<br /><ul><li>AWS</li><li>Other cloud providers (compatible with S3 API)</li></ul><br />Private data centers require the following storage software:<br /><ul><li>MinIO</li><li>Ceph</li><li>CubeFS</li><li>Other software compatible with S3 protocol</li></ul> | Supported mainstream cloud providers:<br /><ul><li>AWS</li><li>Azure</li><li>GCP</li><li>OCI</li><li>Other cloud providers supporting object storage</li></ul>                                                                                                                                                |
| **WAL Storage Engine**<br />          | S3 WAL<br />Produce RT: P99 600+ milliseconds<br />                                                                                                                                                                                                                                        | Low-latency storage WAL (EBS, FSx)<br />Produce RT: P99 \< 10 milliseconds<br />                                                                                                                                                                                                                              |
| **Self-Balancing**<br />              | Basic scheduling strategy, balancing network throughput only<br />                                                                                                                                                                                                                         | Comprehensive multi-metric scheduling strategy, balancing the following metrics:<br /><ul><li>Network throughput</li><li>Request QPS</li><li>Data cold read accumulation</li><li>Slow node isolation</li></ul>                                                                                                |
| **Cluster Migration**<br />           | Open-source MirrorMaker2 has the following issues:<br /><ul><li>Message replication offset inconsistency</li><li>Cluster switching requires downtime</li></ul>                                                                                                                             | Kafka Linking, a non-downtime migration tool, supports:<br /><ul><li>Offset-consistent ByteToByte message synchronization</li><li>Non-downtime rolling cluster switch</li></ul>                                                                                                                               |
| **Enterprise Product Features**<br /> | None<br />                                                                                                                                                                                                                                                                                 | <ul><li>Out-of-the-box UI for managing cluster, Topic, and Group resources</li><li>Metrics integration: Prometheus, CloudWatch, etc.</li><li>Operation audit</li><li>RBAC & SSO</li><li>Terraform</li></ul>                                                                                                   |
| **Disaster Recovery**<br />           | <ul><li>Self-healing server failures</li></ul>                                                                                                                                                                                                                                             | <ul><li>Self-healing server failures</li><li>Object storage redundancy</li><li>Multi-cloud redundancy</li></ul>                                                                                                                                                                                               |
| **Resource Ownership**<br />          | IaaS resources owned by users<br />                                                                                                                                                                                                                                                        | Under BYOC model, IaaS resources belong to the customer<br />                                                                                                                                                                                                                                                 |
| **Data Sovereignty**<br />            | 100% private, data stays within the user's private network<br />                                                                                                                                                                                                                           | Under BYOC model, data remains within user's private network VPC<br />                                                                                                                                                                                                                                        |
| **Operation Method**<br />            | Self-deployment and operation<br />                                                                                                                                                                                                                                                        | One-click activation through cloud marketplace, official website, etc., fully managed services covering the following scenarios:<br /><ul><li>Deployment</li><li>Version upgrades (new features and bug fixes)</li><li>Parameter tuning</li><li>System alerts\[2]</li><li>Online fault handling\[3]</li></ul> |
| **Cost Structure**<br />              | <ul><li>Machine resource costs (self-provided)</li><li>Operations and development manpower investment</li><li>**Commercial paid technical consultation (optional)**</li></ul>                                                                                                              | <ul><li>Subscription fees</li><li>Cloud resource costs (only in BYOC mode), direct payment to cloud providers</li></ul>                                                                                                                                                                                       |
| **Technical Support**<br />           | <ul><li>None</li><li>Community technical support</li></ul>                                                                                                                                                                                                                                 | <ul><li>Expert technical support</li><li>Online emergency response support</li></ul>                                                                                                                                                                                                                          |

* \[1] involves deploying both the control plane and the data plane of AutoMQ BYOC under the customer's primary account VPC.

* \[2] During the operation of the AutoMQ Server, all generated logs, metrics, and diagnostic data will be stored in a separate object storage bucket. Customers need to grant cross-account permissions for this bucket to the AutoMQ team's main account. The AutoMQ operations platform will analyze the logs and metrics of customer clusters in real-time to ensure stable cluster operation, supported by AutoMQ's professional R\&D team.

* \[3] AutoMQ Server is outfitted with automatic fault recovery capabilities, offering real-time monitoring of various cluster performance metrics and automatically isolating any nodes that show anomalies. This ensures fault recovery occurs before any failure impacts the application, usually completing this process within minutes. Following this, engineers from the AutoMQ team carry out a backend analysis to determine the fault's causes.

> If you're interested in evaluating our cloud services or need paid technical support, [please fill out the form, and our product experts will get in touch with you promptly.](https://www.automq.com/contact)


# Overview
Source: https://docs.automq.com/automq/what-is-automq/overview

AutoMQ is a cloud-native, fully Kafka-compatible streaming platform built on S3. It reduces storage costs by up to 90% and delivers seconds-level elasticity.

# What Is AutoMQ

AutoMQ is a cloud-native streaming platform that is fully compatible with Apache Kafka. Built on object storage (S3), AutoMQ delivers up to **10x lower TCO** ([see benchmark](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)) and **seconds-level elasticity** compared to self-managed Kafka.

## Why AutoMQ: The Cloud Changed Everything

Apache Kafka was designed over a decade ago for on-premise data centers, where disks were physically attached to servers and hardware failures were common. To handle this reality, Kafka adopted a Shared Nothing architecture: each broker manages its own local storage, and data is replicated across multiple nodes for durability and availability. This made perfect sense for the infrastructure of that era.

But the cloud fundamentally changed the rules.

In cloud environments, compute and storage are decoupled by design. Services like AWS S3 deliver 99.999999999% durability with production-grade SLAs. Storage is no longer a liability—it's a managed, virtually infinite resource. Yet when Kafka is lifted-and-shifted to the cloud, it still treats storage as unreliable local disks, replicating data multiple times across brokers. The result: redundant replication on top of already-durable cloud storage, leading to unnecessary compute, storage, and network costs.

Kafka remains the de facto standard for data streaming—the backbone of modern data infrastructure. A technology this critical deserves an architecture built for the cloud, not just deployed on it. That's why we created AutoMQ.

## Design Philosophy

### Rewrite Storage, Preserve Compute

AutoMQ is not a ground-up rewrite of Kafka. After a decade of development, Kafka has a battle-tested compute layer with sophisticated APIs, complex protocol handling, and proven production reliability. Rewriting all of this would be both impractical and unnecessary.

The real problem is storage. Kafka's Shared Nothing storage architecture—designed for unreliable local disks—is the root cause of its cloud inefficiency. Our strategy is surgical: replace the storage layer while keeping the compute layer intact.

This approach delivers:

* **100% Kafka API compatibility**—every client, every protocol version works unchanged
* **100% ecosystem compatibility**—Kafka Connect, Kafka Streams, Schema Registry, and MirrorMaker work out of the box
* **Migration without application changes**—existing workloads move to AutoMQ without application changes

### S3-First, Diskless Architecture

We chose S3 as AutoMQ's storage foundation. S3 is the largest-scale storage service in the cloud, and that scale is precisely why it works: massive scale enables elasticity, pay-per-use pricing, and unmatched cost efficiency.

S3 by the numbers:

* **500+ trillion objects** stored globally
* **>1 PB/s** peak traffic capacity
* **1/10 the cost** of block storage
* **Universal availability**—every region, every major cloud, accessible via simple HTTP

S3 isn't just an AWS service—it's a cloud standard. Nearly every provider offers S3-compatible storage, making AutoMQ portable across clouds.

By building on S3, AutoMQ brokers become completely diskless. No local storage means no state, and stateless brokers can be replaced instantly, scheduled anywhere, and scaled without data migration.

### Built for Cloud Economics

True cloud-native means designing for how the cloud actually works—not just running traditional software on cloud VMs.

AutoMQ is architected around cloud primitives:

* **Cloud storage for durability**—leverage S3's eleven 9s instead of application-level replication
* **Cloud APIs for orchestration**—automated scaling, recovery, and resource management
* **Cloud pricing models**—Spot Instances, pay-per-use, no over-provisioning

In on-premise environments, capacity changes require procurement cycles. In the cloud, resources arrive via API in seconds. AutoMQ embraces this reality: scale out instantly when traffic spikes, scale in when it drops, and never pay for idle capacity.

## Key Benefits

### Simplified Operations

Running self-managed Kafka requires significant operational investment: disk provisioning, capacity planning, manual partition rebalancing, and around-the-clock monitoring for disk failures. These tasks create on-call burden and slow down your team.

AutoMQ's stateless architecture eliminates these operational challenges:

* **No disk management**—brokers store no persistent data locally, removing the need for disk provisioning, RAID configuration, and capacity planning
* **Automatic failover**—when a broker fails, a replacement takes over in seconds without data recovery or manual intervention
* **Self-balancing clusters**—partitions automatically redistribute as brokers join or leave the cluster

### Scale in Seconds, Not Hours

Scaling self-managed Kafka is notoriously difficult. Adding brokers triggers partition reassignment, which copies terabytes of data across the network—a process that can take hours or days. Worse, this replication traffic competes with production workloads, potentially degrading performance exactly when you need more capacity.

Because all data resides in S3, AutoMQ scaling requires no data migration:

* **Seconds-level partition reassignment**—what takes hours in traditional Kafka completes in seconds
* **On-demand capacity**—add brokers during traffic spikes, remove them when demand drops
* **Native auto-scaling integration**—connect to cloud auto-scaling groups for fully automated capacity management

This matters for business continuity. When traffic spikes unexpectedly, you can't afford to wait hours for rebalancing to complete. With AutoMQ, capacity scales as fast as your business needs it—protecting revenue and customer experience during peak demand.

### Lower Costs

AutoMQ reduces total cost of ownership by up to 10x lower TCO compared to self-managed Kafka ([see benchmark](/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost)):

* **Storage costs reduced by 90%**—S3 is approximately 1/10 the price of EBS, and AutoMQ eliminates the 3x replication overhead of traditional Kafka
* **Zero cross-AZ traffic fees**—shared storage architecture removes the expensive cross-AZ data transfer that multi-AZ Kafka deployments require
* **Pay for what you use**—scale down during off-peak hours instead of provisioning for peak capacity

## Next Steps

Ready to get started? Here's where to go next:

* [Deploy Multi-Node Test Cluster on Docker](/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker) — Get hands-on with AutoMQ in minutes
* [Technical Architecture](/automq/architecture/overview) — Understand how AutoMQ works under the hood
* [Difference with Apache Kafka](/automq/what-is-automq/difference-with-apache-kafka) — See a detailed comparison


# AutoMQ Documentation
Source: https://docs.automq.com/index



<div>
  <div>
    <div>
      <h1>AutoMQ Documentation</h1>

      <p>
        Comprehensive guides for AutoMQ - the S3-native diskless Kafka® with 10x cost savings and serverless scaling
      </p>
    </div>

    <div>
      <h2>AutoMQ BYOC</h2>

      <div>
        <Card title="Get started" icon="play" href="/automq-cloud/getting-started/install-byoc-environment/aws/install-automq-on-aws">
          <p>
            Install AutoMQ from the AWS Marketplace.
          </p>
        </Card>

        <Card title="Deploy on Kubernetes" icon="https://mintcdn.com/automq/s9vNxeyMGZ1r1DRV/images/kubernetes-blue.svg?fit=max&auto=format&n=s9vNxeyMGZ1r1DRV&q=85&s=2c6361a7796ae2ab3c91f7f433fb2be1" href="/automq-cloud/deploy-automq-on-kubernetes/overview">
          <p>
            Deploy AutoMQ on AWS EKS, Azure AKS, or Google Cloud GKE.
          </p>
        </Card>

        <Card title="Subscriptions & Billings" icon="credit-card" href="/automq-cloud/subscriptions-and-billings/overview">
          <p>
            Understand the subscription and billing model for AutoMQ BYOC.
          </p>
        </Card>

        <Card title="Table Topic" icon="table-2" href="/automq-cloud/table-topic/overview">
          <p>
            Integrate with data lakes like Iceberg for powerful analytics.
          </p>
        </Card>

        <Card title="Zero-Downtime Migration" icon="arrow-right-left" href="/automq-cloud/migrate-to-automq/overview">
          <p>
            Seamlessly migrate from Apache Kafka to AutoMQ with zero downtime using Kafka-Linking.
          </p>
        </Card>

        <Card title="Monitoring & Alerting" icon="chart-line" href="/automq-cloud/monitoring-alert/monitoring-alert-via-prometheus">
          <p>
            Set up comprehensive monitoring and alerting with Prometheus.
          </p>
        </Card>
      </div>
    </div>

    <div>
      <h2>AutoMQ Open Source</h2>

      <div>
        <Card title="Quick Deploy on Docker" icon="rocket" href="/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker">
          <p>
            Deploy a multi-node test cluster in minutes using Docker.
          </p>
        </Card>

        <Card title="Deploy on Kubernetes" icon="https://mintcdn.com/automq/s9vNxeyMGZ1r1DRV/images/kubernetes-blue.svg?fit=max&auto=format&n=s9vNxeyMGZ1r1DRV&q=85&s=2c6361a7796ae2ab3c91f7f433fb2be1" href="/automq/deployment/deploy-multi-nodes-cluster-on-kubernetes">
          <p>
            Deploy and manage AutoMQ clusters on Kubernetes with Helm.
          </p>
        </Card>

        <Card title="Benchmark Report" icon="clipboard-minus" href="/automq/benchmarks/automq-vs-apache-kafka-benchmarks-and-cost">
          <p>
            Compare AutoMQ vs. Apache Kafka performance and see the 10x cost savings.
          </p>
        </Card>

        <Card title="vs. Apache Kafka" icon="https://mintcdn.com/automq/s9vNxeyMGZ1r1DRV/images/kafka.svg?fit=max&auto=format&n=s9vNxeyMGZ1r1DRV&q=85&s=40715a9d81b175d3d8c5d9909fd2d4ca" href="/automq/what-is-automq/difference-with-apache-kafka">
          <p>
            Understand the key architectural differences and advantages over Kafka.
          </p>
        </Card>

        <Card title="Architecture" icon="landmark" href="/automq/architecture/overview">
          <p>
            Dive into the S3-native, shared storage architecture of AutoMQ.
          </p>
        </Card>

        <Card title="Server Configuration" icon="bolt" href="/automq/configuration/broker-and-controller-configuration">
          <p>
            Configure brokers, controllers, and object storage for optimal performance.
          </p>
        </Card>
      </div>
    </div>

    <div>
      <h3>Ready to get started?</h3>

      <p>
        Choose your deployment option and start building with AutoMQ today. From local development to enterprise-scale production.
      </p>

      <div>
        <a href="https://console.automq.cloud/">
          Start for free

          <svg>
            <path />
          </svg>
        </a>

        <a href="https://github.com/AutoMQ/automq">
          Explore Github

          <svg>
            <path />
          </svg>
        </a>
      </div>
    </div>
  </div>
</div>


# AWS BYOC Console Upgrade Guide for Version 8.x
Source: https://docs.automq.com/automq-cloud/appendix/byoc-console-upgrade-to-8.x-guide



## Background

The AutoMQ Console (versions prior to 8.0) was distributed through cloud provider marketplaces and installed via AMI images. Starting from version 8.0, the AutoMQ Console is distributed and installed via Docker images. New users and new environments are recommended to follow the [Install BYOC Console](/automq-cloud/getting-started/install-byoc-environment/aws/prepare-vpc) documentation for installation. This document describes how to smoothly upgrade existing BYOC environments from earlier versions to version 8.x.

## Constraints

* **Cloud Provider:**
  Currently, only AWS environments support the upgrade to version 8.x. Support for other cloud providers is in progress. If you have requirements for other clouds, please contact us by referring to the "Contact Us" documentation.

* **No AWS Managed Service for Prometheus:**
  Confirm that your current environment is not using the AWS Managed Service for Prometheus. If it is, please contact AutoMQ technical personnel to discuss an appropriate solution.

## Upgrade Steps

The overall process is divided into the following steps: Register an AutoMQ account, confirm deployment information, obtain the upgrade command, stop the old console, configure IAM permissions, and execute the upgrade.

### 1. Register an Organization and Account

Go to the AutoMQ official website to register an organization and account. The registration URL is: [https://console.automq.cloud/](https://console.automq.cloud/).

### 2. Confirm Deployment Information

Confirm the deployment information of your current BYOC console and send it to AutoMQ technical personnel. This information is used to migrate environment metadata and generate the installation command. The information to be collected includes:

* Environment ID
* Deployment Region
* Deployment Version
* Installation ID
* OpsBucket

It is recommended to log in to the AutoMQ console and navigate to the settings page to view this information. You also need to provide your newly registered AutoMQ Organization ID to the AutoMQ technical personnel.

### 3. Obtain the Upgrade Command

AutoMQ technical personnel will generate the environment's metadata and an upgrade installation command based on the information from Step 2, which will be used for the subsequent upgrade.

### 4. Stop the Old Console

Log in to the EC2 instance where the AutoMQ console is located and stop the current console application. The command to stop the service is as follows:

```bash theme={null}
sudo systemctl stop cmp.service
```

### 5. Modify EC2 Metadata Access Limit (Optional)

The AutoMQ 8.x console is deployed using Docker containers. Programs inside the container need to access the EC2 instance metadata service, and by default, the installation command uses Host network mode.

If you prefer to use Bridge or other network modes, you must set the EC2 instance metadata response hop limit (`HttpPutResponseHopLimit`) to `2`. The procedure is as follows:

1. Log in to the [AWS EC2 Console](https://console.aws.amazon.com/ec2/home).
2. In the instances list, select the EC2 instance hosting the AutoMQ console.
3. Select **Actions** > **Instance Settings** > **Modify instance metadata options**.
4. In the dialog box, change the value of **Metadata response hop limit** from `1` to `2`.
5. Click **Save**.

### 6. Create and Configure IAM Role

Go to the AWS IAM console ([https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/roles](https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/roles)) to create a new IAM Role for the AutoMQ console and grant the necessary cloud resource operation permissions. Starting from version 8.x, the AutoMQ console requires permissions to create IAM Roles for the data plane clusters. This means the console will manage the IAM permissions and roles required for each data plane cluster to operate.

The procedure is as follows: Go to the IAM console to create a custom IAM Policy, then create an IAM Role, select EC2 as the service type, and attach the newly created IAM Policy. Next, go to the EC2 console to modify the EC2 instance where the console is located and change its attached IAM Role.

<Tabs>
  <Tab title="Example AWS Policy">
    ```json highlight={36,46} theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "BasicIamCheckPermissions",
          "Effect": "Allow",
          "Action": [
            "iam:GetPolicy",
            "iam:ListRoles",
            "iam:GetInstanceProfile",
            "iam:ListInstanceProfilesForRole",
            "iam:GetRole",
            "iam:GetRolePolicy",
            "iam:ListRolePolicies",
            "iam:ListAttachedRolePolicies",
            "iam:GetPolicyVersion",
            "iam:GetUser",
            "iam:GetUserPolicy",
            "iam:ListUserPolicies",
            "iam:ListAttachedUserPolicies"
          ],
          "Resource": "*"
        },
        {
          "Sid": "OpsBucketManagePermissions",
          "Effect": "Allow",
          "Action": [
            "s3:GetLifecycleConfiguration",
            "s3:PutLifecycleConfiguration",
            "s3:ListBucket",
            "s3:PutBucketTagging",
            "s3:GetBucketPolicy",
            "s3:CreateBucket",
            "s3:PutBucketPolicy"
          ],
          "Resource": "arn:aws:s3:::{Replace with your truly OpsBucket}"
        },
        {
          "Sid": "OpsBucketOperationPermissions",
          "Effect": "Allow",
          "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
          ],
          "Resource": "arn:aws:s3:::{Replace with your truly OpsBucket}/*"
        },
        {
          "Sid": "DataBucketManagePermissions",
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetLifecycleConfiguration",
            "s3:CreateBucket",
            "s3:PutBucketTagging",
            "s3:DeleteBucket",
            "s3:ListBucketMultipartUploads"
          ],
          "Resource": "*"
        },
        {
          "Sid": "DataBucketObjectCleanPermissions",
          "Effect": "Allow",
          "Action": [
            "s3:AbortMultipartUpload",
            "s3:DeleteObject"
          ],
          "Resource": "*"
        },
        {
          "Sid": "AutoScalingServiceRolePermissions",
          "Effect": "Allow",
          "Action": "iam:CreateServiceLinkedRole",
          "Resource": "*",
          "Condition": {
            "StringEquals": {
              "iam:AWSServiceName": "autoscaling.amazonaws.com"
            }
          }
        },
        {
          "Sid": "Ec2InstanceProfilePassPermissions",
          "Effect": "Allow",
          "Action": "iam:PassRole",
          "Resource": "*",
          "Condition": {
            "StringLike": {
              "iam:PassedToService": "ec2.amazonaws.com*"
            }
          }
        },
        {
          "Sid": "Ec2AndAutoscalingReadAndCreatePermissions",
          "Effect": "Allow",
          "Action": [
            "ssm:GetParameters",
            "s3:ListAllMyBuckets",
            "route53:ListHostedZones",
            "route53:ListHostedZonesByName",
            "route53:ListHostedZonesByVpc",
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:CreateAutoScalingGroup",
            "ec2:DescribeAvailabilityZones",
            "ec2:DescribeInstanceTypeOfferings",
            "ec2:DescribeVpcs",
            "ec2:DescribeVpcEndpoints",
            "ec2:DescribeSubnets",
            "ec2:DescribeImages",
            "ec2:DescribeKeyPairs",
            "ec2:CreateKeyPair",
            "ec2:DescribeVolumes",
            "ec2:CreateVolume",
            "ec2:DescribeSecurityGroups",
            "ec2:CreateSecurityGroup",
            "ec2:DescribeSecurityGroupRules",
            "ec2:DescribeLaunchTemplates",
            "ec2:DescribeLaunchTemplateVersions",
            "ec2:CreateLaunchTemplateVersion",
            "ec2:CreateLaunchTemplate",
            "ec2:ModifyLaunchTemplate",
            "ec2:DeleteLaunchTemplate",
            "ec2:DescribeTags",
            "ec2:CreateTags",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceAttribute",
            "ec2:RunInstances"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Ec2AndAutoscalingOperationPermissions",
          "Effect": "Allow",
          "Action": [
            "ec2:AttachVolume",
            "ec2:DetachVolume",
            "ec2:DeleteVolume",
            "ec2:DeleteKeyPair",
            "ec2:TerminateInstances",
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:AuthorizeSecurityGroupEgress",
            "ec2:DeleteSecurityGroup",
            "autoscaling:AttachInstances",
            "autoscaling:DetachInstances",
            "autoscaling:SuspendProcesses",
            "autoscaling:ResumeProcesses",
            "autoscaling:UpdateAutoScalingGroup",
            "autoscaling:DeleteAutoScalingGroup",
            "ec2:StopInstances",
            "ec2:RebootInstances"
          ],
          "Resource": "*",
          "Condition": {
            "StringEquals": {
              "aws:ResourceTag/automqVendor": "automq"
            }
          }
        },
        {
          "Sid": "EksReadPermissions",
          "Effect": "Allow",
          "Action": [
            "s3:ListAllMyBuckets",
            "route53:ListHostedZones",
            "route53:ListHostedZonesByName",
            "route53:ListHostedZonesByVpc",
            "eks:ListClusters",
            "eks:DescribeCluster",
            "eks:ListNodegroups",
            "eks:DescribeNodegroup",
            "eks:ListAccessEntries",
            "eks:DescribeAccessEntry",
            "eks:ListAssociatedAccessPolicies",
            "elasticloadbalancing:DescribeTargetGroups"
          ],
          "Resource": "*"
        },
        {
          "Sid": "ClusterRoleManagementPermissions",
          "Effect": "Allow",
          "Action": [
            "iam:ListOpenIDConnectProviders",
            "iam:GetOpenIdConnectProvider",
            "iam:GetPolicy",
            "iam:CreatePolicy",
            "iam:TagPolicy",
            "iam:CreatePolicyVersion",
            "iam:GetPolicyVersion",
            "iam:AttachRolePolicy",
            "iam:DetachRolePolicy",
            "iam:DeletePolicy",
            "iam:ListRoles",
            "iam:GetRole",
            "iam:CreateRole",
            "iam:TagRole",
            "iam:DeleteRole",
            "iam:ListRolePolicies",
            "iam:ListAttachedRolePolicies",
            "iam:ListPolicyVersions",
            "iam:DeletePolicyVersion",
            "iam:ListInstanceProfilesForRole",
            "iam:GetInstanceProfile",
            "iam:CreateInstanceProfile",
            "iam:AddRoleToInstanceProfile",
            "iam:RemoveRoleFromInstanceProfile",
            "iam:DeleteInstanceProfile"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Route53ManagementPermissions",
          "Effect": "Allow",
          "Action": [
            "route53:GetHostedZone",
            "route53:ListResourceRecordSets",
            "route53:ChangeResourceRecordSets",
            "route53:CreateHostedZone",
            "route53:DeleteHostedZone",
            "route53:ChangeTagsForResource"
          ],
          "Resource": "*"
        },
        {
          "Sid": "NfsReadAndCreatePermissions",
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeRouteTables",
            "ec2:DescribeNetworkInterfaceAttribute",
            "ec2:DescribeNetworkInterfaces",
            "ec2:CreateNetworkInterface",
            "fsx:TagResource",
            "fsx:DescribeVolumes",
            "fsx:CreateVolume",
            "fsx:DescribeStorageVirtualMachines",
            "fsx:CreateStorageVirtualMachine",
            "fsx:DescribeFileSystems",
            "fsx:CreateFileSystem",
            "elasticfilesystem:CreateFileSystem",
            "elasticfilesystem:DescribeFileSystems",
            "elasticfilesystem:CreateMountTarget",
            "elasticfilesystem:DescribeMountTargets",
            "elasticfilesystem:DescribeMountTargetSecurityGroups",
            "elasticfilesystem:TagResource"
          ],
          "Resource": "*"
        },
        {
          "Sid": "NfsOperationPermissions",
          "Effect": "Allow",
          "Action": [
            "ec2:DeleteNetworkInterface",
            "fsx:UpdateVolume",
            "fsx:DeleteVolume",
            "fsx:DeleteStorageVirtualMachine",
            "fsx:UpdateFileSystem",
            "fsx:DeleteFileSystem",
            "elasticfilesystem:UpdateFileSystem",
            "elasticfilesystem:DeleteFileSystem",
            "elasticfilesystem:DeleteMountTarget"
          ],
          "Resource": "*",
          "Condition": {
            "StringEquals": {
              "aws:ResourceTag/automqVendor": "automq"
            }
          }
        },
        {
          "Sid": "Ec2PricingPermissions",
          "Effect": "Allow",
          "Action": [
            "pricing:DescribeServices",
            "pricing:GetAttributeValues",
            "pricing:GetProducts"
          ],
          "Resource": "*"
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### 7. Install Docker and Start the AutoMQ Console

Before executing the upgrade command, please ensure that Docker is installed and running in your environment.

```bash theme={null}
# Install docker
sudo yum install docker -y

# Start docker service
sudo systemctl start docker

# Enable docker to start on boot
sudo systemctl enable docker
```

Copy the upgrade installation command from Step 3 to start the new version of the AutoMQ console. An example of the Docker run command is as follows:

```bash theme={null}
sudo docker run -d -v /home/admin:/root \
  -p 8080:8080 \
  -e CLOUD_PROVIDER=aws \
  -e REGION=ap-northeast-1 \
  -e OPS_BUCKET=automq-ops-xxxx \
  -e CONSOLE_INITIAL_USER=admin \
  -e CONSOLE_INITIAL_PASSWORD=admin \
  -e CLIENT_ID=env-xxxx \
  -e CLIENT_SECRET=xxx \
  xxx/automq/automq_byoc_console:xxxx
```

After observing that the Docker container is running correctly and that port 8080 is accessible, you can access the AutoMQ console service via a web browser.

### 8. Copy the Installation ID and Update the License

Because version 8.x changes the installation media and startup method, the installation ID for the new version will be different. After logging into the new console, a "License invalid" prompt will appear. Copy the installation ID displayed on the console page and contact AutoMQ technical personnel to update the information. You will then be able to access the new version of the console.

<Tip>
  **Note**: During the upgrade process, if the console displays a "License invalid" message, this will not affect the normal operation of the cluster. Contact AutoMQ technical personnel and provide the new installation ID to activate the license.
</Tip>


# Cloud Resource List
Source: https://docs.automq.com/automq-cloud/appendix/cloud-resource-list

Deploy AutoMQ on AWS with cloud-native scalability, full Kafka compatibility, and cost efficiency. AutoMQ creates and manages essential AWS resources for you.

## AWS

By default, creating AutoMQ from the AWS Cloud Marketplace will deploy a cluster with 6 AKUs across three availability zones. AutoMQ is responsible for creating network resources such as VPCs and subnets, and will create and rely on the following essential cloud resources:

| Resource Name<br />        | Specification<br /> | Quantity<br /> | Description<br />                                                                                                                                                                                                                                                                                      |
| -------------------------- | ------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| VPC<br />                  | -<br />             | 1<br />        | The VPC used by AutoMQ control and data planes.<br />                                                                                                                                                                                                                                                  |
| Security Group<br />       | -<br />             | 1<br />        | The security group used by AutoMQ, with 1 new rule created,<br />                                                                                                                                                                                                                                      |
| Subnet<br />               | -<br />             | 4<br />        | The control plane will use 1 subnet, and each of the 3 data plane nodes will use 1 subnet,<br />                                                                                                                                                                                                       |
| IGW<br />                  | -<br />             | 1<br />        | For the control plane to be accessible from the public internet,<br />                                                                                                                                                                                                                                 |
| VPC Internet Gateway<br /> | -<br />             | 1<br />        | For the control plane to access the public internet,<br />                                                                                                                                                                                                                                             |
| Network ACL<br />          | -<br />             | 1<br />        | AutoMQ will create the following ACL rules.\[1]<br />                                                                                                                                                                                                                                                  |
| Route Table<br />          | -<br />             | -<br />        | Rules associated with local and endpoint<br />                                                                                                                                                                                                                                                         |
| Security Group<br />       | -<br />             | -<br />        | AutoMQ will create the following security group rules.\[2]<br />                                                                                                                                                                                                                                       |
| EC2<br />                  | t3.large<br />      | 1<br />        | Deploys the control plane<br />                                                                                                                                                                                                                                                                        |
| EBS<br />                  | GP3,20 GB<br />     | 1<br />        | Used for the control plane system volume, encryption is disabled by default.<br />                                                                                                                                                                                                                     |
| EBS<br />                  | GP3,20 GB<br />     | 3<br />        | Used for the data plane system volume, encryption is disabled by default.<br />                                                                                                                                                                                                                        |
| EBS<br />                  | GP3,20 GB<br />     | 3<br />        | Used for the data plane data volume, encryption is disabled by default.<br />                                                                                                                                                                                                                          |
| EC2<br />                  | r6in.large<br />    | 3<br />        | Provides support for AutoMQ data plane with 6 AKU capacity<br />                                                                                                                                                                                                                                       |
| Elastic IP Address<br />   | -<br />             | 1<br />        | Used by AutoMQ control plane<br />                                                                                                                                                                                                                                                                     |
| IAM Instance Profile<br /> | -<br />             | 1<br />        | Grants permissions to the control plane EC2 for executing data plane creation tasks<br />                                                                                                                                                                                                              |
| IAM Policy<br />           | -<br />             | 1<br />        | An IAM policy will be created for AutoMQ, used for authorization. The policy follows the principle of least privilege and can be referenced from the AWS installation documentation. This policy allows AutoMQ to create necessary resources when users need to create a new data plane cluster.<br /> |
| IAM Role<br />             | -<br />             | 1<br />        | An IAM role will be created for AutoMQ, associated with the IAM policy.<br />                                                                                                                                                                                                                          |
| Route 53 Records<br />     | -<br />             | 3<br />        | AutoMQ provides Bootstrap Server endpoints, with a domain name assigned to each node.<br />                                                                                                                                                                                                            |
| EC2 Endpoint<br />         | -<br />             | 1<br />        | Used for EC2 to access AWS OpenAPI and query metadata.<br />                                                                                                                                                                                                                                           |
| S3 Endpoint<br />          | -<br />             | 1<br />        | Ensures that access to S3 is routed through the internal network.<br />                                                                                                                                                                                                                                |
| AutoScaling Group<br />    | -<br />             | 1<br />        | Created only when using the AWS Cloud Marketplace CloudFormation method.<br />                                                                                                                                                                                                                         |
| Launch Template<br />      | -<br />             | 1<br />        | Created only when using AWS CloudFormation from the AWS Marketplace, and will be dependent on the ASG.<br />                                                                                                                                                                                           |
| S3 Bucket<br />            | -<br />             | 2<br />        | Using AutoMQ requires users to provide 2 S3 Buckets, one for storing primary data and the other for storing logs and monitoring data. By default, encryption options do not need to be enabled.<br />                                                                                                  |
| Route 53<br />             | -<br />             | 1<br />        | Used for binding domain names to data nodes in subsequent steps.<br />                                                                                                                                                                                                                                 |

#### \[1] AutoMQ AWS Network ACL

```hcl theme={null}

// aws_default_network_acl
egress {
    action          = "allow"
    from_port       = 0
    ipv6_cidr_block = "::/0"
    protocol        = "-1"
    rule_no         = 101
    to_port         = 0
}
egress {
    action          = "allow"
    cidr_block      = "0.0.0.0/0"
    from_port       = 0
    protocol        = "-1"
    rule_no         = 100
    to_port         = 0
}
ingress {
    action          = "allow"
    from_port       = 0
    ipv6_cidr_block = "::/0"
    protocol        = "-1"
    rule_no         = 101
    to_port         = 0
}
ingress {
    action          = "allow"
    cidr_block      = "0.0.0.0/0"
    from_port       = 0
    protocol        = "-1"
    rule_no         = 100
    to_port         = 0
}

```

#### \[2] AutoMQ AWS Security Group

```hcl theme={null}

// vpc_endpoint_sg
ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
}

egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
}
  
// automq_byoc_console_sg
ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = [var.automq_byoc_env_console_cidr]
}

egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
}

```

## Additional Information

* All cloud resources created by AutoMQ will be tagged with the default label `automqVendor: automq` to facilitate users in filtering cloud resources related to AutoMQ.

* AutoMQ uses the following supported Linux security image on AWS EC2: amzn2-ami-kernel-5.10-hvm-2.0.20241113.1-arm64-gp2


# Deploy AutoMQ Software Via Helm Chart
Source: https://docs.automq.com/automq-cloud/appendix/deploy-automq-enterprise-via-helm-chart

Deploy AutoMQ Software in a private Kubernetes data center using Helm Charts. Achieve 100% Kafka compatibility and cost efficiency with step-by-step setup instructions.

This document details the process for deploying AutoMQ Software in a private Kubernetes data center using a Helm Chart. To deploy AutoMQ in a public cloud environment, refer to [Overview▸](/automq-cloud/getting-started/overview).

## Prerequisites

Before installing AutoMQ with a Helm Chart, ensure the following prerequisites are satisfied:

1. **Prepare a Kubernetes Environment**: Establish an available Kubernetes cluster in advance, ensuring it meets the conditions below:

   1. **Allocate Resources for AutoMQ**: It is recommended to allocate 4 cores and 16GB of memory for each AutoMQ Pod. Deploying on a dedicated Node is advisable for stable network throughput performance.

   2. **Storage Plugin:** If your Kubernetes is provided by a cloud vendor, it is advisable to install the storage plugin offered by the vendor to manage EBS volume resources effectively.

2. **Prepare Object Storage Buckets:** Each AutoMQ cluster requires two separate object storage buckets: one Ops Bucket for system logs and metrics data, and one Data Bucket for message data. Please refer to the object storage product documentation for guidance on creating them.

3. **Install the Helm Chart Tool:** It is recommended to install version 3.6 or higher. You can [refer to the documentation](https://helm.sh/docs/intro/quickstart/) for detailed instructions.

## Obtain the Software Chart.

The AutoMQ Software Chart image is published and made available to the public through an Azure Container Registry (East US). You can test the pull with the following command.

```bash theme={null}
helm pull oci://automq.azurecr.io/helm/automq-enterprise-chart --version 5.3.4
```

## Install AutoMQ

AutoMQ Software offers two types of WAL storage options: EBSWAL and S3WAL. A comparison of the two storage engines is as follows; it is recommended to choose based on your needs. For detailed principles, please refer to the [Technical Architecture](https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/wal-storage).

* **EBSWAL Mode:** WAL storage uses high-speed EBS volumes to deliver low-latency send performance, supported only in public cloud environments like AWS, GCP, and Azure. When using, you need to assign EBS volumes to AutoMQ's Pods via a StorageClass.

* **S3WAL Mode:** Deployment is relatively simple, as WAL storage writes directly to object storage, offering sub-100 ms send RT performance. It supports all public cloud environments as well as private data centers (as long as they provide S3-compatible object storage). Deployment is relatively straightforward, with no need to allocate EBS volumes.

<Tip>
  The following sections provide a simplified deployment scenario example for various cloud vendor environments. This example scenario deploys in S3WAL mode, uses static credentials to access cloud resources, and supports accessing AutoMQ within a Kubernetes cluster via a Headless service access point or from outside the Kubernetes cluster using a LoadBalancer access point.

  The AutoMQ team also provides a variety of advanced feature examples, including TLS, authentication and authorization, Auto-Scaler, and more. For details, please refer to the [Advanced Feature Examples ▸](https://github.com/AutoMQ/automq-labs/tree/main/software-examples/kubernetes/).
</Tip>

### Step 1: Create Credentials and Perform Authorization.

AutoMQ clusters require access to external services such as object storage and storage volumes. Therefore, before installation, you need to create credentials for AutoMQ and complete the authorization process.

<Tabs>
  <Tab title="AWS">
    If AutoMQ is deployed in the AWS public cloud environment using AWS S3 storage, you must access the IAM product to create an authorization policy. AutoMQ must be granted permission for the following operations to access AWS S3:

    ```yaml theme={null}
    - actions:
        - s3:GetLifecycleConfiguration
        - s3:PutLifecycleConfiguration
        - s3:ListBucket
        - s3:PutObject
        - s3:GetObject
        - s3:AbortMultipartUpload
        - s3:PutObjectTagging
        - s3:DeleteObject

    ```

    If you deploy using the EBSWAL mode, additional authorization for the following policy is required:

    ```yaml theme={null}
    - actions:
        - ec2:DescribeVolumes
        - ec2:DetachVolume
        - ec2:DescribeAvailabilityZones

    ```

    After creating an IAM authorization policy, credentials can be generated using two methods.

    * **Using IAM Subaccount Static AccessKey:** In this approach, attach the authorization policy to the IAM subaccount and utilize the subaccount's static AccessKeyId and AccessKeySecret as credentials to access AutoMQ.

    * **Using IAM Role Dynamic Credentials:** For this approach, create an IAM Role and attach the authorization policy to the Role. Dynamic credentials allow access to AutoMQ through a Pod assuming the EC2 Role in EKS.

    <Tip>
      To clarify illustrations, the following configuration file example employs a static AccessKey as credentials.
    </Tip>
  </Tab>

  <Tab title="Azure">
    Deploying AutoMQ in the Azure public cloud environment requires using Azure Blob Storage. Authorization credentials must be configured within the Storage Account's access control (IAM). Users can opt for either of the following permission entities to assign the system role to Blob: `Storage Blob Data Owner`.

    * **Using Service Principal Static Secret**: In this setup, the AccessKey corresponds to the App's clientId, and the SecretKey corresponds to the Secret value along with the tenantId. Together, these serve as static credentials for accessing AutoMQ.

    * **Using Managed Identity Dynamic Credentials**: With this approach, in addition to having the ServiceAccount assigned the necessary roles, it is vital to associate it with the corresponding VMSS (virtual machine scale set) for AutoMQ. This way, you can access AutoMQ on AKS using the clientId of this dynamic credential.
  </Tab>

  <Tab title="OCI">
    In an OCI environment, you need to generate a **Customer Secret Key** for a specific OCI user to serve as static credentials. This user must be granted appropriate permissions to access Object Storage.

    First, create the following IAM authorization policies for the user or user group managing AutoMQ:

    ```text theme={null}
    Allow group <your-group-name> to manage objects in compartment <your-compartment> where any {
      target.bucket.name = '<your-data-bucket>',
      target.bucket.name = '<your-ops-bucket>'
    }

    Allow group <your-group-name> to use buckets in compartment <your-compartment> where any {
      target.bucket.name = '<your-data-bucket>',
      target.bucket.name = '<your-ops-bucket>'
    }
    ```

    * The `manage objects` permission set includes operations like creating, reading, deleting, and listing objects.
    * The `use buckets` permission set includes permissions to manage bucket lifecycle policies.

    If you plan to use **EBSWAL** mode, OKE worker nodes require permissions to manage Block Volumes. Ensure you create a Dynamic Group for your node pool and grant it the following permissions:

    ```text theme={null}
    Allow dynamic-group <your-node-dynamic-group> to manage block-volumes in compartment <your-compartment>
    Allow dynamic-group <your-node-dynamic-group> to manage volume-attachments in compartment <your-compartment>
    Allow dynamic-group <your-node-dynamic-group> to use subnets in compartment <your-compartment>
    Allow dynamic-group <your-node-dynamic-group> to inspect instances in compartment <your-compartment>
    ```

    After creating the credentials, record the **Access Key** and **Secret Key**, as they will be used as static credentials in the subsequent Helm Chart configuration.
  </Tab>
</Tabs>

### Step 2: Create Storage Class

Before installing AutoMQ, you must declare a Storage Class in the Kubernetes cluster for allocating storage volumes. These storage volumes serve several purposes:

* **Storing AutoMQ Controller Metadata:** In the AutoMQ cluster, the Controller Pod responsible for metadata management must mount the storage volumes to store KRaft metadata.

* **EBSWAL Mode Storage for WAL Data (Optional):** If you plan to deploy using the EBSWAL mode, each Broker Pod will also require a mounted data volume for writing WAL data.

<Tip>
  Please specify the Storage Class based on the Kubernetes storage plugin from your cloud provider or private data center, then keep a record of the Storage Class name for later parameter configuration.
</Tip>

<Tabs>
  <Tab title="AWS">
    ```yaml theme={null}
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: automq-disk-eks-gp3
    provisioner: ebs.csi.aws.com
    volumeBindingMode: WaitForFirstConsumer
    parameters:
      type: gp3  # EBS Volume Type
    allowVolumeExpansion: true

    ```
  </Tab>

  <Tab title="Azure">
    ```yaml theme={null}
    # Example StorageClass Definition (Azure Premium SSD Example)
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: automq-disk-storage-class
    provisioner: disk.csi.azure.com
    parameters:
      skuName: PremiumV2_LRS
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true

    ```
  </Tab>

  <Tab title="GCP">
    ```yaml theme={null}
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: automq-disk-gcp-pd-balanced
    provisioner: pd.csi.storage.gke.io
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
    parameters:
      type: pd-balanced  # High Performance Cloud Disk.

    ```
  </Tab>

  <Tab title="OCI">
    ```yaml theme={null}
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: automq-block-volume
    provisioner: blockvolume.csi.oraclecloud.com
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
    parameters:
      # vpusPerGB determines the IOPS and throughput performance of the volume
      # 10: Provides basic performance
      # 20: Provides higher performance, recommended for production environments
      vpusPerGB: "20"
      # attachmentType determines the type of volume attachment
      # iscsi: iSCSI attachment, general purpose
      # paravirtualized: Paravirtualized attachment, offers higher performance
      attachmentType: "paravirtualized"
    ```
  </Tab>
</Tabs>

### Step 3: Initialize the Configuration File

The configuration information for the AutoMQ Software Chart is composed of multiple parts, allowing for user customization via the values.yaml file.

<Tip>
  **Required: Set a Unique Instance ID**

  A critical parameter you must define is `global.automqInstanceId`. This serves as a **required** unique identifier for your AutoMQ cluster.

  Why is it important? This ID ensures that resources are properly isolated, preventing data corruption or conflicts. Each cluster must have its own distinct `automqInstanceId`.
</Tip>

First, create an empty file named `automq-values.yaml`. You can copy the example configuration below and edit it.

```yaml theme={null}
global:
  cloudProvider:
    name: "Replace With Your True Cloud Provider Name"
    credentials: "Replace With Your True Your Credentials"
  config: |
    s3.ops.buckets=Replace With Your True Ops Bucket URL
    s3.data.buckets=Replace With Your True Data Bucket URL
    s3.wal.path=Replace With Your True WAL PATH

controller:
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"
  persistence:
    metadata:
      storageClass: "Replace With Your True StroageClass"
    wal:
      enabled: false

  annotations:

  env:
    - name: "KAFKA_JVM_PERFORMANCE_OPTS"
      value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5"
    - name: "KAFKA_OPTS"
      value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
    - name: "KAFKA_HEAP_OPTS"
      value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

broker:
  replicas: 0
  resources:
    requests:
      cpu: "3000m"
      memory: "12Gi"
    limits:
      cpu: "4000m"
      memory: "16Gi"

  persistence:
    wal:
      enabled: false

  annotations:

  env:
    - name: "KAFKA_JVM_PERFORMANCE_OPTS"
      value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5"
    - name: "KAFKA_OPTS"
      value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
    - name: "KAFKA_HEAP_OPTS"
      value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      annotations:
        <Add Cloud Provider Specific Annotations Here>

      extraPorts:
        - name: "tcp-9092"
          port: 9092
          protocol: "TCP"
          targetPort: 9092
```

In the configuration file generated in the previous step, certain parameters must be updated according to your actual conditions.

#### Modify Common Parameters.

**global.cloudProvider.name**

This parameter specifies the deployment cloud environment. Please insert the enumerated value according to the name of the cloud provider. If it is a private data center, you'll also need to fill it in with the enumerated value.

| **Deployment Environment**<br /> | **Parameter Enumerated Value**<br /> |
| -------------------------------- | ------------------------------------ |
| AWS<br />                        | aws<br />                            |
| Google Cloud<br />               | gcp<br />                            |
| Azure<br />                      | azure<br />                          |
| OCI<br />                        | noop<br />                           |
| Alibaba Cloud<br />              | aliyun<br />                         |

**global.cloudProvider.credentials**

This parameter details the public credentials used by the AutoMQ cluster to access cloud resources. The current example utilizes static credentials of the AccessKey type. To use the IAM Role method, please refer to the advanced parameter documentation for guidance on modifications.

```yaml theme={null}
global:
  cloudProvider:
    credentials: static://?accessKey=<your-accesskey>&secretKey=<your-secretkey>


```

**KAFKA\_S3\_ACCESS\_KEY and KAFKA\_S3\_SECRET\_KEY Environment Variables**

The example configuration file employs static credentials of the AccessKey type. Therefore, in addition to the global.cloudProvider.credentials parameter, you need to update the Controller and Broker environment variables with the correct credentials.

Refer to the credentials created in Step 1 to update the credentials in the example:

```yaml theme={null}
controller:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

broker:
  env:
    - name: "KAFKA_S3_ACCESS_KEY"
      value: "Replace With Your True ACCESS_KEY"
    - name: "KAFKA_S3_SECRET_KEY"
      value: "Replace With Your True SECRET_KEY"

```

**global.config**

This parameter specifies the configuration for accessing object storage using S3URL, which includes three components: s3.ops.buckets, s3.data.buckets, and s3.wal.path.

<Tip>
  Below is an example configuration using the S3WAL mode with static credentials. If you need to use EBSWAL mode, please refer to the advanced configuration section for modification instructions.
</Tip>

<Tabs>
  <Tab title="AWS">
    Fill in the Ops Bucket and Data Bucket created in the prerequisites according to your actual scenario.

    ```yaml theme={null}
    config: |
      s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static
      s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static
      s3.wal.path=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://s3.xxxx.amazonaws.com&authType=static

    ```
  </Tab>

  <Tab title="Azure">
    Fill in the Ops Bucket, Data Bucket, region, and Endpoint parameters according to your actual scenario.

    ```yaml theme={null}
    config: |
      s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static
      s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static
      s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=static

    ```
  </Tab>

  <Tab title="Google Cloud">
    AccessKey-type static credentials are not supported on GCP, requiring a different format. You need to fill in the Ops Bucket, Data Bucket, region, Endpoint, and GCP ServiceAccount parameters.

    ```yaml theme={null}
    config: |
      s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>
      s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>
      s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx_storage-account.blob.core.windows.net&authType=instance&role=<role-id>

    ```
  </Tab>

  <Tab title="OCI">
    Fill in the Ops Bucket, Data Bucket, region, and Endpoint parameters according to your actual scenario.

    ```yaml theme={null}
    config: |
      s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true
      s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true
      s3.wal.path=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://xxx.compat.objectstorage.us-ashburn-1.oraclecloud.com&authType=static&pathStyle=true
    ```
  </Tab>

  <Tab title="Alibaba Cloud">
    Input the Ops Bucket, Data Bucket, region, and Endpoint parameters according to the actual requirements.

    ```yaml theme={null}

    config: |
      s3.data.buckets=0@s3://<your-data-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static
      s3.ops.buckets=1@s3://<your-ops-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static
      s3.wal.path=1@s3://<your-data-bucket>?region=xxxx&endpoint=https://oss-cn-xxxx.aliyuncs.com&authType=static

    ```
  </Tab>
</Tabs>

#### Set LoadBalancer Annotations to Enable External Kubernetes Cluster Access

If you need to access AutoMQ from outside the Kubernetes cluster, you need to enable `externalAccess`. To configure an **internal LoadBalancer**, you should modify the `values.yaml` file, specifically the `externalAccess.controller.service.loadBalancerAnnotations` section, and add the following annotations based on your cloud provider:

<Tabs>
  <Tab title="AWS">
    To create an internal Network Load Balancer (NLB), add the following annotations:

    ```yaml theme={null}
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    # If you need to specify subnets, add:
    # service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxxx,subnet-yyyy"
    ```
  </Tab>

  <Tab title="Azure">
    To create an internal standard Load Balancer, add the following annotations:

    ```yaml theme={null}
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    # If you need to specify a subnet, add:
    # service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "your-subnet-name"
    ```
  </Tab>

  <Tab title="GCP">
    To create an internal TCP/UDP Load Balancer, add the following annotations:

    ```yaml theme={null}
    networking.gke.io/load-balancer-type: "Internal"
    # If you need to specify a subnet, add:
    # cloud.google.com/load-balancer-subnet: "your-subnet-name"
    ```
  </Tab>

  <Tab title="OCI">
    To create an internal Load Balancer, add the following annotations:

    ```yaml theme={null}
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/internal: "true"
    oci-network-load-balancer.oraclecloud.com/subnet: "<Replace With Your True Subnet OCID>"
    ```
  </Tab>
</Tabs>

**controller.persistence.metadata.storageClass**

Substitute this parameter with the name of the Storage Class created in step 2, which is designated for storing metadata in the AutoMQ Controller Pod.

#### Revise the Cluster Topology and Resource Request Parameters.

Adjust the cluster topology and resource request parameters based on the resources allocated to AutoMQ Node. The parameters that need modification are as follows:

**broker.replicas**

The AutoMQ Software Chart will start with three Controller Pods by default. These Controller Pods also provide data read and write capabilities. If users wish to horizontally scale more Brokers, they can set the broker.replicas parameter.

* Default value: 0, which represents a three-node cluster without the need for additional Brokers.

* Setting range: >= 0, configured as needed.

**Resource Request Parameters**

AutoMQ Software Controller and Broker Pods need to have the Request and Limit parameters adjusted, along with the corresponding JVM HEAP settings. The configuration files mentioned earlier default to a 4Core16GB specification. Please update these parameters based on the actual allocated computing resources.

* controller.resources.requests.cpu

* controller.resources.requests.memory

* controller.resources.limits.cpu

* controller.resources.limits.memory

* controller.env.\[KAFKA\_HEAP\_OPTS]

* broker.resources.requests.cpu

* broker.resources.requests.memory

* broker.resources.limits.cpu

* broker.resources.limits.memory

* broker.env.\[KAFKA\_HEAP\_OPTS]

### Step 4: Install Chart and Access the Cluster

After customizing the values.yaml configuration file to suit your deployment requirements, proceed with the installation of AutoMQ.

```bash theme={null}

helm upgrade --install <release-name> oci://automq.azurecr.io/helm/automq-enterprise --version 5.3.4 -f <your-custom-values.yaml> --namespace <namespace> --create-namespace

```

Once the installation is complete, users can access AutoMQ within the Kubernetes cluster, either through a Headless service or a LoadBalancer.

<Tip>
  Note: We recommend deploying an Internal LoadBalancer to prevent changes in Pod IP addresses.
</Tip>

### Step 5: Connect and Test the Cluster

#### Headless Service

1. **Locate the Headless service**

```bash theme={null}

kubectl get svc --namespace <namespace> -l "app.kubernetes.io/component=controller" -w

```

2. **Connecting and Testing Using Kafka Clients**

Use the Headless Service for your Kafka client’s `--bootstrap-server` option to send and receive messages. Here’s the command you can use:

```bash theme={null}

./kafka-console-producer.sh \
  --bootstrap-server <release-name>-automq-enterprise-controller-0.<release-name>-automq-enterprise-controller-headless.<namespace>.svc.cluster.local:9092 \
  --topic test-topic

```

#### LoadBalancer

1. **Find External Address**

Wait for the EXTERNAL-IP to be assigned. Use the following command:

```bash theme={null}

kubectl get svc automq-release-automq-enterprise-controller-loadbalancer -n automq -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

```

You can obtain the external IP of the LoadBalancer.

2. **Connect and Test Using Kafka Clients**

Port `9092` is used for client access.

```bash theme={null}

# Replace <EXTERNAL-IP> with the Address from the Previous Step
./kafka-console-producer.sh \
  --bootstrap-server <EXTERNAL-IP>:9092 \
  --topic test-topic

```

## Other Advanced Configurations

The deployment document above provides a basic example of deploying AutoMQ in S3WAL mode. In real-world production environments, users can choose more advanced configurations like EBSWAL and integrate Auto-Scaler support. For the full configuration file, refer to [Helm Chart Values Readme▸](/automq-cloud/appendix/helm-chart-values-readme).

### Configuring the WAL Type

In the previously mentioned installation steps, S3WAL was used as an example. AutoMQ supports deployment options for both EBSWAL and S3WAL modes.

<Tabs>
  <Tab title="S3WAL Mode">
    In S3WAL mode, there's no need to mount a WAL data volume, making the configuration relatively straightforward. First, configure the `global.config.s3.wal.path` parameter.

    ```yaml theme={null}

    config: |
      s3.wal.path=0@s3://<xxx-data-bucket>?region=<region>&endpoint=<endpoint>&authType=<authType>

    ```

    Then, disable `controller.persistence.wal.enabled` and `broker.persistence.wal.enabled`.

    ```yaml theme={null}
    # Applying StorageClass in Controller/broker
    controller:
      persistence:
        metadata:
          storageClass: "your-storage-class"
        wal:
          enabled: false
    broker:
      persistence:
        wal:
          enabled: false


    ```
  </Tab>

  <Tab title="EBSWAL Mode">
    In EBSWAL mode, each controller and broker requires a mounted WAL data volume. Therefore, you need to set `controller.persistence.wal.storageClass` and `broker.persistence.wal.storageClass`, ensuring these values are set to the previously created Storage Class.

    ```yaml theme={null}
    # Applying StorageClass in Controller/broker
    controller:
      persistence:
        metadata:
          storageClass: "your-storage-class"
        wal:
          storageClass: "your-storage-class"
    broker:
      persistence:
        wal:
          storageClass: "your-storage-class"


    ```

    Next, configure the `global.config.s3.wal.path` parameter.

    ```yaml theme={null}
    config: |
      s3.wal.path=0@block:///dev/waliobandwidth=131072000&iodepth=8&iops=3000&capacity=2147483648

    ```
  </Tab>
</Tabs>

### Setting Credentials

AutoMQ supports accessing external resources using either static AccessKeys or dynamic IAM Roles. To prevent the leakage of static AccessKey configurations in production environments, it is recommended to use dynamically generated credentials provided by the cloud provider's IAM Roles.

<Tabs>
  <Tab title="IAM Role Credentials">
    When using IAM Role Credentials, it is necessary to attach the authorization policy to the Role in Step 1. Then, refer to the example below to modify the Credentials configuration.

    ```yaml theme={null}
    global:
      cloudProvider:
        credentials: instance://?role=<your-instance-profile>

      config: |
        s3.data.buckets=0@s3://<your-bucket>?authType=instance&role=<role-id>
        s3.ops.buckets=1@s3://<your-bucket>?authType=instance&role=<role-id>
    ```

    The format for filling out credentials parameters is outlined in the following table:\*\*

    | **Deployment Environment** | **Parameter Values**                                                                                                 |
    | -------------------------- | -------------------------------------------------------------------------------------------------------------------- |
    | AWS<br />                  | instance://?role=\<your-instance-profile><br />For Role, enter the IAM instance profile, not the Role ARN.<br />     |
    | Google Cloud<br />         | instance://?role=\<your-service-account-name><br />For Role, enter the name of the GCP ServiceAccount.<br />         |
    | Azure<br />                | instance://?role=\<your-managed-identity-client-id><br />For Role, enter the Azure Managed Identity Client ID.<br /> |
    | Alibaba Cloud<br />        | instance://?role=\<your-role-id><br />For Role, enter the RAM Role name of Alibaba Cloud.<br />                      |
  </Tab>

  <Tab title="AccessKey Credentials">
    When using AccessKey Credentials, you need to attach the authorization policy to the sub-account or service account in Step 1. Then modify the Credentials configuration as shown in the example below, setting authType to static.

    ```yaml theme={null}
    global:
      cloudProvider:
        credentials: static://?accessKey=<your-accesskey>&secretKey=<your-secretkey>

      config: |
        s3.data.buckets=0@s3://<your-bucket>?authType=static
        s3.ops.buckets=1@s3://<your-bucket>?authType=static
    ```

    After configuring static AccessKey credentials, you must also set these credentials in the environment variables of the Controller and Broker as required by your cloud provider.

    ```yaml theme={null}
    controller:
      env:
        - name: "KAFKA_S3_ACCESS_KEY"
          value: "Replace With Your True ACCESS_KEY"
        - name: "KAFKA_S3_SECRET_KEY"
          value: "Replace With Your True SECRET_KEY"

    broker:
      env:
        - name: "KAFKA_S3_ACCESS_KEY"
          value: "Replace With Your True ACCESS_KEY"
        - name: "KAFKA_S3_SECRET_KEY"
          value: "Replace With Your True SECRET_KEY"

    ```

    <Tip>
      If deploying in Azure, you need to set an additional environment variable: AZURE\_TENANT\_ID, which corresponds to the current tenant ID (tenantId).
    </Tip>
  </Tab>
</Tabs>

### Set Fine-grained Scheduling Policies

In Kubernetes, AutoMQ's fine-grained scheduling policy is implemented using node affinities and tolerations. Users are advised to customize label matching rules based on their node types:

#### Tolerations

It's recommended to add a taint to the Kubernetes node group with the key "dedicated," operator "Equal," value "automq," and effect "NoSchedule." Then, configure the corresponding toleration rules in global.tolerations to schedule Pods:

```yaml theme={null}
global:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "automq"
    effect: "NoSchedule"

```

#### Node Affinities

Override default values in the controller/agent configuration to align with node labels (e.g., node-type: automq-worker):

```yaml theme={null}
controller:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: "node-type"
          operator: In
          values: ["automq-worker"]

```

### Set up Auto-scaling

#### Number of Controllers

By default, the cluster deploys 3 Controller Pods, but users can customize the number of Controller replicas.

<Danger>
  Note: Once the cluster is deployed, adjusting the number of Controller replicas is not supported to avoid unforeseen risks.
</Danger>

### Number of Brokers

The number of brokers is managed by the `broker.replicas` parameter, which allows for horizontal scaling. By default, there are 0 brokers.

### Auto-scaling Configuration

By default, HPA (Horizontal Pod Autoscaler) is disabled. To activate it, two conditions must be fulfilled:

* broker.replicas > 0

* Enable and configure parameters in `global.autoscaling.hpa`:

```yaml theme={null}
global:
  autoscaling:
    hpa:
      enabled: true        # Enable HPA
      minReplicas: "1"     # Minimum Replicas
      maxReplicas: "3"     # Maximum Replicas
      targetCPU: "60"      # Target CPU Utilization (%)
      targetMemory: ""     # Target Memory Utilization (%) (optional)

```

### Identity Recognition Configuration

AutoMQ allows overriding of protocol listeners and enabling secure authentication. By default, it uses the following ports:

* Client to server access: 9092 (PLAINTEXT).

* Internal communication between Controllers: 9093 (PLAINTEXT).

* Internal communication between Brokers: 9094 (PLAINTEXT).

AutoMQ also enables secure authentication by configuring listener overrides (e.g., enabling SASL authentication) for custom ports and protocols. Allowed values include 'PLAINTEXT', 'SASL\_PLAINTEXT', 'SASL\_SSL', and 'SSL'.

```yaml theme={null}
listeners:
  client:
    - containerPort: 9092
      protocol: SASL_PLAINTEXT
      name: BROKER_SASL
  controller:
    - containerPort: 9093
      protocol: SASL_PLAINTEXT
      name: CONTROLLER_SASL
  interbroker:
    - containerPort: 9094
      protocol: SASL_PLAINTEXT
      name: BROKER_SASL

```

<Tip>
  Additionally, you can set a password for it, which is randomly generated by default.
</Tip>

```yaml theme={null}
sasl:
  controller:
    user: "user1"
    password: "PWDxxx"
  interbroker:
    user: "user2"
    password: "PWDxxx"
  client:
    user: "user3"
    password: "PWDxxx"

```

### Configure Prometheus RemoteWrite Metrics Integration

AutoMQ Server supports pushing cluster metrics directly to a user-defined Prometheus instance via the Prometheus RemoteWrite protocol. This approach eliminates the need to deploy additional Prometheus scraping components in the Kubernetes cluster, simplifying the monitoring architecture.

Configure the `s3.telemetry.metrics.exporter.uri` parameter in `global.config` within your `values.yaml` to enable this feature. Choose the appropriate configuration format based on your Prometheus endpoint's authentication method:

<Tabs>
  <Tab title="No Auth">
    For Prometheus endpoints that do not require authentication:

    ```yaml theme={null}
    global:
      config: |
        s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>
    ```
  </Tab>

  <Tab title="Basic Auth">
    Using username and password for basic authentication:

    ```yaml theme={null}
    global:
      config: |
        s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=basic&username=<your-prometheus-username>&password=<your-prometheus-password>
    ```
  </Tab>

  <Tab title="Bearer Token">
    Using Bearer Token for authentication:

    ```yaml theme={null}
    global:
      config: |
        s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=bearer&token=<your-prometheus-token>
    ```
  </Tab>

  <Tab title="AWS SigV4">
    Using AWS SigV4 signature authentication (suitable for Amazon Managed Prometheus and similar services):

    ```yaml theme={null}
    global:
      config: |
        s3.telemetry.metrics.exporter.uri=rw://?endpoint=<your-prometheus-endpoint>&auth=sigv4&region=<your-prometheus-region>
    ```
  </Tab>
</Tabs>

<Tip>
  Once RemoteWrite is enabled, AutoMQ will automatically push core metrics from Brokers and Controllers to the specified Prometheus endpoint. For detailed information on metrics monitoring and alerting configuration, refer to [Prometheus Monitoring & Alerting](/automq-cloud/monitoring-alert/monitoring-alert-via-prometheus).
</Tip>

## Security and Access Control

AutoMQ supports multiple security configurations to protect your data in transit and control client access. This section covers the two primary security models for client authentication when deploying with the Helm chart: `SASL_SSL` and `SSL` (mutual TLS).

These two paths are mutually exclusive. Choose the one that aligns with your organization's security policies.

<Tip>
  Starting with the latest Helm chart, you only need to provide **one** TLS secret containing `ca.crt`, `tls.crt`, and `tls.key`. AutoMQ reuses this PEM bundle for **all** listeners and internal clients (AutoBalancer/admin) so there is no need to prepare separate certificates, and hostname verification for these built-in clients is automatically disabled when mTLS is enabled.
</Tip>

<Note>
  The Helm values `tls.keystorePassword` and `tls.truststorePassword` are used only when loading JKS/PKCS12 keystores. For PEM-based deployments (the default when `tls.type=PEM` or certificates are auto-generated) the chart mounts the PEM files directly, so these passwords are typically left blank unless your private key itself is protected.
</Note>

***

### Path 1: Configuring SASL\_SSL Authentication

This is a common security model where clients authenticate using a username and password, and the communication channel is encrypted with TLS.

#### Step 1: Configure `values.yaml` for SASL\_SSL

You need to define a `SASL_SSL` listener, enable ACLs, and configure SASL users and their passwords. The server will present a TLS certificate to clients, but clients do not need their own certificate to authenticate.

**Example `values.yaml` configuration:**

````yaml theme={null}
# 1. Define a listener for SASL_SSL clients
listeners:
  client:
    - containerPort: 9112
      protocol: SASL_SSL
      name: CLIENT_SASL_SSL
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

# 2. Reference the secret containing the server's TLS certificate
tls:
  type: PEM
  existingSecret: automq-server-tls # Secret must contain server.crt, server.key, ca.crt

# 3. Expose the SASL_SSL port externally
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      extraPorts:
        - name: "sasl-client"
          port: 9112
          protocol: "TCP"
          targetPort: 9112
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60
      annotations:
        external-dns.alpha.kubernetes.io/evaluate-target-health: "false"

# 4. Enable ACLs and define the SASL superuser
acl:
  enabled: true
  superUsers:
    - "_automq"

# 5. Define credentials for the superuser and regular client users
sasl:
  controller:
    password: "\<strong-password-for-automq\>"
    user: _automq
  interbroker:
    password: "\<strong-password-for-automq\>"
    user: _automq
  client:
    users:
      - "my-user" # A regular application user
    passwords:
      - "<password-for-my-user>"

#### (Optional) Automate Route 53 binding with external-dns

If you deploy on AWS and want Helm to publish the bootstrap DNS record automatically:

1. Install [external-dns](https://github.com/kubernetes-sigs/external-dns) in your cluster with `--source=service --provider=aws --policy=upsert-only --registry=txt`. Grant its IAM role `route53:ListHostedZones`, `route53:ListResourceRecordSets`, and `route53:ChangeResourceRecordSets`, and enable IRSA/credentials accordingly.
2. Ensure the base domain you configured under `listeners.client[].advertisedHostnames.baseDomain` exactly matches the Route 53 hosted zone (for example `automq.private`) and the `privateZoneId` fields reference the same zone ID.
3. Add/verify the following block: the controller Service keeps owning the bootstrap hostname, while the listener block tells Kafka which FQDNs to advertise.

```yaml
listeners:
  client:
    - name: CLIENT_SASL_SSL
      containerPort: 9112
      protocol: SASL_SSL
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60
```yaml

Once the controller Service obtains an NLB and renders those annotations, external-dns will upsert `automq-bootstrap.automq.private` automatically. Set `externalAccess.controller.externalDns.enabled=false` if you prefer to copy the hostname and create the record manually.
````

#### Step 2: Post-Deployment ACL Management

After deploying the cluster, you must use the superuser (`_automq`) to grant permissions to regular users like `my-user`.

1. **Configure an Admin Client (`superuser.properties`):**
   This file allows you to run admin tools by authenticating as `_automq`.
   ```properties theme={null}
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
     username="_automq" \
     password="\<strong-password-for-automq\>";
   # The client needs to trust the server's certificate
   ssl.truststore.certificates=/path/to/your/ca.crt
   ```

2. **Grant Permissions:**
   Use `kafka-acls.sh` with the admin configuration to grant `WRITE` and `READ` permissions to `my-user`.
   ```bash theme={null}
   kafka-acls.sh --bootstrap-server <your-load-balancer-dns>:9112 \
     --command-config superuser.properties \
     --add --allow-principal User:my-user --operation WRITE --topic my-topic
   ```

#### Step 3: Client Configuration

A regular application client (`my-user`) would use the following configuration. The client only needs to trust the server, which can be done by providing the CA certificate in a PEM file or a JKS truststore.

**Example using PEM Truststore:**

```properties theme={null}
# client.properties for 'my-user'
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="my-user" \
  password="<password-for-my-user>";

# Path to the PEM file containing the CA certificate
ssl.truststore.certificates=/path/to/your/ca.crt
```

**Example using JKS Truststore:**

```properties theme={null}
# client.properties for 'my-user'
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="my-user" \
  password="<password-for-my-user>";

ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=your_truststore_password
```

***

### Path 2: Configuring SSL (mTLS) Authentication

In this model, clients authenticate by presenting a TLS certificate that is trusted by the cluster. This is known as mutual TLS (mTLS).

#### Step 1: Prepare Certificates

You will need a hierarchy of certificates:

* **Server Certificate:** For the AutoMQ brokers.
* **Admin Client Certificate:** A certificate with a specific Common Name (e.g., `CN=automq-admin`) for an administrator who will be designated as a superuser.
* **Application Client Certificate:** A unique certificate for each client application (e.g., `CN=my-app`).

#### Step 2: Configure `values.yaml` for mTLS

You need to define an `SSL` listener, require client authentication, and set the admin certificate's principal as the superuser.

**Example `values.yaml` configuration:**

```yaml theme={null}
# 1. Define a listener for mTLS clients and require client certs
listeners:
  client:
    - containerPort: 9122
      protocol: SSL
      name: CLIENT_MTLS
      # This enforces mTLS
      sslClientAuth: required
      advertisedHostnames:
        enabled: true
        baseDomain: automq.private
        externalDns:
          privateZoneId: <your-route53-zone-id>

# 2. Reference the secret containing the server's TLS certificate
tls:
  type: PEM
  existingSecret: automq-server-tls

# 3. Expose the SSL port externally
externalAccess:
  controller:
    enabled: true
    service:
      type: LoadBalancer
      extraPorts:
        - name: "tls-client"
          port: 9122
          protocol: "TCP"
          targetPort: 9122
    externalDns:
      enabled: true
      hostname: automq-bootstrap.automq.private
      privateZoneId: <your-route53-zone-id>
      recordType: A
      ttl: 60

# 4. Enable ACLs and define the certificate-based superuser
acl:
  enabled: true
  # The principal is 'User:' + the full Subject of the admin certificate
  superUsers:
    - "User:CN=automq-admin"

# 5. SASL is still used for internal communication
sasl:
  # ... configuration for _automq ...
```

#### Step 3: Post-Deployment ACL Management

After deployment, use the admin certificate to grant permissions to regular application principals.

1. **Configure an Admin Client (`admin.properties`):**
   This file uses the admin certificate (`CN=automq-admin`) to authenticate.

   **Example using PEM files (Recommended):**

   ```properties theme={null}
   security.protocol=SSL
   ssl.truststore.certificates=/path/to/your/ca.crt
   ssl.keystore.key=/path/to/your/admin.key
   ssl.keystore.certificate.chain=/path/to/your/admin.crt
   ```

   **Example using JKS files:**

   ```properties theme={null}
   security.protocol=SSL
   ssl.truststore.location=/path/to/your/truststore.jks
   ssl.truststore.password=your_truststore_password
   ssl.keystore.location=/path/to/your/admin-keystore.jks
   ssl.keystore.password=your_keystore_password
   ```

2. **Grant Permissions:**
   Use `kafka-acls.sh` to grant permissions to the application principal `User:CN=my-app`.
   ```bash theme={null}
   kafka-acls.sh --bootstrap-server <your-load-balancer-dns>:9122 \
     --command-config admin.properties \
     --add --allow-principal "User:CN=my-app" --operation WRITE --topic my-topic
   ```

#### Step 4: Client Configuration

A regular application client would use its own unique certificate (`CN=my-app`) to connect.

**Example using PEM files (Recommended):**

```properties theme={null}
# client.properties for 'my-app'
security.protocol=SSL
ssl.truststore.certificates=/path/to/your/ca.crt
ssl.keystore.key=/path/to/your/myapp.key
ssl.keystore.certificate.chain=/path/to/your/myapp.crt
```

**Example using JKS files:**

```properties theme={null}
# client.properties for 'my-app'
security.protocol=SSL
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=your_truststore_password
ssl.keystore.location=/path/to/your/myapp-keystore.jks
ssl.keystore.password=your_keystore_password
```

```bash theme={null}

# Create a topic
./kafka-topics.sh --create --bootstrap-server <bootstrap-server> \
  --replication-factor 1 --partitions 3 --topic <your-topic> \
  --command-config client-mtls.properties

```

## Performance Tuning

For more details on performance tuning, refer to the [AutoMQ Performance Tuning Guide](https://docs.automq.com/automq/deployment/performance-tuning-for-broker).


# GCP BYOC Console Upgrade Guide to v8.x
Source: https://docs.automq.com/automq-cloud/appendix/gcp-byoc-console-upgrade-to-8.x-guide



## Background

AutoMQ Console 8.x is distributed and started as a Docker image. For an existing AutoMQ BYOC environment on Google Cloud, the upgrade keeps the existing environment data and instances while moving the console service to Docker image mode.

This guide applies to GCP 7.x environments that need to upgrade to 8.x. For new environments, see [Install AutoMQ on Google Cloud](/automq-cloud/getting-started/install-byoc-environment/google-cloud/install-automq-on-gcp).

## Constraints

* GCP BYOC environments currently support GKE Standard only.
* GCP IaaS and GKE Autopilot are not supported.
* Before the upgrade, confirm that existing GKE clusters, node pools, VPCs, subnets, Cloud DNS private managed zones, and related cloud resources still exist.
* The upgrade requires AutoMQ technical personnel to generate environment metadata and Docker startup parameters.

## Pre-upgrade checks

Confirm the following information:

* You can SSH into the console VM.
* The original console data directory still exists, such as `/home/admin/.cmp/data`.
* The environment ID and deployment region match the current environment.
* The Google Service Account bound to the current VM is still the control plane identity used by the AutoMQ console.

During the GCP upgrade, the console reads some cloud resources during startup. We recommend checking that the console Google Service Account has these read permissions:

| Check item                              | Recommended permission                                                                                                 |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Read the Cloud DNS private managed zone | At least `dns.managedZones.get`. `roles/dns.reader` or `roles/dns.admin` usually covers this.                          |
| Read the GKE node pool                  | At least `container.nodePools.get`. `roles/container.viewer` or `roles/container.admin` usually covers this.           |
| Read Shared VPC network resources       | In Shared VPC environments, grant network read permissions in the host project, such as `roles/compute.networkViewer`. |

<Tip>
  If the console asks you to initialize permissions after the upgrade, follow the GCP permission instructions on the page and grant the required permissions to the console Google Service Account.
</Tip>

<Tip>
  For GCP GKE clusters created by 7.x, the data plane usually still obtains cloud resource permissions through the VM Service Account bound to the node pool. As long as the node pool still uses instance metadata, and the original VM Service Account still has the required GCS, Cloud DNS, Compute, and related permissions, existing clusters usually continue to run normally.

  8.x introduces a new Workload Identity permission model: AutoMQ Pods use a Kubernetes ServiceAccount to bind to an instance Google Service Account (GSA), and that GSA accesses GCP resources. This model mainly affects data plane permission configuration for clusters created or updated in the 8.x mode.

  Therefore, upgrading the console itself does not require you to immediately change the metadata mode of existing node pools, and it does not require you to immediately remove permissions from the node pool VM Service Account. For existing 7.x clusters, keep the original node pool authorization unchanged during the upgrade to avoid affecting the running data plane.
</Tip>

## Upgrade steps

The overall process is: register an AutoMQ account, confirm deployment information, obtain the upgrade command, stop the old console, back up the database, install Docker, start the new console, and update the license.

### 1. Register an organization and account

Go to the AutoMQ website and register an organization and account: [https://console.automq.cloud/](https://console.automq.cloud/).

After registration, provide the organization ID to AutoMQ technical personnel.

### 2. Confirm deployment information

Confirm the deployment information of your current BYOC console and send it to AutoMQ technical personnel. This information is used to migrate environment metadata and generate the installation command.

Collect the following information:

* Environment ID
* Deployment region
* Current console version
* Installation ID
* Console endpoint
* Console Google Service Account

We recommend logging in to the old AutoMQ console and checking the settings page. You can also confirm the information from the original startup script, systemd unit, or environment variable file.

### 3. Obtain the upgrade command

AutoMQ technical personnel create or complete the environment record in AutoMQ Cloud based on the current environment information, and then generate the upgrade installation command. The command already contains the environment ID, cloud provider, region, authentication parameters, and Docker image address.

Depending on the target version, the upgrade installation command may use explicit environment variables or a `CONFIG` parameter that carries the encoded environment configuration. Do not assemble these parameters manually.

Before you run the command, confirm that the cloud provider, region, and image version match the current environment. If the image registry requires login, also confirm that you have the Docker registry credentials.

### 4. Stop the old console

SSH into the console VM and stop the old console service. This prevents the old process from writing to the database while you back it up and prevents two console processes from occupying the same ports.

```bash theme={null}
sudo systemctl stop cmp.service
sudo systemctl status cmp.service --no-pager
```

If the old service is configured to restart automatically, disable it during the upgrade window:

```bash theme={null}
sudo systemctl disable cmp.service
```

Confirm that the old process and ports have been released:

```bash theme={null}
ps -ef | grep -E 'cmp|java' | grep -v grep
ss -ltnp | grep -E ':8080|:8085' || true
```

### 5. Back up the database

Back up the local SQLite database after the old console stops. The wildcard also includes any WAL and SHM files.

```bash theme={null}
ts=$(date -u +%Y%m%dT%H%M%SZ)
backup_dir=/home/admin/cmp-migration-backup-$ts
mkdir -p "$backup_dir"

cp -a /home/admin/.cmp/data/sqlite.db* "$backup_dir"/
```

Confirm that the backup files were created and record the backup directory for rollback:

```bash theme={null}
ls -lh "$backup_dir"
echo "$backup_dir"
```

### 6. Install Docker

Install Docker by following the official [Docker Engine installation instructions](https://docs.docker.com/engine/install/) for the VM's operating system. Then confirm that Docker is installed and running:

```bash theme={null}
sudo systemctl start docker
sudo systemctl enable docker
sudo docker version
```

If the image registry requires authentication, log in first and confirm that you can pull the image.

```bash theme={null}
sudo docker login <registry>
sudo docker pull <registry>/automq/automq_byoc_console:<version>
```

### 7. Start the AutoMQ console

Copy the upgrade installation command shown in Step 3 to start the new AutoMQ console. The following command is only a GCP Docker startup example. Use the command generated in Step 3 for the actual upgrade.

The generated command contains credentials. Do not share it or save it in a shared shell history.

```bash theme={null}
sudo docker run -d \
  --name automq-cmp \
  --network host \
  -v /home/admin:/root \
  -e CLOUD_PROVIDER=gcp \
  -e REGION=us-central1 \
  -e ENVIRONMENT_ID=env-xxxx \
  -e OPS_BUCKET=automq-ops-xxxx \
  -e CLIENT_ID=env-xxxx \
  -e CLIENT_SECRET="<client-secret>" \
  -e CONSOLE_INITIAL_USER="<initial-user>" \
  -e CONSOLE_INITIAL_PASSWORD="<strong-initial-password>" \
  <registry>/automq/automq_byoc_console:<version>
```

After you run the upgrade installation command, check the Docker container status and logs:

```bash theme={null}
sudo docker ps
sudo docker logs -f automq-cmp
```

After the Docker container runs correctly and port 8080 is reachable, you can access the AutoMQ console service in your browser.

### 8. Update the License

Because 8.x changes the installation media and startup method, the installation ID may change. After you log in to the new console, if the page shows that the License is invalid, copy the installation ID displayed on the page and contact AutoMQ technical personnel to update the License information.

<Tip>
  A License invalid prompt during the upgrade does not affect existing clusters.
</Tip>

## Roll back

If Docker startup fails and you have not run any instance updates, you can roll back to the original systemd mode.

```bash theme={null}
sudo docker rm -f automq-cmp
backup_dir="<backup-directory-from-step-5>"
cp -a "$backup_dir"/sqlite.db* /home/admin/.cmp/data/
sudo systemctl enable cmp.service
sudo systemctl start cmp.service
sudo systemctl status cmp.service --no-pager
```

If you have already run instance updates, contact AutoMQ technical personnel to confirm the Helm release, Kubernetes resources, and cloud resource states before rolling back the database. Do not overwrite the database until you understand the current cloud resource state.

## Troubleshooting

### Docker container fails to start

Check the following items:

* Whether the Docker image can be pulled.
* Whether the Docker registry token has expired.
* Whether ports 8080 or 8085 are still occupied by the old process.
* Whether `/home/admin` is mounted to `/root` in the container.
* Whether the SQLite files can be read by the container process.

### Console reports insufficient permissions after startup

Check the following items:

* Whether the Google Service Account bound to the console VM is correct.
* Whether the console Google Service Account can read GKE, Cloud DNS, VPC, subnet, and related resources.
* In Shared VPC environments, whether the host project has granted network read permissions.
* Whether you have completed the recommended permissions shown on the console system-init page.


# Helm Chart Values Readme
Source: https://docs.automq.com/automq-cloud/appendix/helm-chart-values-readme

Parameter definitions and recommended settings for the AutoMQ Software Helm Chart, covering cloud provider configuration, storage options, listeners, TLS, and SASL authentication.

This document provides the parameter definitions and recommended settings for the AutoMQ Software Chart. It is automatically generated by the Helm Docs tool.

## Requirements

| Repository                               | Name   | Version |
| ---------------------------------------- | ------ | ------- |
| oci://registry-1.docker.io/bitnamicharts | common | 2.x.x   |

## Values

| Key                                                                     | Type   | Default                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ----------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| acl.enabled                                                             | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param acl.enabled Enable ACLs for AutoMQ #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| acl.superUsers                                                          | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param acl.superUsers Comma-separated list of super users for AutoMQ ACLs #                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| broker                                                                  | object | `{"annotations":{},"env":[],"extraConfig":"","labels":{},"partition":0,"persistence":{"fallback":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""},"wal":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}},"replicas":0,"resources":{},"topologySpreadConstraints":[]}`                                                                                                                                | @section Broker statefulset parameters #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| broker.annotations                                                      | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.annotations Extra annotations for AutoMQ broker pods #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| broker.env                                                              | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.env Extra env arrays for AutoMQ broker pods # E.g. # env: # - name: "KAFKA\_JVM\_PERFORMANCE\_OPTS" #   value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5" # - name: "KAFKA\_OPTS" #   value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError" #                                                                                                                                                                                                                                                                         |
| broker.extraConfig                                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.extraConfig Extra configuration file for Kafka controller nodes, rendered as a template. Auto-generated based on chart values when not specified. #                                                                                                                                                                                                                                                                                                                                                                                   |
| broker.labels                                                           | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.labels Extra labels for AutoMQ broker pods #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| broker.partition                                                        | int    | `0`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param broker.partition Partition rolling update strategy for AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| broker.persistence                                                      | object | `{"fallback":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""},"wal":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}}`                                                                                                                                                                                                                                                                                | Enable persistence using Persistent Volume Claims # ref: [https://kubernetes.io/docs/concepts/storage/persistent-volumes/](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) #                                                                                                                                                                                                                                                                                                                                                           |
| broker.persistence.fallback                                             | object | `{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""}`                                                                                                                                                                                                                                                                                                                                                                                                  | fallback volume for AutoMQ broker nodes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| broker.persistence.fallback.accessMode                                  | string | `"ReadWriteOnce"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param broker.persistence.fallback.accessModes Persistent fallback Volume Access Modes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| broker.persistence.fallback.annotations                                 | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.persistence.fallback.annotations Annotations for the fallback PVC #                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| broker.persistence.fallback.size                                        | string | `"20Gi"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | @param broker.persistence.fallback.size PVC Storage Request for AutoMQ fallback volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| broker.persistence.fallback.storageClass                                | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.persistence.fallback.storageClass PVC Storage Class for AutoMQ fallback volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| broker.persistence.wal                                                  | object | `{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}`                                                                                                                                                                                                                                                                                                                                                                                                   | wal volume for AutoMQ broker nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| broker.persistence.wal.accessMode                                       | string | `"ReadWriteOnce"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param broker.persistence.wal.accessModes Persistent wal Volume Access Modes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| broker.persistence.wal.annotations                                      | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.persistence.wal.annotations Annotations for the PVC #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| broker.persistence.wal.size                                             | string | `"20Gi"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | @param broker.persistence.wal.size PVC Storage Request for AutoMQ wal volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| broker.persistence.wal.storageClass                                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.persistence.wal.storageClass PVC Storage Class for AutoMQ wal volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| broker.replicas                                                         | int    | `0`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param broker.replicas Number of AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| broker.resources                                                        | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param broker.resources Set container requests and limits for different resources like CPU or memory (essential for production workloads) # E.g. # resources: #   requests: #     cpu: 2 #     memory: 512Mi #   limits: #     cpu: 3 #     memory: 1024Mi #                                                                                                                                                                                                                                                                                            |
| controller                                                              | object | `{"annotations":{},"args":[],"command":[],"env":[],"extraConfig":"","labels":{},"partition":0,"persistence":{"fallback":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""},"metadata":{"accessMode":"ReadWriteOnce","annotations":{},"size":"20Gi","storageClass":""},"wal":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}},"replicas":3,"resources":{},"topologySpreadConstraints":[]}`              | @section Controller statefulset parameters #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.annotations                                                  | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.annotations Extra annotations for AutoMQ Controller pods #                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.env                                                          | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.env Extra env arrays for AutoMQ Controller pods # E.g. # env: # - name: "KAFKA\_JVM\_PERFORMANCE\_OPTS" #   value: "-server -XX:+UseZGC -XX:ZCollectionInterval=5" # - name: "KAFKA\_OPTS" #   value: "-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError" #                                                                                                                                                                                                                                                                 |
| controller.extraConfig                                                  | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.extraConfig Extra configuration file for Kafka controller nodes, rendered as a template. Auto-generated based on chart values when not specified. #                                                                                                                                                                                                                                                                                                                                                                                   |
| controller.labels                                                       | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.labels Extra labels for AutoMQ Controller pods #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| controller.partition                                                    | int    | `0`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param controller.partition Partition rolling update strategy for AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| controller.persistence                                                  | object | `{"fallback":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""},"metadata":{"accessMode":"ReadWriteOnce","annotations":{},"size":"20Gi","storageClass":""},"wal":{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}}`                                                                                                                                                                                     | Enable persistence using Persistent Volume Claims # ref: [https://kubernetes.io/docs/concepts/storage/persistent-volumes/](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) #                                                                                                                                                                                                                                                                                                                                                           |
| controller.persistence.fallback                                         | object | `{"accessMode":"ReadWriteOnce","annotations":{},"enabled":false,"size":"20Gi","storageClass":""}`                                                                                                                                                                                                                                                                                                                                                                                                  | fallback volume for AutoMQ controller nodes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| controller.persistence.fallback.accessMode                              | string | `"ReadWriteOnce"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param controller.persistence.fallback.accessModes Persistent fallback Volume Access Modes #                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.persistence.fallback.annotations                             | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.fallback.annotations Annotations for the fallback PVC #                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| controller.persistence.fallback.size                                    | string | `"20Gi"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | @param controller.persistence.fallback.size PVC Storage Request for AutoMQ fallback volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.persistence.fallback.storageClass                            | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.fallback.storageClass PVC Storage Class for AutoMQ fallback volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| controller.persistence.metadata                                         | object | `{"accessMode":"ReadWriteOnce","annotations":{},"size":"20Gi","storageClass":""}`                                                                                                                                                                                                                                                                                                                                                                                                                  | kraft metadata volume for AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| controller.persistence.metadata.accessMode                              | string | `"ReadWriteOnce"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param controller.persistence.metadata.accessMode Persistent metadata Volume Access Modes #                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| controller.persistence.metadata.annotations                             | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.metadata.annotations Annotations for the PVC #                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.persistence.metadata.size                                    | string | `"20Gi"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | @param controller.persistence.metadata.size PVC Storage Request for AutoMQ metadata volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| controller.persistence.metadata.storageClass                            | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.metadata.storageClass PVC Storage Class for AutoMQ metadata volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| controller.persistence.wal                                              | object | `{"accessMode":"ReadWriteOnce","annotations":{},"enabled":true,"size":"20Gi","storageClass":""}`                                                                                                                                                                                                                                                                                                                                                                                                   | wal volume for AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| controller.persistence.wal.accessMode                                   | string | `"ReadWriteOnce"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param controller.persistence.wal.accessModes Persistent wal Volume Access Modes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| controller.persistence.wal.annotations                                  | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.wal.annotations Annotations for the PVC #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| controller.persistence.wal.size                                         | string | `"20Gi"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | @param controller.persistence.wal.size PVC Storage Request for AutoMQ wal volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| controller.persistence.wal.storageClass                                 | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.persistence.wal.storageClass PVC Storage Class for AutoMQ wal volume #                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| controller.replicas                                                     | int    | `3`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param controller.replicas Number of AutoMQ controller nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| controller.resources                                                    | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param controller.resources Set container requests and limits for different resources like CPU or memory (essential for production workloads) # E.g. # resources: #   requests: #     cpu: 2 #     memory: 512Mi #   limits: #     cpu: 3 #     memory: 1024Mi #                                                                                                                                                                                                                                                                                        |
| externalAccess.broker.enabled                                           | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.annotations                           | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.enabled                               | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.hostname                              | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.privateZoneId                         | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.recordType                            | string | `"A"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.externalDns.ttl                                   | int    | `60`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.service.annotations                               | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.service.extraPorts                                | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.service.labels                                    | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.service.loadBalancerAnnotations                   | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.broker.service.type                                      | string | `"LoadBalancer"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.enabled                                       | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.annotations                       | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.enabled                           | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.hostname                          | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.privateZoneId                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.recordType                        | string | `"A"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.externalDns.ttl                               | int    | `60`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.forceExpose                                   | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.service.annotations                           | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.service.extraPorts                            | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.service.labels                                | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.service.loadBalancerAnnotations               | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| externalAccess.controller.service.type                                  | string | `"LoadBalancer"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.automqInstanceId                                                 | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.autoscaling                                                      | object | `{"hpa":{"annotations":{},"enabled":false,"maxReplicas":"","minReplicas":"","targetCPU":"","targetMemory":""}}`                                                                                                                                                                                                                                                                                                                                                                                    | ref: [https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) #                                                                                                                                                                                                                                                                                                                                                                                         |
| global.autoscaling.hpa.annotations                                      | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.autoscaling.hpa.annotations Annotations for HPA. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| global.autoscaling.hpa.enabled                                          | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param global.autoscaling.hpa.enabled Enable HPA for AutoMQ #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| global.autoscaling.hpa.maxReplicas                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.autoscaling.hpa.maxReplicas >= 3 Minimum number of AutoMQ replicas #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.autoscaling.hpa.minReplicas                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.autoscaling.hpa.minReplicas Minimum number of AutoMQ replicas #                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| global.autoscaling.hpa.targetCPU                                        | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.autoscaling.hpa.targetCPU Target CPU utilization percentage #                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| global.autoscaling.hpa.targetMemory                                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.autoscaling.hpa.targetMemory Target Memory utilization percentage #                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| global.cloudProvider.credentials                                        | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.cloudProvider.credential Cloud provider where AutoMQ is running E.g. instance://?role=\<your\_role\_id> or static://?accessKey=\<your\_encoded\_ak>\&secretKey=\<your\_encoded\_sk> #                                                                                                                                                                                                                                                                                                                                                     |
| global.cloudProvider.name                                               | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.cloudProvider.name Cloud provider where AutoMQ is running E.g. aws, azure, gcp, etc. #                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| global.clusterId                                                        | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.commonAnnotations                                                | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.commonAnnotations Annotations to add to all deployed objects #                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| global.commonLabels                                                     | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.commonLabels Labels to add to all deployed objects #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.config                                                           | string | `"s3.data.buckets=0@s3://xxx_bucket?region=us-east-1\ns3.ops.buckets=1@s3://xxx_bucket?region=us-east-1\ns3.wal.path=0@block:///dev/wal\n"`                                                                                                                                                                                                                                                                                                                                                        | @param Bucket URI Pattern: 0\@s3://$bucket?region=$region\&endpoint=\$endpoint #                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| global.daemonSet                                                        | object | `{"enabled":true}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | @param global.daemonSet.enabled Enable AutoMQ image pull daemonSet                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.daemonSet.enabled                                                | bool   | `true`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param global.daemonSet.enabled Enable AutoMQ daemonSet #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| global.existingSecretConfig                                             | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | NOTE: This will override secretConfig value #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| global.image.pullPolicy                                                 | string | `"Always"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.image.pullSecrets                                                | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.image.registry                                                   | string | `"automq-docker-registry-registry.cn-hangzhou.cr.aliyuncs.com"`                                                                                                                                                                                                                                                                                                                                                                                                                                    | @param global.image.registry Global Docker image registry                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| global.image.repository                                                 | string | `"automq/automq-enterprise"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.image.schemaRegistry                                             | object | `{"repository":"automq/karapace","tag":"4.1.0"}`                                                                                                                                                                                                                                                                                                                                                                                                                                                   | @param global.image.schemaRegistry Global Docker image schema registry                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| global.image.tag                                                        | string | `"5.3.1"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.livenessProbe                                                    | object | `{"failureThreshold":4,"initialDelaySeconds":60,"periodSeconds":15,"probePort":9092,"successThreshold":1,"timeoutSeconds":1}`                                                                                                                                                                                                                                                                                                                                                                      | Configure extra options for Kafka containers' liveness probes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| global.livenessProbe.failureThreshold                                   | int    | `4`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.livenessProbe.failureThreshold Failure threshold for livenessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.livenessProbe.initialDelaySeconds                                | int    | `60`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| global.livenessProbe.periodSeconds                                      | int    | `15`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.livenessProbe.periodSeconds Period seconds for livenessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| global.livenessProbe.probePort                                          | int    | `9092`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param global.probePort tcp socket port for livenessProbe and readinessProbe check #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| global.livenessProbe.successThreshold                                   | int    | `1`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.livenessProbe.successThreshold Success threshold for livenessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.livenessProbe.timeoutSeconds                                     | int    | `1`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| global.nodeAffinities                                                   | list   | `[{"key":"kubernetes.io/arch","values":["amd64"]}]`                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.nodeAffinities Affinity for node assignment #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| global.readinessProbe                                                   | object | `{"failureThreshold":8,"initialDelaySeconds":10,"periodSeconds":10,"probePort":9092,"successThreshold":1,"timeoutSeconds":1}`                                                                                                                                                                                                                                                                                                                                                                      | Configure extra options for Kafka containers' readiness probes #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| global.readinessProbe.failureThreshold                                  | int    | `8`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.readinessProbe.failureThreshold Failure threshold for readinessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| global.readinessProbe.initialDelaySeconds                               | int    | `10`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| global.readinessProbe.periodSeconds                                     | int    | `10`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.readinessProbe.periodSeconds Period seconds for readinessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| global.readinessProbe.probePort                                         | int    | `9092`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param global.probePort tcp socket port for livenessProbe and readinessProbe check #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| global.readinessProbe.successThreshold                                  | int    | `1`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.readinessProbe.successThreshold Success threshold for readinessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| global.readinessProbe.timeoutSeconds                                    | int    | `1`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param global.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe #                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| global.serviceAccount.annotations                                       | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param global.serviceAccount.annotations Annotations for the ServiceAccount #                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| global.serviceAccount.create                                            | bool   | `true`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param global.serviceAccount.create Create a ServiceAccount for AutoMQ                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| global.serviceAccount.name                                              | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | If not set and create is true, a name is generated using "automq-sa" #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| global.tolerations                                                      | list   | `[{"effect":"NoSchedule","key":"dedicated","operator":"Equal","value":"automq"}]`                                                                                                                                                                                                                                                                                                                                                                                                                  | Ref: [https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) #                                                                                                                                                                                                                                                                                                                                                                                               |
| global.topologySpreadWhenUnsatisfiable                                  | string | `"DoNotSchedule"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | @param global.topologySpreadWhenUnsatisfiable Topology spread when unsatisfiable schedule policy #                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| global.zoneSpreadConstraints                                            | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param global.zoneSpreadConstraints Availability Zone Balancing Constraint	of AutoMQ nodes #                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| listeners.client\[0].advertisedHostnames.baseDomain                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.client.advertisedHostnames.baseDomain Expected Route53/Private DNS base domain for validation and advertised listener overrides                                                                                                                                                                                                                                                                                                                                                                                                        |
| listeners.client\[0].advertisedHostnames.enabled                        | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param listeners.client.advertisedHostnames.enabled Enable AutoMQ-managed DNS records for this listener                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| listeners.client\[0].advertisedHostnames.externalDns.privateZoneId      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.client.advertisedHostnames.externalDns.privateZoneId Hosted zone identifier (e.g. AWS Route53 private zone ID)                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.client\[0].advertisedHostnames.hostnamePattern                | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.client.advertisedHostnames.hostnamePattern Optional template for DNS record names. Defaults to kernel fallback when empty                                                                                                                                                                                                                                                                                                                                                                                                              |
| listeners.client\[0].advertisedHostnames.ttl                            | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param listeners.client.advertisedHostnames.ttl TTL override (seconds) for Route53 records created by the kernel. Leave null to use the default                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.client\[0].containerPort                                      | int    | `9092`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param listeners.client.containerPort Port for the Kafka client listener #                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| listeners.client\[0].name                                               | string | `"PLAINTEXT"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | listeners.client.name Name for the Kafka client listener                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| listeners.client\[0].protocol                                           | string | `"PLAINTEXT"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | @param listeners.client.protocol Security protocol for the Kafka client listener. Allowed values are 'PLAINTEXT', 'SASL\_PLAINTEXT', 'SASL\_SSL' and 'SSL' #                                                                                                                                                                                                                                                                                                                                                                                            |
| listeners.client\[0].sslClientAuth                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.client.sslClientAuth Optional. If SASL\_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required' #                                                                                                                                                                                                                                                                                                          |
| listeners.controller\[0].advertisedHostnames.baseDomain                 | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.controller\[0].advertisedHostnames.enabled                    | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.controller\[0].advertisedHostnames.externalDns.privateZoneId  | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.controller\[0].advertisedHostnames.hostnamePattern            | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.controller\[0].advertisedHostnames.ttl                        | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.controller\[0].containerPort                                  | int    | `9093`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param listeners.controller.containerPort Port for the Kafka controller listener #                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| listeners.controller\[0].name                                           | string | `"CONTROLLER"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | @param listeners.controller.name Name for the Kafka controller listener #                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| listeners.controller\[0].protocol                                       | string | `"PLAINTEXT"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | @param listeners.controller.protocol Security protocol for the Kafka controller listener. Allowed values are 'PLAINTEXT', 'SASL\_PLAINTEXT', 'SASL\_SSL' and 'SSL' #                                                                                                                                                                                                                                                                                                                                                                                    |
| listeners.controller\[0].sslClientAuth                                  | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.controller.sslClientAuth Optional. If SASL\_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required' #                                                                                                                                                                                                                                                                                                      |
| listeners.interbroker\[0].advertisedHostnames.baseDomain                | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.interbroker\[0].advertisedHostnames.enabled                   | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.interbroker\[0].advertisedHostnames.externalDns.privateZoneId | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.interbroker\[0].advertisedHostnames.hostnamePattern           | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.interbroker\[0].advertisedHostnames.ttl                       | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| listeners.interbroker\[0].containerPort                                 | int    | `9094`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | @param listeners.interbroker.containerPort Port for the Kafka inter-broker listener #                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| listeners.interbroker\[0].name                                          | string | `"BROKER"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | @param listeners.interbroker.name Name for the Kafka inter-broker listener #                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| listeners.interbroker\[0].protocol                                      | string | `"PLAINTEXT"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | @param listeners.interbroker.protocol Security protocol for the Kafka inter-broker listener. Allowed values are 'PLAINTEXT', 'SASL\_PLAINTEXT', 'SASL\_SSL' and 'SSL' #                                                                                                                                                                                                                                                                                                                                                                                 |
| listeners.interbroker\[0].sslClientAuth                                 | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param listeners.interbroker.sslClientAuth Optional. If SASL\_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required' #                                                                                                                                                                                                                                                                                                     |
| networkPolicy.enabled                                                   | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| networkPolicy.ingressRules                                              | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param ingressRules Add ingress rules to the AutoMQ pod(s) # e.g: # ingressRules: #   - ports: target-ports #       - 9102 #     ipBlocks: ip block settings #       - cidr: 0.0.0.0/0                                                                                                                                                                                                                                                                                                                                                                  |
| sasl.client                                                             | object | `{"passwords":null,"users":["user1"]}`                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Credentials for client communications. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| sasl.client.passwords                                                   | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param sasl.client.passwords Comma-separated list of passwords for client communications when SASL is enabled, must match the number of client.users #                                                                                                                                                                                                                                                                                                                                                                                                  |
| sasl.client.users                                                       | list   | `["user1"]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | @param sasl.client.users Comma-separated list of usernames for client communications when SASL is enabled #                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| sasl.controller                                                         | object | `{"password":null,"user":"controller_user"}`                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Credentials for controller communications. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| sasl.controller.password                                                | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param sasl.controller.password Password for controller communications when SASL is enabled. If not set and SASL is enabled for the inter-broker listener, a random password will be generated. #                                                                                                                                                                                                                                                                                                                                                       |
| sasl.controller.user                                                    | string | `"controller_user"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | @param sasl.controller.user Username for controller communications when SASL is enabled #                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| sasl.controllerMechanism                                                | string | `"PLAIN"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | @param sasl.controllerMechanism SASL mechanism for controller communications. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| sasl.enabledMechanisms                                                  | string | `"PLAIN,SCRAM-SHA-256,SCRAM-SHA-512"`                                                                                                                                                                                                                                                                                                                                                                                                                                                              | NOTE: At the moment, Kafka Raft mode does not support SCRAM, that is why only PLAIN is configured. #                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| sasl.existingSecret                                                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| sasl.interBrokerMechanism                                               | string | `"PLAIN"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | @param sasl.interBrokerMechanism SASL mechanism for inter broker communication. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| sasl.interbroker                                                        | object | `{"password":null,"user":"inter_broker_user"}`                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Credentials for inter-broker communications. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| sasl.interbroker.password                                               | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param sasl.interbroker.password Password for inter-broker communications when SASL is enabled. If not set and SASL is enabled for the controller listener, a random password will be generated. #                                                                                                                                                                                                                                                                                                                                                      |
| sasl.interbroker.user                                                   | string | `"inter_broker_user"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param sasl.interbroker.user Username for inter-broker communications when SASL is enabled #                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| schemaRegistry                                                          | object | `{"create":false,"env":null}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | SchemaRegistry configuration #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| schemaRegistry.env                                                      | string | `nil`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | schemaRegistry.env Env for SchemaRegistry #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| service                                                                 | object | `{"headless":{"annotations":{},"labels":{}}}`                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Service for AutoMQ, service type: ClusterIP #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| service.headless                                                        | object | `{"annotations":{},"labels":{}}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Headless service properties #                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| service.headless.annotations                                            | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param service.headless.annotations Annotations for the cluster headless service. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| service.headless.labels                                                 | object | `{}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param service.headless.labels Labels for the cluster headless service. #                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| tls                                                                     | object | `{"autoGenerated":false,"customAltNames":[],"endpointIdentificationAlgorithm":"https","existingSecret":"","jksKeystoreKey":"","jksTruststoreKey":"","jksTruststoreSecret":"","keyPassword":"","keystorePassword":"","passwordsSecret":"","passwordsSecretKeystoreKey":"keystore-password","passwordsSecretPemPasswordKey":"","passwordsSecretTruststoreKey":"truststore-password","pemChainIncluded":false,"selfConfigure":false,"sslClientAuth":"required","truststorePassword":"","type":"JKS"}` | Kafka TLS settings, required if SSL or SASL\_SSL listeners are configured #                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| tls.autoGenerated                                                       | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param tls.autoGenerated Generate automatically self-signed TLS certificates for Kafka brokers. Currently only supported if `tls.type` is `PEM` # Note: ignored when using 'jks' format or `tls.existingSecret` is not empty #                                                                                                                                                                                                                                                                                                                          |
| tls.customAltNames                                                      | list   | `[]`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.customAltNames Optionally specify extra list of additional subject alternative names (SANs) for the automatically generated TLS certificates. #                                                                                                                                                                                                                                                                                                                                                                                              |
| tls.endpointIdentificationAlgorithm                                     | string | `"https"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | @param tls.endpointIdentificationAlgorithm The endpoint identification algorithm to validate server hostname using server certificate # Disable server host name verification by setting it to an empty string. # ref: [https://docs.confluent.io/current/kafka/authentication\_ssl.html#optional-settings](https://docs.confluent.io/current/kafka/authentication_ssl.html#optional-settings) #                                                                                                                                                        |
| tls.existingSecret                                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.existingSecret Name of the existing secret containing the TLS certificates for the Kafka nodes. # - For `tls.type=PEM`, the secret must contain a shared CA (`kafka-ca.crt`) plus a single certificate/key pair (`kafka.crt` + `kafka.key` or `tls.crt` + `tls.key`). The chart converts this bundle into the JKS keystore/truststore files automatically. # - For `tls.type=JKS`, the secret must contain `kafka.keystore.jks` and `kafka.truststore.jks` (also shared by all nodes). Per-role or per-pod keystores are no longer required. |
| tls.jksKeystoreKey                                                      | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.jksKeystoreKey The secret key from the `tls.existingSecret` containing the keystore # Note: ignored when using 'pem' format for certificates. #                                                                                                                                                                                                                                                                                                                                                                                              |
| tls.jksTruststoreKey                                                    | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.jksTruststoreKey The secret key from the `tls.existingSecret` or `tls.jksTruststoreSecret` containing the truststore # Note: ignored when using 'pem' format for certificates. #                                                                                                                                                                                                                                                                                                                                                             |
| tls.jksTruststoreSecret                                                 | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.jksTruststoreSecret Name of the existing secret containing your truststore if truststore not existing or different from the one in the `tls.existingSecret` # Note: ignored when using 'pem' format for certificates. #                                                                                                                                                                                                                                                                                                                      |
| tls.keyPassword                                                         | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.keyPassword Password to access the PEM key when it is password-protected. # Note: ignored when using 'tls.passwordsSecret' #                                                                                                                                                                                                                                                                                                                                                                                                                 |
| tls.keystorePassword                                                    | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.keystorePassword Password to access the JKS keystore when it is password-protected. Ignored when 'tls.passwordsSecret' is provided. # When using tls.type=PEM, the generated keystore will use this password or randomly generate one. #                                                                                                                                                                                                                                                                                                     |
| tls.passwordsSecret                                                     | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.passwordsSecret Name of the secret containing the password to access the JKS files or PEM key when they are password-protected. (`key`: `password`) #                                                                                                                                                                                                                                                                                                                                                                                        |
| tls.passwordsSecretKeystoreKey                                          | string | `"keystore-password"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | @param tls.passwordsSecretKeystoreKey The secret key from the tls.passwordsSecret containing the password for the Keystore. #                                                                                                                                                                                                                                                                                                                                                                                                                           |
| tls.passwordsSecretPemPasswordKey                                       | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.passwordsSecretPemPasswordKey The secret key from the tls.passwordsSecret containing the password for the PEM key inside 'tls.passwordsSecret'. #                                                                                                                                                                                                                                                                                                                                                                                            |
| tls.passwordsSecretTruststoreKey                                        | string | `"truststore-password"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param tls.passwordsSecretTruststoreKey The secret key from the tls.passwordsSecret containing the password for the Truststore. #                                                                                                                                                                                                                                                                                                                                                                                                                       |
| tls.pemChainIncluded                                                    | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param tls.pemChainIncluded Flag to denote that the Certificate Authority (CA) certificates are bundled with the endpoint cert. # Certificates must be in proper order, where the top certificate is the leaf and the bottom certificate is the top-most intermediate CA. #                                                                                                                                                                                                                                                                             |
| tls.selfConfigure                                                       | bool   | `false`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param tls.selfConfigure Enable self-configured TLS certificates for AutoMQ #                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| tls.sslClientAuth                                                       | string | `"required"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | @param tls.sslClientAuth Sets the default value for the ssl.client.auth Kafka setting. # ref: [https://docs.confluent.io/current/kafka/authentication\_ssl.html#optional-settings](https://docs.confluent.io/current/kafka/authentication_ssl.html#optional-settings) #                                                                                                                                                                                                                                                                                 |
| tls.truststorePassword                                                  | string | `""`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | @param tls.truststorePassword Password to access the JKS truststore when it is password-protected. Ignored when 'tls.passwordsSecret' is provided. # When using tls.type=PEM, the generated keystore will use this password or randomly generate one. #                                                                                                                                                                                                                                                                                                 |
| tls.type                                                                | string | `"JKS"`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | @param tls.type Format to use for TLS certificates. Allowed types: `JKS` and `PEM` #                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

***

Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)


# Performance Benchmark
Source: https://docs.automq.com/automq-cloud/appendix/performance-benchmark

AutoMQ delivers cloud-native Kafka with 100% API compatibility, seconds-level scaling, and significant efficiency improvements over Apache Kafka across partition reassignment, elasticity, and catch-up read scenarios.

This document is based on tests conducted using the AutoMQ BYOC kernel, comparing it to Apache Kafka in scenarios like elasticity efficiency, send/receive latency, and cold reads. To replicate and verify the performance test data, please refer to the [Overview▸](/automq-cloud/getting-started/overview) document for installing AutoMQ BYOC.

<Tip>
  In this report, AutoMQ BYOC uses EBS as WAL storage to achieve low-latency message sending and receiving.
</Tip>

## Comparison Conclusion

* **300x Efficiency Improvement in Partition Reassignment Compared to Apache Kafka:** The partition reassignment speed of AutoMQ is approximately 300 times faster than that of Apache Kafka. AutoMQ transforms Kafka's high-risk routine maintenance actions into automated, nearly imperceptible low-risk operational tasks.

* **4min to 1GiB/s Ultimate Elasticity:** AutoMQ's cluster can boost its emergency elasticity from 0 MiB/s to 1 GiB/s in just 4 minutes. This rapid scalability allows the system to accommodate sudden traffic spikes efficiently.

* **200x Cold Read Efficiency over Apache Kafka:** AutoMQ optimizes performance by separating reads and writes, achieving a 200-fold reduction in send latency compared to Apache Kafka and a fivefold increase in throughput. AutoMQ handles peak shaving and filling for online messaging and offline batch processing scenarios.

## Test Preparation

### Configuration Parameters

AutoMQ is set to flush data to disk by default before responding, using the following settings:

```properties theme={null}
acks=all
flush.message=1
```

<Info>
  AutoMQ BYOC ensures high data reliability through the EBS multi-replica mechanism, so you don't need to configure multiple replicas on the Kafka side.
</Info>

Apache Kafka is using version 3.6.0. Following Confluent's guidance, it doesn't set flush.messages = 1. Instead, it relies on a three-replica in-memory asynchronous flush to ensure data reliability (power outages in data centers can lead to data loss), configured as follows:

```properties theme={null}
acks=all
replicationFactor=3
min.insync.replicas=2
```

### Machine Specifications

When considering cost-effectiveness, small instance types combined with EBS are more advantageous than larger machines equipped with SSDs.

For example, small instance r6in.large + EBS versus large instance i3en.2xlarge + SSD:

* i3en.2xlarge, 8 cores, 64 GB memory, network baseline bandwidth 8.4 Gbps, with two 2.5 TB NVMe SSDs, maximum disk throughput of 600 MB/s; price \$0.9040/h.

* r6in.large \* 5 + 5 TB EBS, 10 cores, 80 GB memory, network baseline bandwidth 15.625 Gbps, EBS baseline bandwidth 625 MB/s; price (compute) 0.1743 \* 5 + (storage) 0.08 \* 5 \* 1024 / 24 / 60 = \$1.156/h.

At first glance, the price and performance of the two are almost the same. Considering that in actual production, data is expected to be retained for longer periods, using i3en.2xlarge would require horizontal scaling of compute nodes to increase cluster storage space, which wastes computing resources. With r6in.large + EBS, only the capacity of EBS needs adjustment.

Thus, considering both cost and performance, AutoMQ and Apache Kafka compute both choose r6in.large as the minimum elastic unit for brokers, with storage using GP3 type EBS and Standard S3.

* r6in.large: 2 cores, 16 GB memory, network baseline bandwidth 3.125 Gbps, EBS baseline bandwidth 156.25 MB/s; price \$0.1743/h.

* GP3 EBS: The free tier includes 3000 IOPS and 125 MB/s bandwidth. Pricing for storage is $0.08 per GB per month, with additional bandwidth costing $0.040 per MB/s per month and extra IOPS priced at \$0.005 per month.

AutoMQ and Apache Kafka have different methods of utilizing EBS:

* AutoMQ uses EBS as a write buffer, requiring only 3 GB of storage, and leverages the free tier for IOPS and bandwidth.

* Apache Kafka stores all its data on EBS, with the required space determined by traffic and retention times in specific test scenarios. Additional EBS bandwidth of 31 MB/s is purchased, which further increases Apache Kafka's unit cost per throughput.

## Second-level Partition Reassignment

In a production environment, a Kafka cluster often serves multiple businesses. Fluctuations in business traffic and partition distribution may lead to insufficient cluster capacity or machine hotspots. Kafka operators need to scale the cluster and reassign hotspot partitions to idle nodes to ensure the cluster remains available.

The time taken for partition reassignment determines the efficiency of emergency responses and operations:

* The shorter the partition reassignment time, the quicker the cluster can scale to meet capacity demands, thus reducing the duration of any service impairment.

* Faster partition reassignment shortens the observation period for operators, allowing for quicker feedback and informed subsequent operational decisions.

<Info>
  300x improvement in efficiency: AutoMQ reduces the time required for reassigning a 30 GiB partition compared to Apache Kafka from 12 minutes to just 2.2 seconds.
</Info>

#### Test

This test assesses the reassignment time and the effects of moving a partition containing 30 GiB of data to a node without a replica of that partition, under typical send-consume traffic scenarios using AutoMQ and Apache Kafka. The specific test setup is as follows:

1. Two r6in.large brokers are used to create:

   * One single-partition, single-replica Topic A, with continuous read and write operations at a throughput of 40 MiB/s.

   * A topic B with 4 partitions and a single replica was used, with a continuous read and write throughput of 10 MiB/s serving as background traffic.

After 13 minutes, the only partition of Topic A was reassigned to another node, with reassignment throughput limited to 100 MiB/s.

<Info>
  Each Apache Kafka broker was additionally mounted with a 320GB 156MiB/s gp3 EBS for data storage.
</Info>

| Comparison Item<br />       | AutoMQ<br />                       | Apache Kafka<br />                                                      |
| --------------------------- | ---------------------------------- | ----------------------------------------------------------------------- |
| Reassignment Duration<br /> | 2.2s<br />                         | 12min<br />                                                             |
| Reassignment Impact<br />   | Maximum send latency of 2.2s<br /> | Continuous send latency within 12min fluctuating from 1ms to 90ms<br /> |

***

<img alt="Partition reassignment duration comparison between AutoMQ and Apache Kafka" />

#### Analysis

The reassignment of AutoMQ partitions involves only uploading the buffered data from EBS to S3 to securely open on a new node, typically uploading 500 MiB within 2 seconds. The duration of AutoMQ partition reassignment is independent of the partition's data volume, averaging around 1.5 seconds. During reassignment, AutoMQ partitions return a *NOT\_LEADER\_OR\_FOLLOWER* error code to the client. Upon completion, the client updates to the new Topic routing table, internally retrying to send to the new node, temporarily increasing the partition's send latency until it returns to normal levels after reassignment.

Apache Kafka partition reassignment requires copying the partition's replicas to a new node, catching up with new writes while copying historical data. The reassignment duration equals the partition's data volume divided by (the reassignment throughput limit minus the partition's write throughput). In real-world production environments, partition reassignment often takes hours; in this test, the reassignment of a 30 GiB partition took 12 minutes. Besides the lengthy reassignment, Apache Kafka needs to read cold data from the hard disk, and even with throttling, page cache contention can cause fluctuations in send latency, affecting service quality, depicted in the graph as the green curve's jittery portions.

## 0 -> 1 GiB/s Ultimate Elasticity

Kafka administrators typically plan Kafka cluster capacity based on historical experience, yet unexpected surges in traffic due to hot events and activities continually arise. In such cases, rapid cluster expansion and partition rebalancing become necessary to handle sudden spikes in traffic.

<Info>
  With ultimate elasticity, the AutoMQ cluster can boost its emergency elasticity from 0 MiB/s to 1 GiB/s in just 4 minutes.
</Info>

#### Test

The purpose of this test is to measure the emergency elasticity capability of AutoMQ, specifically the speed of expansion from 0 MiB/s to 1 GiB/s. The specific test scenario is as follows:

1. Initially, the cluster consists of only 1 Broker, with emergency elastic capacity set at 1 GiB/s, and a Topic with 1000 partitions is created.

2. OpenMessaging is started to directly set the sending traffic to 1 GiB/s.

<Tip>
  For AutoMQ BYOC, it is necessary to deploy at least 3 Brokers in the cluster. To test elasticity, you can directly scale AKU via the console UI and observe the scaling speed.
</Tip>

| Analysis Item             | Monitoring Alert | Batch Expansion | Auto Balancing | Total |
| ------------------------- | ---------------- | --------------- | -------------- | ----- |
| 0 -> 1 GiB/s Elastic Time | 70s              | 80s             | 90s            | 4min  |

***

<img alt="AutoMQ cluster scaling from 0 to 1 GiB/s in 4 minutes" />

## Catch-up Read

Catch-up read is a common scenario in messaging and streaming systems:

* In messaging, messages are typically used to decouple business processes and even out peaks and troughs. Smoothing peaks requires that the message queue holds the data sent upstream, allowing it to be consumed gradually downstream. In this scenario, the downstream systems read "catch-up" data that are cold and not in memory.

* In streams, periodic batch processing tasks may need to start scanning and computing data from several hours or even a day ago.

* There are additional fault scenarios: the consumer goes offline for several hours due to a failure and then comes back online; consumer logic issues are fixed, followed by a rewind to consume historical data.

Catch-up reads focus mainly on two aspects:

* The speed of catch-up reads: The faster the catch-up reads, the more quickly the consumer can recover from the failure, and the faster batch processing tasks can produce analytical results.

* Isolation of reads and writes: Catch-up reads should aim to minimally impact the production rate and latency.

<Info>
  With a 200x efficiency improvement, AutoMQ's read-write separation compared to Apache Kafka in catch-up read scenarios reduced sending time from 800ms to 3ms and shortened catch-up time from 215 minutes to 42 minutes.
</Info>

#### Test

This test evaluates the catch-up read performance of AutoMQ and Apache Kafka using an identical cluster size. The test scenarios are outlined as follows:

1. Deploy a cluster with 20 Brokers and create a Topic with 1000 partitions.

2. Continuously send data at a throughput of 800 MiB/s.

3. After sending 4 TiB of data, initiate the consumer to consume from the earliest offset.

<Info>
  Each Apache Kafka broker is equipped with a separate 1000GB 156MiB/s gp3 EBS for data storage.
</Info>

| Comparison Item <br /> | Send Latency During Catch-up Read <br /> | Impact on Send Throughput During Catch-up Read <br /> | Peak Throughput During Catch-up Read <br />         |
| ---------------------- | ---------------------------------------- | ----------------------------------------------------- | --------------------------------------------------- |
| AutoMQ<br />           | Less than 3ms<br />                      | Read-write isolation, maintains 800 MiB/s<br />       | 2500 \~ 2700 MiB/s<br />                            |
| Apache Kafka<br />     | Approximately 800ms<br />                | Interference, drops to 150 MiB/s<br />                | 2600 \~ 3000 MiB/s (at the expense of writes)<br /> |

***

<img alt="Catch-up read throughput comparison between AutoMQ and Apache Kafka" />

***

<img alt="Catch-up read send latency comparison between AutoMQ and Apache Kafka" />

#### Analysis

* In an equivalent cluster size scenario, during catch-up reads, AutoMQ's sending throughput remains stable, whereas Apache Kafka's sending throughput drops by 80%. This is because Apache Kafka performs disk reads during catch-up without IO isolation, consuming the read-write bandwidth of AWS EBS. This reduces the bandwidth available for writing to disk, thereby lowering sending throughput. In contrast, AutoMQ separates reads and writes by utilizing object storage for catch-up reads instead of the disk, which doesn't consume disk read-write bandwidth and thus doesn't affect sending throughput.

* For the same cluster size, during catch-up reads, the average send latency for AutoMQ rises by about 0.4 ms compared to just sending, while Apache Kafka's latency surges to around 800 ms. This spike in send latency for Apache Kafka can be attributed to two factors: first, as noted earlier, catch-up reads consume AWS EBS read-write bandwidth, leading to decreased write throughput and increased latency; second, during catch-up reads, retrieving cold data from the disk contaminates the page cache, similarly resulting in higher write latency.

* It's important to note that when catching up to read 4 TiB of data, AutoMQ took 42 minutes, whereas Apache Kafka took just 29 minutes. The shorter time for Apache Kafka can be attributed to two reasons:

  * During catch-up reads, Apache Kafka's sending traffic was reduced by 80%, which decreased the amount of data it needed to handle while catching up.

  * Apache Kafka does not implement I/O isolation, prioritizing reading speed over maintaining the sending rate.

  If we assume Apache Kafka implements I/O isolation, meaning it reads while maintaining the sending rate as much as possible, the calculation would proceed as follows:

  * Assume that during catch-up reads, Apache Kafka maintains a sending rate of 700 MiB/s. Considering triple-replica writes that occupy EBS bandwidth, it would be 700 MiB/s \* 3 = 2100 MiB/s.

  * The total EBS bandwidth in the cluster is 156.25 MiB/s \* 20 = 3125 MiB/s.

  * The bandwidth available for reading is 3125 MiB/s - 2100 MiB/s = 1025 MiB/s.

  * In a catch-up read scenario where data is simultaneously sent and read, reading 4 TiB of data would take 4 TiB \* 1024 GiB/TiB \* 1024 MiB/GiB / (1025 MiB/s - 700 MiB/s) / 60 s/min = 215 min.

  Apache Kafka requires 215 minutes to catch up on reading 4 TiB of data without significantly impacting the sending process, which is 5 times longer than with AutoMQ.

## Summary

This benchmark illustrates that AutoMQ, after redesigning Kafka for the cloud, offers substantial improvements in both efficiency and cost savings compared to Apache Kafka:

1. In partition reassignment scenarios, AutoMQ reduces the partition reassignment time of a 30GB partition from Apache Kafka's 12 minutes down to 2.2 seconds, achieving a 300x efficiency improvement.

2. With extreme elasticity, AutoMQ can scale out from 0 to 1 GiB/s in just 4 minutes to reach the desired capacity.

3. In scenarios involving historical data catch-up reads, AutoMQ's read-write separation not only improves average send latency by 200-fold, reducing it from 800ms to 3ms, but also achieves catch-up read throughput that is five times that of Apache Kafka.

## Additional Notes

* AutoMQ supports AWS Graviton instances, and this performance comparison is equally applicable to instances supported by Graviton.

* When comparing cloud resource costs in an AWS environment, Apache Kafka incurs cross-availability zone traffic fees. AutoMQ can eliminate these fees through its cross-availability zone routing component. For details, refer to [Save cross-AZ traffic costs with AutoMQ▸](/automq-cloud/best-practice/save-cross-az-traffic-costs-with-automq)


# Kafka Client Config Tuning
Source: https://docs.automq.com/automq-cloud/best-practice/kafka-client-config-tuning

AutoMQ offers cloud-native scalability, Kafka compatibility, and zero partition fees, enhancing performance and cost efficiency. Learn to optimize Kafka client configs for better throughput and reliability.

## Resource Allocation Management

### Recommendation 1: Set the Topic partition count appropriately to avoid throughput bottlenecks and wastage

The number of Kafka Topic partitions affects the production and consumption throughput a Topic can support. To ensure message order, messages with the same partition key are sent to the same partition, and each Kafka partition can only be processed by one consumer.

AutoMQ is built on object storage, which, compared to Apache Kafka's local file architecture, can support several times the partition performance under the same cluster scale. Additionally, AutoMQ does not charge for partitions.

When allocating partitions using AutoMQ, there's no need to consider costs. You only need to evaluate the appropriate number of partitions based on the estimated partition throughput performance (AutoMQ's single partition write throughput limit is 4MB/s) to avoid having to expand partitions later when business volume increases. The number of partitions does not need to be a multiple of the number of nodes, as AutoMQ will automatically balance the partitions.

<Tip>
  **Example:**

  An application estimates that the Topic data write throughput will be about 100MB/s within the next year, while the downstream consumers can each handle a maximum of 2MB of data per second.

  When allocating partitions, you need to consider:

  Based on the producer's perspective, 100MB/s ÷ 4MB/s = 25 partitions are required.

  Based on the consumer's perspective, 100MB/s ÷ 2MB/s = 50 consumers are needed, and 50 consumers require at least 50 partitions.

  Therefore, at least 50 partitions need to be allocated to meet the application's requirements.
</Tip>

### Recommendation 2: For platform-based application scenarios, enable ACL to achieve strict upstream and downstream access control

In platform-based scenarios, such as real-time computing platforms, it is recommended to enable ACL when using Kafka to achieve fine-grained resource control. Once resource control is enabled, Kafka clients must authenticate to access specific Topics and Groups. The benefits of this approach are as follows:

* It prevents business parties from arbitrarily creating new Topics and other resources, which can lead to resource abuse and governance issues.

* It helps avoid subscription chaos and impacts on load balancing caused by different business parties sharing Consumer Groups.

* By identifying Topics and Consumer Groups, it becomes easier to identify related subscribers and upstream and downstream business groups, facilitating business governance.

## Producer Application

### Recommendation 1: For Kafka Producer clients version 2.1 or lower, set the retry count

Producer applications need to check the SDK version. If the current version is less than 2.1, it is necessary to manually set the retry count to ensure automatic retries in case of message send failures. This helps avoid failures caused by server maintenance, self-balancing, etc.

Retry parameter settings can be referenced as follows:

```properties theme={null}
// Maximum retry count
retries=Integer.MAX_VALUE 

// Initial backoff delay time
retry.backoff.ms=100

// Maximum backoff delay time
retry.backoff.max.ms=1000

// RPC timeout for each send request
request.timeout.ms =30000

// The total timeout for the entire send call. If this time is exceeded, no further retries will be made, and an exception will be returned to the caller.
delivery.timeout.ms=120000
```

### Recommendation 2: Optimize the batch parameters of the Producer to avoid excessive QPS consumption by fragmented requests

Kafka is a stream storage system designed for high throughput scenarios. The most typical use case for Kafka is to improve the efficiency and throughput of data transmission through batching. In the process of sending messages by the Producer, it is necessary to set the batching parameters reasonably to avoid the following situations:

* **Avoid sending only one message per request:** If the Producer sends only one message per request, it will generate a large number of Produce requests, consuming server-side CPU and degrading cluster performance.

* **Avoid setting excessively long batching wait times:** When setting batch parameters, the Producer needs to set a reasonable wait time to avoid delays in sending messages due to incomplete batching in low-traffic scenarios.

Specific batch parameter settings are as follows:

```properties theme={null}
// Batch size, up to 128KB of data can be accumulated at a time.
batch.size=131072
// Batch linger time, for the Producer, if the batch limit is not reached within the specified time, sending will also be triggered. This time represents the maximum delay for sending.
linger.ms=10
```

### Recommendation 3: Set ack=all to ensure message durability before responding

Producers can adjust the ack parameter to balance between data durability and sending latency:

* ack=all (default value): The server responds to the client only after the data has been persisted to cloud storage. In the event of a server crash, successfully acknowledged messages will not be lost.

* ack=1: In alignment with Apache Kafka®, the AutoMQ server responds to the client immediately after the message is received in memory. If the server crashes, messages that haven't been persisted will be lost.

It is recommended for producers to maintain the default configuration of ack=all to ensure data reliability.

<Tip>
  AutoMQ achieves message durability and reliable storage using object storage, and the server does not produce ISR replica copies. Setting ACK=ALL handles traffic as if writing a single piece of data (synchronous persistence).
</Tip>

## Consumer Application

### Recommendation 1: For consumers using the Assign mode, upgrade to version 3.2 or above to ensure offset commit success

When a Kafka Consumer application consumes messages using the Assign mode, i.e., self-assigning partition load balancing mode, it is necessary to ensure that the SDK version is upgraded to version 3.2 or above. This is because earlier versions have a defect in the Assign mode where committed consumer offsets are not updated promptly, resulting in consumers failing to commit and update offsets in a timely manner. For detailed defect records, refer to [Kafka Issue KAFKA-13563](https://issues.apache.org/jira/browse/KAFKA-13563).

### Recommendation 2: Control the Consumer heartbeat timeout and messages polled per poll to avoid frequent rebalancing

Kafka Consumers are grouped and load-balanced through the same Group Id. If a single consumer experiences a heartbeat timeout, it will be expelled from the consumer group, and the remaining consumers will undergo self-balancing to reassign partitions.

In production scenarios, program parameter errors that lead to unexpected heartbeat timeouts and self-balancing should be avoided; otherwise, it will cause the consumer group to keep changing and fail to consume messages.

**Causes of unexpected self-balancing:**

* Client versions before v0.10.2: Consumers did not have an independent thread to maintain the heartbeat. Instead, heartbeat maintenance was coupled with the poll interface. As a result, if the user's consumption is delayed, it leads to a Consumer heartbeat timeout, triggering self-balancing.

* Client versions v0.10.2 and later: If the consumption time is too slow and exceeds a certain period (set by max.poll.interval.ms, default is 5 minutes) without polling for messages, the client will voluntarily leave the queue, triggering self-balancing.

<Tip>
  **Optimization Recommendations:**

  Upgrade the client to version 0.10.2 or above.

  Adjust the parameter values according to the following instructions:

  Increase the client's consumption speed as much as possible to avoid lag.

  Try to avoid having a single Consumer Group subscribe to a large number of Topics. Aim for a dedicated Consumer Group to subscribe to a dedicated Topic.
</Tip>

### Recommendation 3: In production scenarios, commit the Consumer offset, but avoid committing offsets too frequently

When consuming messages using Kafka, whether you use the standard Kafka Consumer SDK or frameworks like Flink Connector, it is advisable to commit consumer offsets. This allows you to monitor consumption lag and mitigate risks.

There are two ways to commit consumer offsets: automatic and manual. The controlling parameter is:

* **enable.auto.commit**: Whether to use the automatic offset commit mechanism. The default value is true, indicating that the automatic commit mechanism is used by default.

* **auto.commit.interval.ms**: The interval for auto-committing offsets. The default value is 1000, which is 1 second.

The combination of these two parameters means that before each poll, the client will check the time since the last offset commit. If the elapsed time exceeds the value set by **auto.commit.interval.ms**, the client will initiate an offset commit action.

Therefore, if **enable.auto.commit** is set to true, you need to ensure that the data polled in the previous poll is fully consumed before polling new data. Otherwise, this might lead to offset jumping.

If you prefer to control offset commits manually, set **enable.auto.commit** to false and use the commit(offsets) function to handle offset commits.

<Tip>
  In production scenarios, it is advisable to commit offsets, but avoid committing them too frequently. Frequent commits can cause request hotspots in the Compact Topic partitions handling the offset data.

  For production scenarios, it is generally recommended to commit offsets every 5-10 seconds or manually as needed, rather than committing after every message consumption.
</Tip>

### Recommendation 4: Avoid consumption blocking and accumulation in production scenarios

Kafka consumers process messages in the order of partitions. If a message causes a consumption block due to business logic, it will affect the consumption of subsequent messages in the current partition. Therefore, in production environments, ensure that the consumption logic does not lead to permanent blocking.

If unexpected blocking occurs, it is recommended to follow the steps below:

* **Determine if it can be skipped:** If the abnormal message can be skipped, stop the consumer first, then go to the AutoMQ Console to reset the consumption offset to the next message to achieve a skip.

* **Determine if it cannot be skipped:** Fix the consumption logic to handle the abnormal message.

## Flink Task

### Recommendation 1: Upgrade the Flink Connector version to 1.17.2 or above to ensure offset commit success

When using Flink Connector to consume Kafka messages, pay attention to the Flink Connector version. It is recommended to upgrade to version 1.17.2 or above. Earlier versions of Flink Connector rely on older versions of the Kafka SDK, which cannot retry upon failure when submitting consumption offsets. This can result in the inability to accurately observe the backlog metrics of the Consumer Group in Kafka. For detailed defect records, refer to [Kafka Issue KAFKA-13563](https://issues.apache.org/jira/browse/KAFKA-13563).


# Broker Configuration
Source: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/broker-configuration

AutoMQ optimizes Kafka operations with cloud-native scalability and cross-AZ routing for cost efficiency, reducing inter-zone traffic and ensuring data locality.

In cloud environments like AWS and GCP, cross-AZ traffic incurs data transfer fees. AutoMQ can help avoid the traffic generated by Kafka's production, consumption, and server-side replica replication.

To configure AutoMQ to save on cross-AZ data transfer costs, the following conditions must be met:

1. **Cloud Provider Environment Constraints**: It is only recommended to configure AutoMQ's multi-AZ write feature when deploying in a **multi-AZ setup** on **AWS or GCP**. If your application architecture is single-AZ or you are using other cloud providers (where cross-AZ traffic may be free or lower-cost depending on the cloud provider, region, and pricing policy), this configuration is not recommended.

2. **Multi-AZ Deployment Constraints**: To use AutoMQ's multi-AZ write architecture to save on cross-AZ traffic, both the application and AutoMQ must be deployed in a multi-AZ setup.

   1. **AZ Quantity Constraint**: Currently, only three-AZ deployments for Producers, Consumers, and AutoMQ are supported. If Producers and Consumers are in more than three AZs, some cross-AZ traffic will still be generated.

   2. **AZ Capacity Balance Constraint**: Currently, only balanced AZ deployments are supported, meaning the capacity of Producer and Consumer applications is the same in each AZ. If the capacity is unbalanced, some cross-AZ traffic will still be generated.

## Create an AutoMQ Instance

To use AutoMQ to save on cross-AZ data transfer costs, you first need to ensure that you create a compliant AutoMQ instance. The specific steps are as follows:

1. Log in to the AutoMQ BYOC environment.

2. Create an instance, selecting three availability zones in the network configuration. Choose **S3 WAL or NFS WAL** as the WAL type.

<Tip>
  **Note:**

  When creating an AutoMQ instance, the AZs must be consistent with the deployment regions of the Kafka producers and consumers, and the capacity within all AZs must be balanced. If the capacity is not balanced, a certain proportion of cross-AZ transfer traffic may still be generated.

  *For example, if Kafka producers and consumers are deployed in three regions, A, B, and C, and the application capacity in these three regions is the same, then when creating the AutoMQ instance, you should also select regions A, B, and C.*
</Tip>

## Configure Mapping Rules to Identify Client's Source CIDR Rack

Due to the limitations of the Kafka protocol, the server cannot perceive the Rack to which the Client belongs. To perceive the Client's Rack and avoid cross-AZ production and consumption, AutoMQ provides two Rack configuration methods: [client-side configuration▸](/automq-cloud/eliminate-inter-zone-traffics/client-configuration) and server-side dynamic configuration. This chapter introduces the server-side dynamic configuration method using `automq.zone.cidr.blocks`.

<Tip>
  In a cloud environment, Rack is usually set to [ZoneId](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html). The concepts of Rack and Zone mentioned later can be used interchangeably.
</Tip>

### Parameter format

The format for `automq.zone.cidr.blocks` is as follows:

```text theme={null}
# ex. use1-az1@10.0.0.0/19,10.0.32.0/19<>use1-az2@10.0.64.0/19<>use1-az3@10.0.96.0/19
$zone1@$CIDRBlock1,$CIDRBlock2<>$zone2@$CIDRBlock3<>$zone3@$CIDRBlock4,$CIDRBlock5
```

* Configurations for different Zones are separated by `<>`.
* Each Zone's configuration starts with the Zone name, followed by `@`, and then a list of CIDRs for that Zone. CIDRs in the list are separated by `,`.

AutoMQ will match the Client's connecting IP with the CIDRs sequentially to determine the Client's Zone.

The server-side configuration `automq.zone.cidr.blocks` is suitable for the following scenarios:

* There are clear CIDR segments between the Zones where the Clients are located.
* It is inconvenient to set different Client configurations based on the load's Zone for computing engines like Flink, Spark, or Presto.

If both server-side and client-side configurations are set, the priority is as follows: `client.id` > `automq.zone.cidr.blocks` > Consumer `client.rack`.

### Configuration procedure

1. Sign in to the AutoMQ Console and open the target AutoMQ instance.
2. On the instance details page, choose the **Configurations** tab.
3. Edit `automq.zone.cidr.blocks`, submit the change, and wait for the configuration update to complete.

<Tip>
  You can update `automq.zone.cidr.blocks` in real time. The new CIDR mapping takes effect immediately after the update succeeds.
</Tip>


# Client Configuration
Source: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/client-configuration

Configure AutoMQ clients with zone identifiers and Follower Fetching to eliminate cross-availability-zone data transfer costs.

## Set Zone Identifier

In addition to configuring client Zone identification rules on the server-side ([Server-side Configuration▸](/automq-cloud/eliminate-inter-zone-traffics/broker-configuration)), AutoMQ also supports clients specifying Zone information. This is done by appending `automq_az=$zoneId` to the `client.id` to identify the client's Zone. The `client.id` format is as follows:

```text theme={null}
client.id=automq_az=$zoneId&existing-client-id
```

<Tip>
  On AWS, it is recommended to use the [zoneId](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html) instead of the zoneName, as the zoneId is consistent across all accounts.
</Tip>

## Follower Fetching

Besides setting the Zone identifier via `client.id`, clients with a version greater than 2.4.0 can also use the [KIP-392: Allow consumers to fetch from closest replica](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica) feature to configure consumption Zone affinity. The configuration is as follows:

```text theme={null}
client.rack=$zoneId
```


# Monitor Cross-AZ Traffic
Source: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/monitor-inter-zone-traffic

Monitor cross-AZ traffic with AutoMQ's Kafka-compatible service for cloud-native data flow optimization. Gain insights into network performance efficiently.

After completing the configuration for cross-AZ traffic optimization, users can monitor their local network traffic and match it with the accessed brokers to identify the actual situation of cross-AZ data transfer.

## Monitoring Cross-AZ Traffic with Command-Line Tools

1. **Monitor network traffic**: On Linux, you can use `iftop` to view the incoming and outgoing traffic between the local machine and other IPs. The command is as follows:

   ```bash theme={null}
   iftop -t -s 60 -L 100 -n
   ```

2. **Identify node availability zones**: Query the cluster nodes of AutoMQ and the zones to which the nodes belong. The command is as follows:

   ```bash theme={null}
   ./bin/kafka-broker-api-versions.sh --bootstrap-server $bootstrap_server | grep id
   ```

   By matching the output of these two commands, you can observe whether cross-AZ traffic is being generated.

## Other Methods

If your application is deployed on AWS, you can also use the [Network Flow Monitor](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-NetworkFlowMonitor.html) feature of CloudWatch to monitor cross-Zone traffic.


# Overview
Source: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/overview

AutoMQ eliminates cross-zone data transfer costs using S3 storage architecture, offering cloud-native scalability and 100% Kafka compatibility.

AutoMQ's shared storage architecture, based on S3, can prevent cross-AZ replica replication between brokers and cross-AZ data read/write traffic from clients. This document describes how to use AutoMQ to save on cross-AZ transfer traffic.

<Info>
  The AZ mentioned in this article stands for Available Zone, specifically referring to the availability zones provided by cloud vendors in public cloud environments. Each independent AZ may have its own separate data center facilities.
</Info>

## Background

### Apache Kafka Cross-AZ Traffic Analysis

When users deploy Apache Kafka across multiple AZs in cloud environments like AWS and GCP, they can achieve multi-AZ service disaster recovery. However, this also incurs cross-AZ data transfer costs, which can account for **60-70%** of the total cost in large-scale clusters.

* **Cross-AZ Production Traffic**: Assuming the Producer has not set a ShardingKey and partitions are evenly distributed among cluster nodes, at least 2/3 of the Producer traffic will be sent across AZs. For example, 1/3 of the traffic from a Producer in AZ1 will be sent to AZ2 and AZ3 respectively.

* **Server-side Cross-AZ Replication Traffic**: After a Kafka Broker receives a message, to ensure high data reliability, it replicates the data to Brokers in other AZs, generating twice the amount of Produce cross-AZ traffic.

* **Cross-AZ Consumption Traffic**: Consumers can avoid generating cross-AZ traffic by setting `client.rack` to consume from partitions/replicas in the same AZ.

<img alt="Diagram showing Apache Kafka cross-AZ traffic patterns including producer, replication, and consumer traffic across availability zones" />

### AutoMQ Eliminates Cross-AZ Data Transfer

AutoMQ uses S3 to implement a storage-compute separation architecture. Based on S3, AutoMQ achieves zero cross-AZ traffic costs:

* **Production Traffic**: AutoMQ's built-in Proxy layer intercepts and identifies cross-AZ Produce requests. It builds a cross-AZ proxy channel based on S3 to forward Produce requests to the true partition Leader. Producers only need to communicate with Broker nodes in the same AZ, resulting in no cross-AZ traffic costs.

* **Replication Traffic**: AutoMQ uses S3 as a shared storage layer. S3 uses erasure coding to generate multiple replicas across multiple AZs. There is no replication traffic between AutoMQ Broker nodes as in Apache Kafka.

* **Consumption Traffic**: In addition to the Leader partition, AutoMQ generates a read-only partition in each of the other AZs. The read-only partition reads data written by the Leader partition directly from S3 on demand. Consumers only need to communicate with the Leader partition or the read-only partition in the same AZ, resulting in no cross-AZ traffic costs.

<img alt="Diagram showing AutoMQ's S3-based shared storage architecture eliminating cross-AZ production, replication, and consumption traffic" />

## Configuration Documents

To use AutoMQ's cross-AZ data routing feature to eliminate cross-AZ data transfer, you need to enable configurations on both the server and client sides. The relevant configuration documents are as follows:

* [Server-side Configuration▸](/automq-cloud/eliminate-inter-zone-traffics/broker-configuration)

* [Client-side Configuration▸](/automq-cloud/eliminate-inter-zone-traffics/client-configuration)

After completing the configuration, you can perform data sending and receiving tests and refer to the [Monitor Cross-AZ Traffic▸](/automq-cloud/eliminate-inter-zone-traffics/monitor-inter-zone-traffic) document to observe the actual cross-AZ traffic generated.


# Manage Connectors
Source: https://docs.automq.com/automq-cloud/kafka-connect/manage-connectors

Create a Connect Cluster, create Connectors on it, and manage Connector lifecycle operations.

AutoMQ Connect uses two resource types. A **Connect Cluster** provides the Kafka Connect worker runtime and compute capacity. A **Connector** defines a source or sink data integration workload that runs on a Connect Cluster.

## Prerequisites

Before you begin, prepare the following resources:

* A running AutoMQ instance.
* Kafka credentials with permissions for the topics and consumer groups that the Connector will use. For more information, see [Manage Kafka ACLs](/automq-cloud/manage-security/manage-kafka-acls).
* The source or sink system and its access credentials.
* A Kubernetes cluster, namespace, and ServiceAccount for the Connect Cluster workers. On AWS, bind the ServiceAccount to an IAM role that grants access to the Ops Bucket and any external AWS resources required by the plugin. For EKS preparation instructions, see [Deploy to Amazon EKS](/automq-cloud/deploy-automq-on-kubernetes/deploy-to-aws-eks).

<Tip>
  Grant only the permissions required by the workload. A Sink Connector requires read permission for its Kafka topics and consumer group. A Source Connector requires write permission for its destination topics.
</Tip>

## Create a Connect Cluster

1. Sign in to the AutoMQ Console.
2. In the navigation pane, choose **Kafka Connect**, choose the **Connect Clusters** tab, and then choose **Create Connect Cluster**.
3. Configure the Connect Cluster parameters.

| Parameter               | Description                                                                                                                                          |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Name                    | A name that is unique within the current environment.                                                                                                |
| Description             | An optional description of the worker cluster.                                                                                                       |
| Version                 | The Kafka Connect runtime version.                                                                                                                   |
| Plugins                 | The Managed or custom plugin versions and connector classes to install on the worker cluster.                                                        |
| AutoMQ Instance         | The AutoMQ instance used by the Connect Cluster.                                                                                                     |
| Number of Workers       | The number of Kafka Connect workers. Valid values are 1 through 200.                                                                                 |
| Worker Tier             | The CPU and memory allocated to each worker: 0.5 vCPU and 1 GiB, 1 vCPU and 2 GiB, 2 vCPU and 4 GiB, or 4 vCPU and 8 GiB.                            |
| Kubernetes Cluster      | The Kubernetes cluster where the workers run.                                                                                                        |
| Namespace               | The Kubernetes namespace for the workers.                                                                                                            |
| ServiceAccount          | The Kubernetes ServiceAccount used by the worker Pods.                                                                                               |
| Cloud Identity          | The cloud identity associated with the ServiceAccount, such as an AWS IAM role. It must grant access to the cloud resources required by the plugins. |
| SchedulingSpec          | Optional Kubernetes scheduling settings, including node selectors, affinity and anti-affinity rules, and tolerations.                                |
| Worker Configuration    | Optional Kafka Connect worker properties.                                                                                                            |
| Prometheus Remote Write | Optional endpoint and authentication settings for pushing Connect Cluster metrics to Prometheus-compatible storage.                                  |

4. Review the settings and choose **Create**. Wait until the Connect Cluster status is **Running** before creating a Connector.

<Tip>
  Ensure that the selected Kubernetes node pool has enough allocatable CPU and memory for the worker count and Worker Tier that you select.
</Tip>

## Create a Connector

1. In **Kafka Connect**, choose the **Connectors** tab and then choose **Create Connector**.
2. Configure the Connector parameters.

| Parameter            | Description                                                                                        |
| -------------------- | -------------------------------------------------------------------------------------------------- |
| Connect Cluster      | The running Connect Cluster on which the Connector runs.                                           |
| Name                 | A name that is unique within the selected Connect Cluster.                                         |
| Description          | An optional description of the data integration workload.                                          |
| Plugin               | A plugin installed on the selected Connect Cluster.                                                |
| Connector Type       | Source or Sink. The available type depends on the selected plugin.                                 |
| Connector Class      | The connector implementation class provided by the plugin.                                         |
| Task Count           | The maximum number of Connector tasks that can run in parallel.                                    |
| Kafka Authentication | The Kafka security protocol and credentials used to connect to the AutoMQ instance.                |
| Sink Topics          | For a Sink Connector, the topics to consume and, when applicable, the consumer group.              |
| Source Offsets       | For a Source Connector, the offset topic and initial offset settings when supported by the plugin. |
| Plugin Configuration | Connection information and connector-specific properties for the external system.                  |

3. Enter the plugin configuration. Use the structured form when it is available, or enter key-value properties in custom mode.
4. Review all parameters and choose **Create**.

## Manage a Connector

Open a Connector details page to view its status, tasks, metrics, and logs. You can update supported plugin properties and the task count, and you can perform the following lifecycle operations:

* **Pause:** Stops data processing without deleting the Connector.
* **Resume:** Resumes a paused Connector.
* **Restart:** Restarts the Connector and its tasks to recover from an error or apply supported runtime changes.

Worker count, Worker Tier, Kubernetes placement, installed plugins, and Prometheus Remote Write settings belong to the Connect Cluster. Update those settings on the Connect Cluster rather than on an individual Connector.

## Delete a Connector

On the **Connectors** tab, locate the Connector, choose **Delete**, and confirm the operation.

<Warning>
  Deleting a Connector cannot be undone. Verify the Connector name before you confirm the operation.
</Warning>


# Manage Plugins
Source: https://docs.automq.com/automq-cloud/kafka-connect/manage-plugins

View AutoMQ managed Kafka Connect plugins and manage custom plugins.

Kafka Connect plugins contain the connectors, transforms, and converters that transfer or process data between Kafka and external systems. AutoMQ Connect supports two plugin types:

* **Managed plugins:** AutoMQ maintains, validates, and makes these plugins available for Connect Clusters.
* **Custom plugins:** You upload and manage these plugins when the Managed plugin catalog does not meet your requirements.

## View the plugin list

1. Sign in to the AutoMQ Console.
2. In the navigation pane, choose **Kafka Connect**.
3. Choose the **Plugins** tab.

As of August 2026, AutoMQ provides the following Managed plugins:

| Plugin name          | Version | Functions and features                                                                        |
| -------------------- | ------- | --------------------------------------------------------------------------------------------- |
| debezium-postgresql  | 3.1.2   | Captures row-level changes from PostgreSQL databases and writes change events to Kafka.       |
| debezium-mysql       | 3.1.2   | Captures row-level changes from MySQL databases and writes change events to Kafka.            |
| aerospike-inbound    | 3.2.0   | Writes Kafka records to Aerospike.                                                            |
| mongodb-kafka        | 2.0.1   | Writes Kafka records to MongoDB.                                                              |
| clickhouse-sink      | 1.3.4   | Writes Kafka records to ClickHouse.                                                           |
| clickzetta-connector | 1.0.0   | Writes Kafka records to ClickZetta.                                                           |
| snowflake            | 3.4.0   | Writes Kafka records to Snowflake.                                                            |
| bigquery             | 2.14.0  | Uses the Aiven BigQuery sink to write Kafka records to Google BigQuery.                       |
| google-pubsub        | 1.3.3   | Provides source and sink connectors for data transfer between Kafka and Google Cloud Pub/Sub. |

<Tip>
  **Tip**

  If the Managed plugin catalog does not include the plugin that you require, submit a support ticket to request an update, or create a custom plugin as a temporary solution.
</Tip>

## Create a custom plugin

1. Upload the plugin package to a location that the AutoMQ environment can download, and obtain its download URL.
2. On the **Plugins** tab, choose **Create Custom Plugin**.
3. Enter the plugin name, version, description, main class, and other required metadata.
4. Confirm the configuration. AutoMQ validates the package before creating the plugin version.

## Delete a custom plugin

On the **Plugins** tab, locate the custom plugin version and choose **Delete**. Managed plugins cannot be deleted. A custom plugin version that is referenced by a Connect Cluster cannot be deleted; remove the reference before deleting the plugin.

## Update a custom plugin

To update a custom plugin, create another version by following the custom plugin creation procedure. Multiple versions of a custom plugin can coexist, and existing Connect Clusters continue to use their configured version until you update them.


# Overview
Source: https://docs.automq.com/automq-cloud/kafka-connect/overview

Introduces the basic concepts of Kafka Connect, the core value and deployment principles of AutoMQ managed Kafka Connect, and provides theoretical support for users to subsequently use AutoMQ Connect.

Kafka Connect is a data ETL framework provided by the Apache Kafka community for streaming data between Apache Kafka and other data systems. AutoMQ BYOC provides a fully managed Kafka Connect service, aiming to further lower the entry barrier and operational costs for users.

AutoMQ managed Kafka Connect eliminates the need for customers to install and deploy the Connect framework themselves. Users only need to create Connector tasks in the AutoMQ Console or using Terraform, and the underlying AutoMQ Console is responsible for starting the Connect Worker and Connector tasks.

The Connect Worker is deployed directly in the EKS cluster within the user's specified VPC, ensuring that data flows within the user's network environment, balancing both security and the convenience of a managed service.

* **Network Dependency**: AutoMQ Connect runs on an EKS (Amazon Elastic Kubernetes Service) cluster within the user's VPC. The Connect Worker is deployed as a Pod in EKS and can directly access other resources within the user's VPC (such as RDS, Redis, etc.) without complex network configurations.
* **Runtime Isolation**: To ensure stability and security, each custom plugin runs on a separate Worker node, preventing a single plugin's defects or high resource consumption from affecting the stability of the entire Connect cluster.

## Related Operation Documents

* **Plugin Management**: [《Connect Plugin Management》](./manage-plugins)
* **Connect Task Management**: [《Connect Task Configuration》](./manage-connectors)
* **EKS Environment Preparation**: [《AutoMQ on EKS Deployment Guide》](../deploy-automq-on-kubernetes/deploy-to-aws-eks)


# Mutual TLS (mTLS) Authentication
Source: https://docs.automq.com/automq-cloud/manage-security/authentication/mutual-tls-mtls-authentication

Configure mTLS client authentication for AutoMQ Kafka clients.

Mutual TLS (mTLS) authenticates both sides of a Kafka connection. The client validates the AutoMQ server certificate, and AutoMQ validates a client certificate issued by a trusted client CA. AutoMQ maps the certificate identity to a Kafka ACL user and applies ACL authorization.

## Certificate responsibilities

Server certificate responsibilities depend on the DNS Zone mode selected when the instance is created. In both modes, you provide the client CA and issue a unique client certificate for each client identity.

| DNS Zone mode                     | Server certificate                                                                                                                            | Client certificates                                                                                                                                                                          |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AutoMQ Managed Public DNS Zone    | AutoMQ provides, installs, and renews the authoritative server certificate. Do not upload a user-issued Server Certificate or Private Key.    | You provide the client CA certificate and use the client CA private key to issue client certificates.                                                                                        |
| Customer-managed Private DNS Zone | You provide and rotate the server CA, Server Certificate, and Private Key. The Server Certificate SAN must match the private endpoint domain. | You provide the client CA and use its private key to issue client certificates. If the server and client use different CAs, combine their public certificates into one CA bundle for upload. |

<Note>
  Select the DNS Zone mode and enable mTLS when you create the instance. Existing instances cannot be changed from an unencrypted listener to mTLS.
</Note>

## Configuration workflow

Complete the following steps in order:

1. Create or select a client CA and issue a Client Certificate. This step is required for both DNS Zone modes.
2. Prepare the server identity. AutoMQ manages it for a Public DNS Zone; you provide it for a Private DNS Zone.
3. Configure the AutoMQ instance.
4. Configure the Kafka client.

## Step 1: Create a client CA and Client Certificate

Both DNS Zone modes require a client CA and a Client Certificate. Use your organization's certificate management system when available. If you need to create and maintain a private client CA, use the following procedure as a reference.

<Warning>
  Protect `client-ca.key` and `client.key` according to your organization's key management requirements. For client authentication, upload `client-ca.crt`, but never upload the client CA private key or Client Private Key. A customer-managed Private DNS Zone also requires the server materials described in Step 3.
</Warning>

### Create a client CA

Create `client-ca.cnf`. Replace the distinguished-name values before running the commands.

```ini theme={null}
[ req ]
prompt = no
distinguished_name = distinguished_name
x509_extensions = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName = <client-ca-common-name>

[ extensions ]
basicConstraints = critical,CA:true,pathlen:1
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
```

Generate the client CA private key and certificate:

```bash theme={null}
openssl genrsa -out client-ca.key 2048
chmod 400 client-ca.key
openssl req -new -x509 \
  -config client-ca.cnf \
  -key client-ca.key \
  -days 3650 \
  -out client-ca.crt
```

### Issue a Client Certificate

Create the corresponding Kafka ACL user before issuing the Client Certificate. Unless you configure a different principal mapping rule, set the certificate Common Name (CN) to the ACL username.

The default principal mapping extracts the CN from the Client Certificate:

```text theme={null}
RULE:.*CN=([^,]+).*/$1/
```

For example, a certificate with `CN=user01` maps to the ACL user `user01`.

Create `client.cnf`:

```ini theme={null}
[ req ]
prompt = no
distinguished_name = distinguished_name
req_extensions = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName = <kafka-acl-username>

[ extensions ]
basicConstraints = critical,CA:false
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = clientAuth
```

Generate the Client Private Key and CSR, and then sign the CSR with the client CA:

```bash theme={null}
openssl genrsa -out client.key 2048
chmod 400 client.key
openssl req -new \
  -config client.cnf \
  -key client.key \
  -out client.csr
openssl x509 -req \
  -in client.csr \
  -CA client-ca.crt \
  -CAkey client-ca.key \
  -CAcreateserial \
  -days 365 \
  -extfile client.cnf \
  -extensions extensions \
  -out client.crt
```

Verify the certificate identity, purpose, and validity period:

```bash theme={null}
openssl x509 -in client.crt -noout -subject -issuer -dates -ext extendedKeyUsage
```

## Step 2: Prepare the server identity

### AutoMQ Managed Public DNS Zone

No user action is required. AutoMQ provides, installs, and renews the authoritative Server Certificate. Do not create or upload a Server Certificate or Server Private Key.

### Customer-managed Private DNS Zone

You must provide a Server Certificate whose Subject Alternative Name (SAN) covers the AutoMQ private endpoint names associated with the Private DNS Zone. Use a certificate issued by your organization's certificate authority when available.

The following reference procedure uses the client CA created in Step 1 to sign the Server Certificate. If your organization requires separate server and client CAs, use the server CA to sign the Server Certificate and retain both public CA certificates for the CA bundle in Step 3.

Create `server.cnf`. Use the exact DNS name or wildcard required by the instance endpoint. Do not rely on the Common Name for hostname verification.

```ini theme={null}
[ req ]
prompt = no
distinguished_name = distinguished_name
req_extensions = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName = <server-common-name>

[ extensions ]
basicConstraints = critical,CA:false
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = <private-endpoint-dns-name-or-wildcard>
```

Generate the Server Private Key and CSR, and then sign the CSR:

```bash theme={null}
openssl genrsa -out server.key 2048
chmod 400 server.key
openssl req -new \
  -config server.cnf \
  -key server.key \
  -out server.csr
openssl x509 -req \
  -in server.csr \
  -CA client-ca.crt \
  -CAkey client-ca.key \
  -CAserial client-ca.srl \
  -days 365 \
  -extfile server.cnf \
  -extensions extensions \
  -out server.crt
```

Verify the certificate SAN, purpose, and validity period:

```bash theme={null}
openssl x509 -in server.crt -noout -subject -issuer -dates -ext subjectAltName,extendedKeyUsage
```

## Step 3: Configure AutoMQ

### AutoMQ Managed Public DNS Zone

1. When creating the instance, select `Public DNS Zone` and the AutoMQ managed mode.
2. Enable the mTLS listener.
3. Upload `client-ca.crt` as the Client CA certificate.
4. Do not upload a Server Certificate or Server Private Key.
5. Create the ACL user that matches the Client Certificate identity and grant only the required Topic and Consumer Group permissions.

### Customer-managed Private DNS Zone

1. When creating the instance, select the customer-managed `Private DNS Zone` and enable the mTLS listener.
2. Upload a PEM CA bundle that contains the client CA and server CA public certificates. If one CA issues both certificates as in this example, upload `client-ca.crt` once.
3. Upload `server.crt` as the Server Certificate and `server.key` as the Server Private Key.
4. Create the ACL user that matches the Client Certificate identity and grant only the required Topic and Consumer Group permissions.

## Step 4: Configure a Kafka client

Convert the Client Certificate and Client Private Key to a PKCS#12 keystore:

```bash theme={null}
openssl pkcs12 -export \
  -in client.crt \
  -inkey client.key \
  -certfile client-ca.crt \
  -out client.p12 \
  -name automq-client \
  -password pass:<keystore-password>
```

Configure the Kafka client:

```properties theme={null}
bootstrap.servers=<mtls-bootstrap-server>
security.protocol=SSL
ssl.keystore.type=PKCS12
ssl.keystore.location=/path/to/client.p12
ssl.keystore.password=<keystore-password>
ssl.key.password=<keystore-password>
```

For an AutoMQ Managed Public DNS Zone, the client normally trusts the public Server Certificate through the operating system or Java runtime. No custom server truststore is required unless your organization requires one.

For a customer-managed Private DNS Zone whose server CA is not already trusted, import the server CA certificate into a client truststore. In the single-CA example above, use `client-ca.crt` as the server CA certificate.

```bash theme={null}
keytool -importcert \
  -alias automq-server-ca \
  -file <server-ca-certificate>.crt \
  -keystore truststore.jks \
  -storepass <truststore-password>
```

Add the truststore settings to the client configuration:

```properties theme={null}
ssl.truststore.type=JKS
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=<truststore-password>
```

## Certificate expiration responsibility

* **AutoMQ Managed Public DNS Zone:** AutoMQ monitors and renews the server certificate. You monitor and rotate the client CA and all client certificates.
* **Customer-managed Private DNS Zone:** You monitor and rotate the server CA, Server Certificate, client CA, and client certificates.

The server exposes `kafka_stream_cert_expiry_timestamp_milliseconds` and `kafka_stream_cert_days_remaining` for the installed server certificate. For collection methods, see [Monitoring and Alerting with Prometheus](/automq-cloud/monitoring-alert/monitoring-alert-via-prometheus). Client certificate monitoring must be implemented in your certificate management process.


# SASL Authentication
Source: https://docs.automq.com/automq-cloud/manage-security/authentication/sasl-authentication

Configure SASL_PLAINTEXT or SASL_SSL authentication for AutoMQ Kafka clients.

AutoMQ supports SASL authentication with Kafka ACL users. Use `SASL_PLAINTEXT` only on a trusted private network. Use `SASL_SSL` to encrypt client traffic with TLS.

## Enable SASL\_PLAINTEXT

### Server configuration

When you create the instance, enable SASL authentication and the `SASL_PLAINTEXT` listener. Create an ACL user and grant only the Topic and Consumer Group permissions required by the application.

### Client configuration

Configure the Kafka client with the credentials created in AutoMQ. The following example uses SASL/PLAIN:

```properties theme={null}
bootstrap.servers=<sasl-plaintext-bootstrap-server>
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
```

## Enable SASL\_SSL

`SASL_SSL` combines SASL client authentication with TLS server authentication and transport encryption. Server certificate responsibilities depend on the DNS Zone mode selected when the instance is created.

| DNS Zone mode                     | Server certificate responsibility                                                                               | User-provided server TLS material                                         |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| AutoMQ Managed Public DNS Zone    | AutoMQ provides and manages the authoritative server certificate for the instance endpoint.                     | None. Do not upload a self-signed CA, Server Certificate, or Private Key. |
| Customer-managed Private DNS Zone | You provide and rotate a server certificate whose Subject Alternative Name matches the private endpoint domain. | Server CA certificate, Server Certificate, and Private Key.               |

<Note>
  Select the DNS Zone mode and enable `SASL_SSL` when you create the instance. Existing instances cannot be changed from an unencrypted listener to `SASL_SSL`.
</Note>

### Use an AutoMQ Managed Public DNS Zone

1. When creating the instance, select **Public DNS Zone** and the AutoMQ managed mode.
2. Enable the `SASL_SSL` listener and configure SASL authentication.
3. Do not upload user-managed server certificate files. AutoMQ provisions and renews the server certificate.
4. Create an ACL user and grant the required resource permissions.

Because the server certificate is issued by a public certificate authority, Kafka clients normally use the trust anchors supplied by the operating system or Java runtime. Do not configure a custom truststore unless your organization requires one.

```properties theme={null}
bootstrap.servers=<sasl-ssl-bootstrap-server>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
```

### Use a customer-managed Private DNS Zone

For a customer-managed Private DNS Zone, you must provide a CA certificate, Server Certificate, and Server Private Key. Use certificates issued by your organization's certificate authority when available. If you need to create and maintain a private CA, use the following procedure as a reference.

<Warning>
  The following commands create a private CA. Protect `ca.key` and `server.key` according to your organization's key management requirements. Do not upload the CA private key to AutoMQ or distribute either private key to Kafka clients.
</Warning>

#### 1. Create a private CA

Create `ca.cnf`. Replace the distinguished-name values before running the commands.

```ini theme={null}
[ req ]
prompt = no
distinguished_name = distinguished_name
x509_extensions = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName = <private-ca-common-name>

[ extensions ]
basicConstraints = critical,CA:true,pathlen:1
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
```

Generate the CA private key and CA certificate:

```bash theme={null}
openssl genrsa -out ca.key 2048
chmod 400 ca.key
openssl req -new -x509 \
  -config ca.cnf \
  -key ca.key \
  -days 3650 \
  -out ca.crt
```

#### 2. Issue the Server Certificate

Create `server.cnf`. The Subject Alternative Name (SAN) must cover the AutoMQ private endpoint names associated with the Private DNS Zone. Use the exact DNS name or wildcard required by the instance endpoint. Do not rely on the Common Name for hostname verification.

```ini theme={null}
[ req ]
prompt = no
distinguished_name = distinguished_name
req_extensions = extensions

[ distinguished_name ]
organizationName = <organization-name>
commonName = <server-common-name>

[ extensions ]
basicConstraints = critical,CA:false
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = <private-endpoint-dns-name-or-wildcard>
```

Generate the Server Private Key and certificate signing request (CSR), and then sign the CSR with the private CA:

```bash theme={null}
openssl genrsa -out server.key 2048
chmod 400 server.key
openssl req -new \
  -config server.cnf \
  -key server.key \
  -out server.csr
openssl x509 -req \
  -in server.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -days 365 \
  -extfile server.cnf \
  -extensions extensions \
  -out server.crt
```

Verify the certificate SAN and validity period before uploading it:

```bash theme={null}
openssl x509 -in server.crt -noout -subject -issuer -dates -ext subjectAltName
```

#### 3. Configure AutoMQ

1. When creating the instance, select the customer-managed **Private DNS Zone** and enable the `SASL_SSL` listener.
2. Upload `ca.crt` as the Server CA certificate, `server.crt` as the Server Certificate, and `server.key` as the Private Key.
3. Create an ACL user and grant only the Topic and Consumer Group permissions required by the application.

#### 4. Configure a Kafka client

Import `ca.crt` into the client truststore when the client runtime does not already trust the private CA:

```bash theme={null}
keytool -importcert \
  -alias automq-server-ca \
  -file ca.crt \
  -keystore truststore.jks \
  -storepass <truststore-password>
```

```properties theme={null}
bootstrap.servers=<sasl-ssl-bootstrap-server>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
ssl.truststore.type=JKS
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=<truststore-password>
```

## Certificate expiration responsibility

* **AutoMQ Managed Public DNS Zone:** AutoMQ monitors and renews the server certificate. You do not rotate the server certificate.
* **Customer-managed Private DNS Zone:** You are responsible for monitoring and rotating the server CA and certificate before expiration. Use `kafka_stream_cert_expiry_timestamp_milliseconds` and `kafka_stream_cert_days_remaining` to monitor the installed server certificate. For collection methods, see [Monitoring and Alerting with Prometheus](/automq-cloud/monitoring-alert/monitoring-alert-via-prometheus).


# Data Encryption at Rest
Source: https://docs.automq.com/automq-cloud/manage-security/data-encryption-at-rest

AutoMQ ensures transparent data encryption at rest with cloud-native scalability and Kafka compatibility, utilizing cloud vendor services for robust data security.

AutoMQ provides at-rest encryption for application data, ensuring that data encryption is fully transparent to the user's application and is facilitated through the cloud vendor's storage services. This document outlines the steps to configure at-rest data encryption.

## **Operation Principle**

In AutoMQ's BYOC service, data storage is primarily divided into two categories:

* **Object Storage:** This service is utilized for storing message data, system logs, metrics, and other data types.

* **Block Storage:** This service is employed for storing control plane metadata, such as Kafka KRaft, instances, and accounts.

Both types of storage services use transparent data encryption provided by cloud vendors to encrypt data at rest. Simply enable encryption of data at rest during the creation of the BYOC console and instance to achieve transparent encryption.

For more information on how cloud storage implements transparent encryption, you can refer to the following:

* [https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html)

* [https://docs.aws.amazon.com/ebs/latest/userguide/ebs-encryption.html](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-encryption.html)

<Tip>
  **Usage Restrictions:**

  The encryption keys for AutoMQ BYOC service storage are managed by the cloud vendor, and BYOK (Bring Your Own Key) custom keys are not supported.

  The data encryption feature is only supported on AWS. If needed, contact us via [Obtaining Services▸](/automq-cloud/support/obtaining-services).
</Tip>

## Configuration Instructions

When enabling end-to-end static data encryption, follow the steps outlined below. It's important to enable this feature both during the creation of a BYOC environment console and when setting up an instance. Making changes to existing environment consoles or instances is not supported.

1. When creating a BYOC console, **enable the DataEncryption option**.

<img />

2. When creating an instance, click on advanced options and **enable DataEncryption**.

<img />


# Manage Kafka ACLs
Source: https://docs.automq.com/automq-cloud/manage-security/manage-kafka-acls

Enable Kafka ACLs with AutoMQ, ensuring cloud-native scalability, full Kafka compatibility, and cost efficiency. Learn crucial tips for optimal security.

This article introduces the basic concepts of Kafka ACL functionality, relevant constraints, and how to enable and use ACL functionality in AutoMQ instances.

## Kafka ACL Concepts

### Overview

Apache Kafka provides ACL (Access Control List) functionality, supporting authentication and authorization for Kafka clients. AutoMQ is fully compatible with Apache Kafka, so when using an AutoMQ instance, the Kafka ACL functionality can be enabled and utilized.

<Tip>
  When using Kafka ACL, be aware of the following constraints:

  **Version Constraint:** Ensure that the AutoMQ instance version is >= 1.10.x. If not, upgrade the version before enabling it.

  **Optional Enablement:** By default, AutoMQ instances do not have ACL enabled. Users can manually enable ACL during instance creation or afterward.

  **Access Protocols:** AutoMQ supports accessing the cluster through the VPC internal network using SASL\_PLAINTEXT, SASL\_SSL, and mTLS protocols.

  **Resource Types:** AutoMQ supports ACL permission control for resource types such as Kafka Cluster, Topic, Consumer Group, and Transaction ID, but not for DelegationToken. If needed, refer to [Obtaining Services▸](/automq-cloud/support/obtaining-services) to contact us.
</Tip>

### ACL User

In the Kafka ACL feature, the user is the subject for ACL identity recognition. The Kafka client transmits its user identity through the access protocol, and the Kafka server confirms the identity of the user corresponding to the current client request before proceeding with subsequent permission verification and other operations.

Note the difference between a Kafka ACL user and an AutoMQ environment member:

| **Scenario Comparison**<br /> | **Kafka ACL User**<br />                  | **AutoMQ Environment Member**<br />                                      |
| ----------------------------- | ----------------------------------------- | ------------------------------------------------------------------------ |
| **Usage Scenario**<br />      | Access via Kafka client API and CLI<br /> | Access via AutoMQ environment web<br />                                  |
| **Scope of Coverage**<br />   | Acts on a single Kafka instance<br />     | Acts on the entire environment, allowing cross-instance operations<br /> |

<Tip>
  **Note**:

  When accessing AutoMQ instances via the mTLS protocol, the server will by default map the client's certificate identity to an ACL user. The default mapping rule is set as RULE:.*CN=(\[^,]+).*/\$1/.

  Example: For a client certificate identity of: *CN=user01,O=xxxx,OU=xxx,L=xxx,S=xxx,C=xxx*

  AutoMQ extracts the CN field and constructs the Principal= User:user01.
</Tip>

### ACL Authorization Policy

Apache Kafka specifies the range of resources that different users can access and the corresponding list of operations that can be executed through authorization policies. Refer to Apache Kafka's [design documentation](https://kafka.apache.org/documentation/#security_authz). The authorization policy for an AutoMQ instance includes the following information:

| **Parameter**<br />           | **Description**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Resource Type**<br />       | <ul><li>Description: Resource type refers to the categories of Kafka resources applicable to Kafka ACL policies, supporting the following types:</li><ul><li>Cluster: Defines operation permissions at the instance (cluster) level, such as viewing cluster configurations and cloud information.</li><li>Topic: Defines operation permissions at the Topic level, such as deleting, subscribing, sending messages, etc.</li><li>Consumer Group: Defines the permissions for Kafka Consumer clients to join Groups and participate in receiving messages.</li><li>TransactionID: Defines the usage permissions for transactional messages.</li></ul></ul> |
| **Resource Name**<br />       | <ul><li>Description: Resource Name defines the scope of resources covered by the ACL policy. It can be an exact name or a prefix defined through custom string conventions.</li><li>Restrictions: Refer to [Restrictions▸](/automq-cloud/overview/restrictions) .</li></ul>                                                                                                                                                                                                                                                                                                                                                                                |
| **Match Type**<br />          | <ul><li>Description: Match Type describes whether the Resource Name field in the current ACL rule is an exact match or a prefix match.</li><li>Types:</li><ul><li>literal: Exact match, meaning the resource name is the actual effective name.</li><li>prefixed: Prefix match, meaning the resource name is the prefix of the actual effective resource.</li></ul></ul>                                                                                                                                                                                                                                                                                   |
| **Operation Group**<br />     | <ul><li>Description: An Operation Group is a bundled combination of fine-grained atomic authorization operations (Operations) in Apache Kafka by AutoMQ. Each operation group contains multiple Kafka operations. Operation groups simplify the authorization process for a series of typical use cases.</li><li>For the mapping relationship between operation groups and Kafka ACL operations, refer to the table below.</li></ul>                                                                                                                                                                                                                       |
| **Authorization Types**<br /> | <ul><li>Description: Apache Kafka ACL supports the following two authorization types:</li><ul><li>ALLOW: This ACL policy permits users to perform operations on the associated resources.</li><li>DENY: This ACL policy prevents users from performing operations on the associated resources.</li></ul></ul><Tip><br />Note: A DENY type authorization policy functions like a denylist capability, is exclusive, and takes precedence over ALLOW.<br /></Tip><br />                                                                                                                                                                                      |
| **Access Source**<br />       | <ul><li>Description: Access Source defines when the current ACL policy is effective only for the current IP access source. It is not effective for an IP access source that is not matched.</li><li>Limitation: Access Source supports only precise IP addresses and does not support CIDR networks or domain names.</li></ul>                                                                                                                                                                                                                                                                                                                             |

AutoMQ defines operation groups to simplify authorization processes, and the mapping between operation groups and Kafka operations is displayed in the table below:

<table>
  <thead>
    <tr>
      <th>
        Resource Type

        <br />
      </th>

      <th>
        Operation Group

        <br />
      </th>

      <th>
        Included Operations

        <br />
      </th>

      <th>
        Recommended Scenarios

        <br />
      </th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        Cluster

        <br />
      </td>

      <td>
        All

        <br />
      </td>

      <td>
        <ul>
          <li>Alter</li>

          <li>AlterConfigs</li>

          <li>ClusterAction</li>

          <li>Create</li>

          <li>Describe</li>

          <li>DescribeConfigs</li>

          <li>All</li>
        </ul>
      </td>

      <td>
        The All operation group grants full permissions for all actions on the Kafka cluster (instance). It is generally advised to assign this to user identities utilizing Kafka UI or CLI tools.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Topic

        <br />
      </td>

      <td>
        All

        <br />
      </td>

      <td>
        <ul>
          <li>Alter</li>

          <li>AlterConfigs</li>

          <li>Create</li>

          <li>Delete</li>

          <li>Describe</li>

          <li>DescribeConfigs</li>

          <li>Read</li>

          <li>Write</li>
        </ul>
      </td>

      <td>
        The All operation group provides full permissions for all actions on a single Topic. It is typically suggested to assign this to user identities requiring both production and subscription permissions.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Produce

        <br />
      </td>

      <td>
        <ul>
          <li>Create</li>

          <li>Write</li>

          <li>Describe</li>
        </ul>
      </td>

      <td>
        The Produce operation group defines the minimum permissions required for sending messages to a single Topic. It is advised to grant this authority to the user identity responsible for message production.

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Consume

        <br />
      </td>

      <td>
        <ul>
          <li>Read</li>

          <li>Describe</li>
        </ul>
      </td>

      <td>
        The Consume operation group defines the minimum permissions required for subscribing to messages from a single Topic. It is advised to grant this authority to the user identity responsible for message subscription.

        <br />

        <Tip>
          <br />

          If you intend to use a Consumer Group to subscribe to messages, besides granting permissions for the Consume operation group on the Topic, you must also authorize the target Consumer Group.

          <br />
        </Tip>

        <br />
      </td>
    </tr>

    <tr>
      <td>
        Consumer Group

        <br />
      </td>

      <td>
        All

        <br />
      </td>

      <td>
        <ul>
          <li>Delete</li>

          <li>Describe</li>

          <li>Read</li>
        </ul>
      </td>

      <td>
        The All operation group grants complete operational permissions for a specific Consumer Group. It is advised to grant this authority to the user identity responsible for message subscription.

        <br />

        <Tip>
          <br />

          To utilize a Consumer Group for subscribing to messages, besides authorizing the Consumer Group, you must also grant permissions for the Consume operation group on the target Topic.

          <br />
        </Tip>

        <br />
      </td>
    </tr>

    <tr>
      <td>
        TransactionID

        <br />
      </td>

      <td>
        All

        <br />
      </td>

      <td>
        <ul>
          <li>Describe</li>

          <li>Write</li>
        </ul>
      </td>

      <td>
        All operation groups agreed on granting permission to read state and commit transactions for a specific TransactionID. It is recommended to authorize this for user identities that need to send transactional messages.

        <br />
      </td>
    </tr>
  </tbody>
</table>

## Using Kafka ACL

To use the Kafka ACL feature in an AutoMQ instance, follow the process outlined below:

<img />

It is important to note that:

* When creating an instance, ACL is not enabled by default. At this time, clients can access the cluster anonymously via the PLAINTEXT protocol and have permissions for all operations.

* After enabling ACL, the PLAINTEXT protocol will continue to provide services. However, all client applications should promptly switch to SASL\_PLAINTEXT endpoints that support authentication and authorization to prevent unauthorized access. Once the switch is complete, close the PLAINTEXT protocol endpoints.


# Overview
Source: https://docs.automq.com/automq-cloud/manage-security/overview

AutoMQ ensures secure Kafka-compatible cloud-native data links with flexible identity recognition, permission control, and robust encryption for transit and rest.

AutoMQ is a widely-used tool in enterprise core data links for transmitting vital business data. AutoMQ Cloud provides extensive and flexible security configuration options for the data plane, including identity recognition, permission control, and data encryption. This document outlines the security configuration features supported by AutoMQ on the data plane.

<Tip>
  This section only addresses the security configuration of data plane instances. For details on managing member accounts, RBAC, and other aspects related to the BYOC console and the control plane, refer to the documentation [Overview▸](/automq-cloud/manage-identities-and-access/overview).
</Tip>

## Identity Recognition

The AutoMQ data plane (AutoMQ instance) offers Kafka API access capabilities and adheres to the identity recognition protocols provided by the Apache Kafka community, supporting the following identity recognition mechanisms.

| **Recognition Protocol**<br /> | **Description**<br />                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Anonymous Mode**<br />       | <ul><li>**Protocol**: `PLAINTEXT`</li><li>**Description:** Users can access the cluster via the Kafka API without setting access credentials, thus accessing the AutoMQ instance anonymously with full operational privileges.</li></ul><Tip><br />It is not recommended to use anonymous mode in production environments. It is advised to perform strict identity verification using protocols such as SASL or mTLS.<br /></Tip><br />                                                                                                                                    |
| **SASL Mode**<br />            | <ul><li>**Protocols**: `SASL_PLAINTEXT`, `SASL_SSL`</li><li>**Authentication Mechanisms**: `SCRAM-SHA-256`, `SCRAM-SHA-512`, `PLAIN`</li><li>**Description:** When accessing the cluster via the Kafka API, users need to set access credentials to identify the identity principal. This, in conjunction with the Kafka ACL authorization mechanism, verifies the operational permissions of the principal. [Manage Kafka ACLs▸](/automq-cloud/manage-security/manage-kafka-acls)</li></ul>                                                                                |
| **Mutual TLS (mTLS)**<br />    | <ul><li>**Protocols**: `SSL`</li><li>**Authentication Mechanism**: `SSL`</li><li>**Description:** When accessing the cluster via the Kafka API, a unique TLS certificate must be assigned to each client. Each TLS certificate corresponds to an identity principal in the Kafka ACL. After the server performs TLS verification, it identifies the corresponding identity principal and verifies the operational permissions of the principal using the Kafka ACL authorization mechanism. [Manage Kafka ACLs▸](/automq-cloud/manage-security/manage-kafka-acls)</li></ul> |

For configuration and usage methods of identity recognition protocols, refer to the following documentation:

* [SASL Authentication▸](/automq-cloud/manage-security/authentication/sasl-authentication)

* [Mutual TLS (mTLS) Authentication▸](/automq-cloud/manage-security/authentication/mutual-tls-mtls-authentication)

## Access Control

AutoMQ data plane (AutoMQ instance) provides access to Kafka APIs, following the ACL access control protocol established by the Apache Kafka community to offer permission management capabilities.

For configuration and instructions on Kafka ACLs, refer to [Manage Kafka ACLs▸](/automq-cloud/manage-security/manage-kafka-acls).

## Data Protection and Encryption

AutoMQ Cloud provides customers with robust encryption capabilities for data in transit and at rest.

* **Transmission Encryption:** As noted in the referenced document, AutoMQ supports the mTLS protocol. When accessing data using mTLS, the transmission link is encrypted via TLS, ensuring data remains secure and protected against leakage during network transmission.

* **Data Encryption at Rest:** AutoMQ is built upon cloud storage, inherently supporting data storage encryption. This feature can be enabled when creating an installation in the BYOC console and during the creation of an AutoMQ instance. It supports transparent data encryption using cloud vendor-managed keys.

For configuring data encryption at rest, refer to [Data Encryption at Rest▸](/automq-cloud/manage-security/data-encryption-at-rest).


# Executing Migration
Source: https://docs.automq.com/automq-cloud/migrate-to-automq/executing-migration

From Apache Kafka to AutoMQ: Migrate data without loss and maintain consumption progress. Also ensure a smooth transition for producers and consumers, achieving efficient data migration.

With the [Prerequisites▸](/automq-cloud/migrate-to-automq/prerequisites) completed, you can proceed with the subsequent migration process. This document provides a detailed overview of the strategies and implementation procedures for migrating from Apache Kafka to AutoMQ.

Before starting a production migration, review the [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices) and incorporate the applicable endpoint, client cutover, offset, promotion, and rollback checks into your migration runbook.

## Migration plan

Using Kafka Linking to migrate from Apache Kafka® to AutoMQ, the following tasks should be considered:

* **Message data synchronization**: Kafka stores historical data of both consumed and unconsumed messages. Migrating clusters must ensure that message data is replicated to the new cluster as needed, without losing any messages.

* **Producer switching**: In addition to data synchronization, the migration process requires switching producer applications at an appropriate time so that producers connect to the target cluster to produce new messages.

* **Consumer switching**: In addition to data synchronization, the migration process requires switching consumer applications at an appropriate time so that consumers connect to the target cluster and continue consuming messages from the previous consumption progress.

Refer to the following flowchart for the overall migration plan:

<img alt="Overall migration plan flowchart showing message data sync, producer switch, and consumer switch steps" />

## Steps

### Step 1: Create a Kafka Link and start synchronizing data

After completing the [Prerequisites▸](/automq-cloud/migrate-to-automq/prerequisites), the source cluster, target cluster, and the scope of Topics and Consumer Groups to be migrated have been clearly defined. The next step is to start creating the migration task.

Create a **Kafka Link** by clicking on **Target Instance >> Kafka Links** and filling in the relevant parameters as guided.

| **Parameter Name**<br />           | **Description**<br />                                                                                                                                                                                                                                                                                                                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Kafka Link Id**<br />            | <ul><li>Meaning: Used to represent a group of migration tasks, generally recommended to be a short and recognizable name.</li><li>Constraint: Supports only English uppercase and lowercase letters, numbers, hyphens (-), and underscores (\_).</li></ul>                                                                                                                                             |
| **Source Cluster**<br />           | Specify the source cluster to be migrated. Here, fill in the Bootstrap address and ACL identity parameters of the source cluster.<br />                                                                                                                                                                                                                                                                |
| **Target Cluster**<br />           | Specify the target cluster for the migration. Here, select the already created AutoMQ instance.<br />                                                                                                                                                                                                                                                                                                  |
| **Starting Sync Position**<br />   | Specifies the initial synchronization position after the migration task starts. The following options are supported:<br /><ul><li>**From Latest Position:** Kafka Linking only replicates new messages generated subsequently to the target cluster. This mode is suitable for cold read sensitivity scenarios where migration needs to catch up with both clusters from the current moment.</li></ul> |
| **Sync Topic List**<br />          | Select Source Topics from the source cluster. Kafka Linking creates a one-to-one mapped Mirror Topic in the target instance. The target instance must not already contain a Topic with the same name.<br />                                                                                                                                                                                            |
| **Sync Consumer Group List**<br /> | Select several Consumer Groups from the source cluster. Kafka Linking will create a one-to-one mapped Consumer Group on the target instance.<br />                                                                                                                                                                                                                                                     |

The console builds the candidate lists from the source cluster's Kafka APIs. After you enter a search keyword, the console matches resource names by containment. For example, `order` matches `prod-order-v1`. Each search displays up to 100 matching resources. The following filters also apply:

* **Source Topic:** Includes Topics visible to the Kafka Linking source identity that are not identified as internal Topics. Excludes Topics marked as internal in Kafka metadata and Topic names that start with `__` or `.`, or end with `-internal` or `.internal`.
* **Source Consumer Group:** Includes Groups visible to the Kafka Linking source identity whose `protocolType` is `consumer` or empty and whose Group IDs do not match a product-internal prefix. Excludes other protocol types, such as `connect` used by Kafka Connect or `connector` reported by some implementations. It also excludes product-internal Groups whose IDs start with `sys-cmp` or `karapace-autogenerated`.

Appearing in a candidate list means only that the source identity can discover the resource. It does not guarantee successful creation. Conflicts such as a same-name Topic on the target or an existing Mirror resource are not prefiltered from the source candidate list and are still validated during creation. If an expected Topic or Consumer Group does not appear, shorten the search keyword, then check its name, Group protocol type, Group ID prefix, and source Kafka ACLs. Internal Topics and coordination Groups used by components such as Kafka Connect are outside the Kafka Linking business Topic and Consumer Group migration scope; migrate them by following the component-specific procedure. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#verify-the-mirror-topic-and-consumer-group-candidate-lists) for the complete checklist.

<Danger>
  If the target instance already contains a Topic with the same name as a selected Source Topic, Kafka Linking cannot create the corresponding Mirror Topic. Resolve the naming conflict before adding that Source Topic to the Kafka Link.
</Danger>

<img alt="Kafka Link creation form showing source cluster, target cluster, and sync configuration fields" />

Select the target topic and Consumer Group.

<img alt="Topic and Consumer Group selection interface for Kafka Link synchronization" />

After creation, you can enter the Kafka Link details to check that the specified Topic and Group have already entered the **synchronization** status. Once Kafka Link is created, it also supports adding new Topics and Consumer Groups. You can add the required business resources for migration as needed.

<Danger>
  If you delete Mirror Topics or Consumer Groups in Kafka Link, the corresponding Topics and Consumer Groups will be deleted from the target cluster (AutoMQ instance). This operation cannot be undone; you will need to recreate them afterward.
</Danger>

<img alt="Kafka Link details page showing Topic and Consumer Group synchronization status" />

### Step 2: Switch producers and consumers, execute migration process

When the Kafka Link is created, users need to update the Producer and Consumer configurations, switching the Producer and Consumer from the source cluster to the target cluster. This mainly involves the following three operations:

* **Producer Cutover:** Update the producer's access parameters to point to the target cluster.

* **Consumer Cutover:** Update the consumer's access parameters to point to the target cluster.

* **Promote Mirror Topic Status:** In the AutoMQ console, select the Mirror Topic for status promotion. The promotion operation essentially controls the Kafka Linking component to stop proxying Producer write traffic and replicating data from the source cluster.

Specific steps for switching the Producer and Consumer stream can be found below:

#### Phase 1: Switch the producer to the target cluster

During the migration process, first switch the Producer's access configuration so that the Producer connects to the target cluster (AutoMQ instance). The traffic topology during the switching process is shown in the figure below:

<img alt="Traffic topology diagram during Phase 1: producers switching to the target AutoMQ cluster" />

**Operations steps:**

* The producer updates the access parameters in batches and restarts the application, redirecting the production traffic to the designated AutoMQ instance.

<Warning>
  This rolling cutover applies only to producers that do not use Kafka transactions. For applications that set `transactional.id` or depend on exactly-once semantics, stop source transactional producers, wait for the target to catch up, promote the related Mirror Topics, and then start the producers on the target. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#cut-over-transactional-producers) for the full procedure.
</Warning>

**Expected Outcome:**

* Production traffic is migrated to the target instance in phases, ensuring no downtime or disruption in production traffic.

* Consumers in the source cluster continue to consume all messages without interruption.

* Messages in the source cluster are synchronized to AutoMQ using replication tasks.

**Rollback Operation:**

* Producer rollback configuration: switch back to the source cluster.

#### Phase 2: Switch the consumer to the target cluster

The second stage involves switching the Consumer's access configuration, allowing the Consumer to connect to the target cluster (AutoMQ instance). The traffic topology during the switch is shown in the figure below:

<img alt="Traffic topology diagram during Phase 2: consumers switching to the target AutoMQ cluster" />

**Operations steps:**

* Consumers update their connection parameters and use a rolling deployment so that instances gradually connect to the AutoMQ instance.

* For standard Consumers that participate in Group management by using `subscribe` and `group.id`, instances moved to the target cannot receive partition assignments or start consuming while the target Mirror Group is `LINKING`. As source instances leave in batches, the remaining source instances must carry the full consumption workload. Control the rollout batch size and monitor source processing capacity and lag.

* After every source Consumer instance exits, the data plane detects that the source Group has no active members and automatically promotes the Consumer Group. The data plane schedules registered Groups every 10 seconds. If the source Group still had active members at the previous check, the same Group is checked again after a default minimum interval of 30 seconds.

* A brief consumption pause exists until Consumer Group promotion, target rebalance, and partition assignment complete. Wait for the Group to enter `PROMOTED`, then confirm that target Consumers have assignments and their consumption progress is advancing.

<Danger>
  **Before switching Consumers, the following conditions should be met**

  For every partition, confirm that the Consumer's actual startup offset is within the readable range of the current Topic on the AutoMQ instance:

  ```text theme={null}
  target earliest offset <= Consumer actual startup offset <= target end offset
  ```

  For example, if a Consumer's actual startup offset for Partition X is 100, it can cut over when the target readable range is `[80, 150]`. A range of `[120, 150]` means that the historical record at offset 100 is no longer readable. A range of `[80, 90]` means that the target has not replicated through offset 100, so wait for the target to catch up.

  The actual startup offset can come from a committed Kafka Consumer Group offset, a Flink checkpoint or savepoint, or application-managed storage. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#validate-the-consumers-actual-offset-source) for the validation procedure.
</Danger>

**Expected Outcome:**

* Production traffic remains in the same state as the previous phase, continuing to proxy back to the source cluster and synchronize to the target cluster.

* The Consumer Group will continue to consume from the offsets of the source cluster and will not be affected.

**Rollback Operation:**

* For the consumer rollback configuration, switch back to the source cluster.

<Danger>
  Since the consumption offsets of the source cluster will not be updated automatically, it is recommended to reset the offsets before rolling back to avoid potential duplicate consumption.
</Danger>

#### Phase 3: Promote Mirror Topic status

Once all Producers and Consumers have successfully switched and are functioning as expected, you can elevate the status of the Mirror Topic, stop the traffic proxy, and synchronization.

<img alt="AutoMQ console showing the Mirror Topic promotion step" />

**Operations steps:**

* After the Consumer switch is completed, it is recommended to monitor for a certain period to ensure that Producer and Consumer applications are running as expected.

* Once confirmed, in the AutoMQ console, click on "Promote Mirror Topic" and stop traffic proxy and replication.

<Danger>
  **Prerequisites for Promoting Mirror Topics:**

  * Every standard Producer instance connects to the target AutoMQ instance, and no workload Producer still writes directly through a source-cluster endpoint. Kafka Linking itself remains connected to the source during `LINKING`, so source-cluster connections alone do not indicate an incomplete workload cutover.

  * For transactional Producers, source instances have stopped, transactions have ended, the target has caught up, and target transactional Producers have not started.

  * Every Consumer instance connects to the target, the source Consumer Group has no active members, Consumer Group promotion is complete, and target consumers are stable.

  * Replication lag has converged, no sustained network, authentication, or request errors remain, and the post-promotion rollback and reconciliation plan is recorded.

  See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#promote-mirror-topics) for the complete gates and validation procedure.
</Danger>

**Expected Outcome:**

* All read and write operations for mirror topics are centralized in the target cluster, without proxying back to the source cluster.

**Rollback Operation:**

* If the message data in the target cluster exceeds that of the source cluster after promoting the Mirror Topic, you should be cautious about data inconsistencies during the rollback.

### Step 3: Migration complete, delete Kafka Link

After every Mirror Topic and Consumer Group in the Kafka Link is promoted, and production, consumption, and critical business results on the target pass acceptance and the observation window, delete the Kafka Link itself. Deleting the Kafka Link marks the migration as fully complete.

<img alt="AutoMQ console showing the Kafka Link deletion step after migration is complete" />

<Danger>
  To complete the migration, delete only the Kafka Link itself. Do not delete promoted Mirror Topics or Consumer Groups from the Kafka Link; doing so deletes the corresponding topic or Group from the target AutoMQ instance.
</Danger>

After deleting the Kafka Link, stop new workload writes to the source and retain it in an idle state for an observation period, preserving its data and required access. Reclaim the source cluster resources only after the target remains stable and rollback or data reconciliation is no longer needed. See [Kafka Linking best practices](/automq-cloud/migrate-to-automq/kafka-linking-best-practices#complete-the-migration) for the full exit criteria and checklist.


# Kafka Linking best practices
Source: https://docs.automq.com/automq-cloud/migrate-to-automq/kafka-linking-best-practices

Plan and run Kafka Linking migrations with reliable endpoints, safe client cutovers, offset validation, promotion gates, and rollback boundaries.

This guide provides production best practices for migrating Apache Kafka workloads to AutoMQ with Kafka Linking. Complete the [prerequisite review](/automq-cloud/migrate-to-automq/prerequisites), then use this guide to create a runbook for each migration batch.

<Warning>
  Kafka Linking synchronizes Kafka topic data and committed Kafka Consumer Group offsets. It does not rewrite offsets stored in Flink checkpoints, savepoints, application databases, or other external systems. Identify the offset source that each consumer actually uses before cutover.
</Warning>

## Plan each migration batch

Start with a non-critical but representative workload. Expand the scope only after the pilot passes. Keep dependent producers, topics, and consumers in the same batch so that a business data flow is not split across migration stages.

Record the following information before migration:

| Object             | Information to confirm                                                                              |
| ------------------ | --------------------------------------------------------------------------------------------------- |
| Topic              | Partition count, cleanup policy, retention, maximum message size, peak throughput, and dependencies |
| Producer           | Client version, idempotence setting, `transactional.id` usage, instance inventory, and owner        |
| Consumer           | `group.id`, client version, offset source, acceptable replay window, and owner                      |
| External component | Flink, Kafka Connect, Kafka Streams, Schema Registry, and application-managed offset stores         |
| Security           | Listener protocol, credentials, certificate lifetime, Kafka ACLs, firewalls, and security groups    |
| Rollback           | Steps before and after Mirror Topic promotion, stop conditions, and decision owner                  |

Avoid unrelated source-cluster upgrades, broker replacements, network changes, authentication changes, and topic recreation during the migration window. Limiting concurrent changes makes failures easier to attribute.

## Configure reliable source endpoints

A Kafka client uses `bootstrap.servers` to obtain cluster metadata. It then connects to the broker addresses returned through `advertised.listeners`. A successful connection to only the bootstrap address does not prove that Kafka Linking can reach the cluster.

<Warning>
  Enter a stable bootstrap DNS name or the formal endpoint provided by the source Kafka service. Do not use Kubernetes Pod IPs, node IP and NodePort combinations, or other temporary addresses that change during Pod recreation, node replacement, or scaling.
</Warning>

Validate the source connection as follows:

* Use stable DNS names or the source Kafka service's formal endpoint. When you configure multiple endpoints, distribute them across failure domains.
* Confirm that the AutoMQ data plane can resolve and connect to **every** broker address and port returned in metadata.
* For a Kubernetes source cluster, use the Kafka Operator or platform-provided stable bootstrap service and stable per-broker external listeners.
* Allow both the bootstrap endpoints and all advertised broker endpoints through firewalls, security groups, routes, and network access controls.
* For TLS or mTLS, verify the trust chain, the hostname against the certificate SAN, and the certificate lifetime.
* Use a dedicated migration identity with the permissions required to read the selected topics, inspect Consumer Groups, and discover metadata. Do not rotate or delete this identity during migration.

Run a metadata check from an environment with a network path equivalent to the AutoMQ data plane. For example:

```bash theme={null}
kafka-broker-api-versions.sh \
  --bootstrap-server <stable-bootstrap-host-1>:<port>,<stable-bootstrap-host-2>:<port> \
  --command-config source-client.properties
```

The command must return the expected broker list. Then validate DNS resolution, port connectivity, and the TLS handshake for every returned endpoint. Fix `advertised.listeners` or network access before creating the Kafka Link if any broker is unreachable.

## Prepare the source and target clusters

Complete these checks before creating a Kafka Link:

* Reserve enough target capacity for migration traffic as described in [Prerequisites](/automq-cloud/migrate-to-automq/prerequisites).
* Make sure that a topic with the same name does not already exist on the target. Kafka Linking must create a Mirror Topic for the source topic.
* Keep source topic identity and partition structure stable within a migration batch. Do not delete and recreate a topic with the same name.
* Keep target data long enough to cover the migration and observation windows. The target must not expire records that a consumer still needs before cutover.
* Validate Kafka ACLs, certificates, Schema Registry, connectors, and other external dependencies on the target. Do not assume that topic data synchronization migrates every surrounding configuration or state.
* Save a baseline of topic configurations, partition ranges, Consumer Group offsets, and business traffic on both clusters.

### Verify the Mirror Topic and Consumer Group candidate lists

When you create a Mirror Topic or Consumer Group, the console retrieves candidate resources through the source cluster's Kafka APIs. After you enter a search keyword, the console matches resource names by containment. For example, `order` matches `prod-order-v1`. Each search displays up to 100 matching resources. The candidate lists include only resources visible to the Kafka Linking source identity and then apply these filters:

| Resource              | Product filter                                                                                                                                                                                                                                                                                   |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Source Topic          | Excludes Topics marked as internal in Kafka metadata. It also excludes Topic names that start with `__` or `.`, or end with `-internal` or `.internal`                                                                                                                                           |
| Source Consumer Group | Includes only Groups whose `protocolType` is `consumer` or empty. It excludes other protocol types, such as `connect` used by Kafka Connect or `connector` reported by some implementations. It also excludes product-internal Groups whose IDs start with `sys-cmp` or `karapace-autogenerated` |

If an item in the migration inventory does not appear, check the following in order:

1. Shorten the search keyword, confirm that the resource name matches by containment, and check whether the result falls outside the 100 items displayed per search.
2. Confirm that the name or Group ID does not match an internal-resource rule above.
3. For a Consumer Group, use a Kafka administration tool to inspect its `protocolType`. A Group with a non-Consumer protocol cannot be created as a Kafka Linking Consumer Group.
4. Confirm that the Kafka Linking source identity has the Kafka ACLs required to discover the Topic or Consumer Group. Resources hidden from that identity do not appear in the candidate lists.
5. Separately check whether the target already has a Topic with the same name and whether the resource is already a Mirror resource. These conflicts are not prefiltered from the source candidate list, but they can cause creation to fail.

Internal Topics, coordination Groups, and external state used by components such as Kafka Connect are outside the business Topic and Consumer Group migration scope supported by these candidate lists. Follow the component-specific migration procedure instead of renaming resources to bypass the product filters.

<Danger>
  From the time a Mirror Topic is created until it is promoted to `PROMOTED`, do not increase the partition count of either the source topic or the target Mirror Topic. The source and target partition counts must remain unchanged while Kafka Linking is in progress.
</Danger>

After the Kafka Link enters the linking state, monitor replication lag and errors. Start client cutover only when lag converges and no sustained authentication, network, or request errors remain.

<Danger>
  Keep the source cluster, its network path, DNS endpoints, and migration identity available until every Mirror Topic and Consumer Group is promoted and business acceptance passes. Do not stop, scale to zero, delete, or release the source cluster early, and do not revoke Kafka Linking's access permissions.
</Danger>

## Cut over standard producers and consumers

This workflow applies to producers that do not use Kafka transactions. Producers with a `transactional.id` must use the [transactional producer workflow](#cut-over-transactional-producers).

1. Move standard producers to the target AutoMQ instance in batches, then recreate or restart the clients. While a Mirror Topic is `LINKING`, Kafka Linking routes these writes to the source and replicates them back to the target.
2. Check every producer deployment and send callback. Confirm that no untracked producer instance continues to write directly to the source.
3. Before moving a Consumer Group, confirm that the actual startup offset for every partition is within the target topic's readable range. See [Validate the consumer's actual offset source](#validate-the-consumers-actual-offset-source) for the offset sources used by different clients.
4. Change the Consumer connection configuration and perform a rolling deployment so that instances connect to the target AutoMQ instance in batches. For standard Consumers that participate in Group management by using `subscribe` and `group.id`, target instances cannot complete JoinGroup or receive partition assignments while the target Mirror Group is `LINKING`, so they do not start consuming. The source instances that have not yet moved continue to carry the consumption workload.
5. Control the rollout batch size and continuously monitor source Consumer capacity and lag. As source instances leave in batches, effective source consumption capacity decreases. Make sure that the remaining source instances can carry the full consumption workload at that point.
6. Confirm that every source Consumer instance has exited. After Kafka Linking detects that the source Consumer Group has no active members, it automatically promotes the target Consumer Group. A brief consumption pause exists until promotion completes.
7. Wait for the Consumer Group to enter `PROMOTED`. Confirm that target Consumers successfully join the Group, receive partition assignments, and resume consumption. Validate consumption rate, lag, business results, and errors, and complete the observation window before moving the next workload.

<Warning>
  The rolling cutover must converge with every instance connected to the target. Do not run the same `group.id` on the source and target for an extended period. The Consumer Group cannot be automatically promoted while the source Group still has active members. A target Consumer process being started or connected to a broker does not mean that it is consuming. Treat the cutover as complete only after the Group is `PROMOTED`, target Consumers have partition assignments, and consumption progress is advancing.
</Warning>

### Understand Consumer Group auto-promotion and the consumption pause

Data-plane auto-promotion uses the following flow:

1. When a target Consumer attempts to join a Mirror Group that is still `LINKING`, the data plane registers the Group for auto-promotion checks. This JoinGroup attempt receives a retriable error, so the target Consumer temporarily has no partition assignment.
2. The data plane queries the source Group. It attempts auto-promotion only when the source Group does not exist, or when it has no active members and its state is `EMPTY` or `DEAD`. If source members remain, the target Group stays `LINKING` and target Consumers remain unable to join.
3. After the condition is met, Kafka Linking fetches the source Group's committed offsets, validates that they are within the readable ranges of the target topics, commits them to the target Group, and changes the Group to `PROMOTED`.
4. The target Consumer retries according to its client configuration. It resumes consumption on AutoMQ only after it joins the Group, completes the rebalance, and receives partition assignments.

The data plane schedules registered Groups every 10 seconds. After the first JoinGroup registration, a source check can normally start on the next scheduler run. If the source Group still has active members at that check, subsequent source-state checks for the same Group are eligible at a default minimum interval of 30 seconds. Therefore, consumption does not resume immediately after the last source instance exits. The workflow must still wait for the next eligible source check, offset synchronization and validation, Group promotion, and the client retry and rebalance. Plan for a consumption pause measured in tens of seconds. Network, authentication, out-of-range offsets, or promotion failures can extend this window.

## Cut over transactional producers

Kafka transactions cannot use the rolling proxy workflow for standard producers. Use the following order for any application that sets `transactional.id` or depends on exactly-once semantics:

1. Keep the transactional producer on the source while target replication lag converges.
2. Stop transactional producers on the source. Confirm that every open transaction has committed or aborted.
3. Compare source and target end offsets for every partition. Confirm that the target has caught up with the source.
4. Promote every Mirror Topic involved in the transactions and wait for the `PROMOTED` state.
5. Confirm that old producer instances cannot restart. Then start the transactional producers on the target.
6. Use `read_committed` consumers to verify committed records from before and after promotion, and validate the end-to-end transactional result.

<Warning>
  Do not point a transactional producer at the target while its Mirror Topic is still `LINKING`. For transactions across multiple topics, place all involved topics in one migration batch and stop, catch up, and promote them together.
</Warning>

## Validate the consumer's actual offset source

The consumer startup position depends on how the client manages state:

| Offset source                          | Kafka Linking behavior                                                                    | Pre-cutover check                                                                                                                                                            |
| -------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Committed Kafka Consumer Group offset  | After all source instances exit, Kafka Linking synchronizes and promotes the Group offset | Confirm that the source Group committed offset for every partition is within the target readable range, and ensure that the rollout eventually removes every source instance |
| Flink checkpoint or savepoint          | Kafka Linking does not rewrite Flink state                                                | Inspect the restored offset for every partition                                                                                                                              |
| Application-managed offset             | Kafka Linking does not rewrite a database, Redis, file, or business snapshot              | Identify the source of truth and inspect the offset passed to `seek`                                                                                                         |
| Kafka Connect or other component state | State can reside in internal topics or an external store                                  | Follow the component's migration procedure instead of checking only a business Consumer Group                                                                                |

For every topic-partition, verify:

```text theme={null}
target earliest offset <= client startup offset <= target end offset
```

Pause the consumer cutover if any partition fails this check. Options include continuing on the source until the checkpoint advances, changing the synchronization or retention range, or resetting client state after assessing the business impact.

For example, if a Consumer's actual startup offset for Partition X is 100, it can cut over when the target readable range is `[80, 150]`. A range of `[120, 150]` means that the historical record at offset 100 is no longer readable. A range of `[80, 90]` means that the target has not replicated through offset 100, so wait for the target to catch up.

<Note>
  `auto.offset.reset` applies only when the client has no valid starting offset. It does not override a Flink restore or an explicit `assign` and `seek` from an external offset store.
</Note>

## Promote Mirror Topics

Promotion stops write routing and replication from the source for a topic. Pass every gate before promotion:

| Gate                      | Pass condition                                                                                                                   |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Standard producers        | Every workload Producer instance connects to the target, and no instance still writes directly through a source-cluster endpoint |
| Transactional producers   | Source instances have stopped, transactions have ended, the target has caught up, and target instances have not started          |
| Consumer Groups           | Source Groups have no active members, and target consumers run normally                                                          |
| External-offset consumers | Every partition startup offset is readable on the target                                                                         |
| Kafka Link                | Replication lag has converged, with no sustained network, authentication, or request errors                                      |
| Business acceptance       | Produce, consume, lag, error rate, and critical business results meet the acceptance criteria                                    |
| Rollback readiness        | The rollback owner, reconciliation method, and stop conditions are recorded                                                      |

Kafka Linking remains connected to the source and routes writes during `LINKING`, so source-cluster connections alone do not indicate that workload Producers have not switched. For a low-frequency topic, do not use a short zero-traffic window as the only proof that a Producer has stopped. Check the workload Producer inventory and deployment configuration, and compare source and target write requests over a window longer than the normal message interval.

Promote topics by business batch instead of promoting the entire migration at once. Validate each batch before continuing.

<Note>
  During Mirror Topic promotion, a small number of writes that still reach the old routing path can briefly receive `OUT_OF_ORDER_SEQUENCE_NUMBER`. For non-transactional producers with idempotence enabled and normal retry settings, Kafka clients typically reset the sequence state and retry automatically, so applications usually do not need to intervene. Monitor final send failures. Investigate client configuration, Topic state, and residual routed traffic only if the error is ultimately returned to the application or continues after promotion; compensate only records confirmed as failed according to the application's idempotency policy.
</Note>

## Define rollback boundaries

Rollback semantics change after promotion:

* **Before promotion:** Standard producers can return to the source. Before moving consumers back, reconcile the source Consumer Group offset or expect records already processed on the target to be replayed.
* **After promotion:** New target records no longer replicate to the source. Changing only `bootstrap.servers` would return clients to a source cluster that is missing post-promotion records. Stop the cutover and use a data reconciliation or reverse-migration plan.

## Complete the migration

End a Kafka Linking migration only after all of these conditions are met:

* Every Mirror Topic in the Kafka Link is `PROMOTED`.
* Every Consumer Group in the Kafka Link has completed promotion.
* Production, consumption, lag, error rate, and critical business results on the target AutoMQ instance pass acceptance, including the agreed observation window.

After all conditions pass, delete the **Kafka Link itself** in AutoMQ Console. Deleting the Kafka Link marks the Kafka Linking migration as fully complete.

<Danger>
  Do not delete the promoted Mirror Topics or Consumer Groups one by one. Deleting these resources deletes the corresponding topic or Group from the target AutoMQ instance and can cause message data or consumer offsets to be lost, or interrupt the workload. To complete the migration, delete only the Kafka Link itself.
</Danger>

After deleting the Kafka Link, retain the source cluster in an idle state for an observation period. Stop new workload writes, but preserve the original data and required access. Reclaim the source cluster's network, storage, compute resources, and migration identity only after the target remains stable and rollback or data reconciliation is no longer needed. See [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) for the operational steps.

## Use the migration checklist

Keep at least the following evidence for each batch:

* Stable source bootstrap endpoints and connectivity results for every advertised broker endpoint.
* Migration identity permissions, certificate lifetime, and firewall approvals.
* Topic, producer, Consumer Group, external offset source, owner, and dependency inventories.
* Partition earliest offsets, end offsets, committed Group offsets, and replication lag before and after cutover.
* The selected standard or transactional producer cutover order.
* Evidence that source Groups became `Empty`, target consumers stabilized, and business acceptance passed.
* Mirror Topic promotion approval, execution time, observation result, and rollback boundary.
* Promotion results for every Mirror Topic and Consumer Group, the Kafka Link deletion record, and approval to reclaim the source cluster.

After completing these checks, follow [Executing migration](/automq-cloud/migrate-to-automq/executing-migration) to create the Kafka Link in AutoMQ Console and run the migration.


# Integrating Metrics with Datadog
Source: https://docs.automq.com/automq-cloud/monitoring-alert/integrating-metrics-with-datadog

Send AutoMQ metrics to Datadog using a Gateway-style Datadog Agent or OTEL Collector.

AutoMQ supports exporting metrics to [Datadog](https://www.datadoghq.com/) for monitoring and alerting. This guide covers two Gateway-style integration approaches based on where the collector runs:

* **Kubernetes**: Use the Datadog Agent Helm chart's OTEL Agent Gateway to scrape Prometheus metrics from AutoMQ pods.
* **Linux**: Use a Gateway OTEL Collector or Datadog DDOT Collector on Linux to scrape AutoMQ Prometheus metrics endpoints and forward metrics to Datadog.

## Prerequisites

* A running AutoMQ Cloud BYOC cluster. BYOC clusters expose the AutoMQ Prometheus metrics endpoint by default.
* A valid [Datadog API Key](https://docs.datadoghq.com/account_management/api-app-keys/)
* Your Datadog Site URL (e.g., `datadoghq.com`, `us3.datadoghq.com`, `datadoghq.eu`)

## Option 1: Kubernetes with Datadog Agent Gateway

In this approach, the Datadog Agent Helm chart runs a single OTEL Agent Gateway Deployment. The Gateway scrapes Prometheus metrics from AutoMQ pods using Kubernetes service discovery, reduces metric cardinality before ingest, then exports the metrics to Datadog.

```text theme={null}
AutoMQ Pods (Prometheus metrics endpoint on port 9090)
    ↓  Prometheus scrape (Kubernetes service discovery)
Datadog OTEL Agent Gateway
    ↓  Export to Datadog
Datadog
```

### Step 1: Confirm AutoMQ metrics export

<Info>
  AutoMQ Cloud BYOC Kubernetes clusters created by the Console expose Prometheus metrics on the AutoMQ broker and controller pods by default. No AutoMQ Helm values changes are required in the standard BYOC flow.

  If you manage an AutoMQ Software deployment on Kubernetes and need to manually enable metrics export, see [Appendix: Manually enable AutoMQ metrics export for Kubernetes Software](#appendix-manually-enable-automq-metrics-export-for-kubernetes-software).
</Info>

### Step 2: Create the Datadog Agent Gateway values file

Create a `datadog-values.yaml` file with the OTEL Agent Gateway configuration. The Gateway uses Kubernetes service discovery to find AutoMQ broker and controller pods.

Set `<your-automq-namespace>` to the Kubernetes namespace where your AutoMQ cluster runs. If you need to collect multiple AutoMQ namespaces with the same Datadog Agent, add each namespace to `kubernetes_sd_configs.namespaces.names`.

The same collection profile can be used for AutoMQ Software and AutoMQ Cloud BYOC/CMP clusters. The standard dashboards show `cluster_id` as the required cluster filter; this dashboard filter reads the Datadog `service` tag derived from AutoMQ's Prometheus `job` label. The dashboards also include `env_id` and `instance_id` as optional filters when those labels are exposed by the AutoMQ metrics. The profile does not synthesize `env_id` or `instance_id` from Kubernetes metadata. If AutoMQ metrics already expose those labels, `honor_labels: true` preserves the AutoMQ-provided values.

```yaml theme={null}
agents:
  enabled: false
clusterAgent:
  enabled: false
  admissionController:
    enabled: false
datadog:
  kubeStateMetricsCore:
    enabled: false
  orchestratorExplorer:
    enabled: false
  operator:
    enabled: false
  instrumentationCrd:
    enabled: false
  autoscaling:
    workload:
      enabled: false
datadog-crds:
  crds:
    datadogMetrics: false
    datadogPodAutoscalers: false
    datadogPodAutoscalerClusterProfiles: false
    datadogInstrumentations: false
otelAgentGateway:
  enabled: true
  replicas: 1
  rbac:
    create: true
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "list", "watch"]
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "automq"
              scrape_interval: 15s
              honor_labels: true
              kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - <your-automq-namespace>
                  selectors:
                    - role: pod
                      label: 'app.kubernetes.io/name=automq-enterprise,app.kubernetes.io/component in (broker,controller)'
                      field: 'status.phase=Running'
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_container_name]
                  action: keep
                  regex: 'automq-(broker|controller)'
                - source_labels: [__meta_kubernetes_pod_container_port_number]
                  action: keep
                  regex: "9090"
              metric_relabel_configs:
                - source_labels: [__name__]
                  action: keep
                  regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$"
                - action: labeldrop
                  regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$"
    exporters:
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: ${env:DD_SITE}
    processors:
      cumulativetodelta/automq_counters:
        initial_value: auto
        include:
          match_type: strict
          metrics:
            - kafka_network_io_bytes_total
            - kafka_message_count_total
            - kafka_topic_request_count_total
            - kafka_broker_network_io_bytes_total
            - kafka_request_error_count_total
      transform/automq_rollups:
        error_mode: ignore
        metric_statements:
          - context: metric
            statements:
              - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size"
              - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset"
              - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total"
              - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total"
              - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total"
              - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total"
              - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total"
              - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count"
              - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count"
              - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count"
              - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count"
              - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count"
              - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count"
              - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count"
              - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count"
              - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset"
              - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count"
              - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds"
              - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds"
          - context: datapoint
            statements:
              - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil
              - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil
          - context: metric
            statements:
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag"
      filter/drop_raw_replaced_metrics:
        metrics:
          metric:
            - 'name == "kafka_log_size"'
            - 'name == "kafka_log_end_offset"'
            - 'name == "kafka_network_io_bytes_total"'
            - 'name == "kafka_message_count_total"'
            - 'name == "kafka_broker_network_io_bytes_total"'
            - 'name == "kafka_topic_request_count_total"'
            - 'name == "kafka_request_error_count_total"'
            - 'name == "kafka_stream_topic_partition_count"'
            - 'name == "kafka_group_count"'
            - 'name == "kafka_group_stable_count"'
            - 'name == "kafka_group_empty_count"'
            - 'name == "kafka_group_dead_count"'
            - 'name == "kafka_group_preparing_rebalance_count"'
            - 'name == "kafka_group_completing_rebalance_count"'
            - 'name == "kafka_group_commit_offset"'
            - 'name == "consumer_offset_lag_count"'
            - 'name == "consumer_estimated_time_lag_milliseconds"'
      infraattributes:
        cardinality: 2
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, infraattributes]
          exporters: [datadog]
```

This example uses the following components:

| Component                                     | Purpose                                                                                                                                                                                                                                                                                                                                                                          |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `otelAgentGateway`                            | Runs one OTEL Agent Gateway Deployment for this collection profile. A single Gateway performs the scrape and rollup before exporting metrics to Datadog.                                                                                                                                                                                                                         |
| `prometheus` receiver                         | Uses Kubernetes service discovery to find running AutoMQ broker and controller pods in the selected namespace and scrape port `9090`. `honor_labels: true` preserves AutoMQ-provided labels such as Prometheus `job`, `env_id`, and `instance_id`. The container-name relabel rule keeps only AutoMQ broker/controller containers and excludes sidecars such as Schema Registry. |
| `metric_relabel_configs`                      | Keeps only the exact raw metrics required by the standard-minimal dashboards or by the `automq_*` rollup processor. The profile does not copy `job` to `cluster_id`; Datadog maps the Prometheus `job` label to the `service` tag, and dashboards keep the `cluster_id` filter name while reading values from `service`.                                                         |
| `cumulativetodelta/automq_counters` processor | Converts selected Prometheus cumulative counters to deltas before rollup. This avoids aggregating cumulative counter values from multiple AutoMQ nodes before a rate is calculated.                                                                                                                                                                                              |
| `transform/automq_rollups` processor          | Creates lower-cardinality `automq_*` rollup metrics before Datadog ingest. This profile uses Collector-side OTTL aggregation.                                                                                                                                                                                                                                                    |
| `filter/drop_raw_replaced_metrics` processor  | Drops the raw source metrics after their rollup metrics have been created, reducing Datadog custom metric volume while keeping the customer-facing dashboard panels functional.                                                                                                                                                                                                  |
| `datadog` exporter                            | Sends metrics to Datadog. The API key and site come from the Agent's environment variables.                                                                                                                                                                                                                                                                                      |
| `infraattributes` processor                   | Optional. Adds Datadog infrastructure tags when the required resource attributes are available.                                                                                                                                                                                                                                                                                  |

#### Standard dashboard collection profile

The recommended configuration is designed for the AutoMQ customer-facing Datadog dashboards. It keeps only the minimum dimensions needed by those dashboards:

| Metric area                                        | Retained dimensions                             |
| -------------------------------------------------- | ----------------------------------------------- |
| Cluster health and capacity                        | `service`                                       |
| CPU and broker metadata                            | `service`, `broker_id`                          |
| Requests, errors, latency, and broker traffic      | `service`, `topic`                              |
| Topic throughput, message count, size, and offsets | `service`, `topic`                              |
| Topic partition distribution across racks          | `service`, `topic`, `rack`                      |
| Consumer group lag, offset, and throughput         | `service`, `consumer_group` or `group`, `topic` |

The standard-minimal profile does not keep raw partition-level series, topic-by-broker traffic / offset / size series, broker/node-level drilldown for JVM, connection, request, latency, or traffic series, `consumer_group * topic * broker` series, non-dashboard `kafka_stream_*` internal metrics, Kafka Linking metrics, or Detailed Metrics-only internal diagnostic dimensions.

<Info>
  This example is the recommended standard-minimal collection profile for AutoMQ customer-facing dashboards. It reduces Datadog custom metric volume and cardinality by collecting only the exact raw metrics required by the standard AutoMQ dashboards or rollup processor, converting request, traffic, topic, partition, and consumer group sources to lower-cardinality `automq_*` rollups, and dropping the raw sources after the rollups are created. The standard-minimal profile keeps cluster, topic, consumer group, and rack dimensions, plus CPU and broker metadata, but it does not keep partition-level series, topic-by-broker series, broker/node-level drilldown series, or consumer-group/topic/broker series.
</Info>

<Info>
  This example includes a Prometheus scrape job and a metrics pipeline for AutoMQ. It also uses the optional `infraattributes` processor to add Datadog infrastructure tags when the required resource attributes are available. The OTLP receiver and other pipelines (traces, logs) are omitted. If you need to collect OTLP data from other applications through the same Collector, refer to the [Datadog OTEL Agent Gateway documentation](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes/?tab=helm) to add the corresponding receivers and pipelines.
</Info>

If your AutoMQ nodes run on dedicated Kubernetes nodes with taints, add tolerations so the OTEL Agent Gateway can schedule on an allowed node. Replace the following taint key, value, operator, and effect with the actual taint settings used by your Kubernetes nodes:

```yaml theme={null}
otelAgentGateway:
  tolerations:
    - key: "<your-taint-key>"
      operator: "Equal"
      value: "<your-taint-value>"
      effect: "NoSchedule"
```

### Step 3: Deploy the Datadog Agent

Add the Datadog Helm repository and create a Kubernetes secret for your API key:

```bash theme={null}
helm repo add datadog https://helm.datadoghq.com
helm repo update
```

```bash theme={null}
kubectl create namespace datadog
kubectl -n datadog create secret generic datadog-secret \
  --from-literal api-key=<your-datadog-api-key>
```

Install the Datadog Agent using the values file from Step 2:

```bash theme={null}
helm install datadog-agent datadog/datadog \
  -f datadog-values.yaml \
  -n datadog \
  --set datadog.apiKeyExistingSecret=datadog-secret \
  --set datadog.site=<your-datadog-site>
```

Replace `<your-datadog-site>` with your Datadog Site (e.g., `datadoghq.com`, `us3.datadoghq.com`, `datadoghq.eu`).

### Step 4: Verify the integration

Verify that the Datadog Agent service account can discover pods:

```bash theme={null}
kubectl auth can-i list pods \
  --as=system:serviceaccount:datadog:datadog-agent \
  --all-namespaces
```

Verify that the OTEL Agent Gateway Deployment is running:

```bash theme={null}
kubectl -n datadog rollout status deployment/datadog-agent-datadog-otel-agent-gateway
```

Check the OTEL Collector logs and confirm that the AutoMQ scrape job has been added:

```bash theme={null}
kubectl -n datadog logs deployment/datadog-agent-datadog-otel-agent-gateway --since=10m | grep -i automq
```

Then verify metrics in Datadog:

1. Open Datadog and go to **Metrics > Explorer**.
2. Search for AutoMQ metrics by typing a metric name prefix such as `kafka_broker_active_count`.
3. Filter by `service:<your-cluster-id>` to confirm that the metrics belong to the expected AutoMQ cluster. In the imported dashboards, the filter is still named `cluster_id`, but it reads values from the Datadog `service` tag. If your AutoMQ environment exposes native `env_id` or `instance_id` labels, you can use them as additional filters.
4. New metrics may take 1–3 minutes to appear.

<img alt="AutoMQ metrics visible in Datadog Metrics Explorer after Kubernetes integration" />

<img alt="AutoMQ metrics detail view in Datadog Metrics Explorer" />

### Step 5: Import AutoMQ Datadog dashboards

After AutoMQ metrics appear in Datadog, download the Datadog dashboard JSON templates and import them into your Datadog organization.

| Dashboard        | JSON template                                                                                                         |
| ---------------- | --------------------------------------------------------------------------------------------------------------------- |
| Cluster Overview | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-cluster-overview.datadog.json) |
| Topic Metrics    | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-topic-metrics.datadog.json)    |
| Group Metrics    | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-group-metrics.datadog.json)    |

To import a dashboard, open Datadog, go to **Dashboards**, create or open a dashboard, choose the dashboard JSON import option, and paste or upload the downloaded JSON content.

<Info>
  These dashboard templates are designed for the metric names, tags, and collection reduction profile shown in Step 2. Cluster, Topic, and Group views use the `automq_*` rollup metrics created by the OTEL transform processor. The standard dashboards show `cluster_id` as the required filter, mapped to the Datadog `service` tag, and expose `env_id` and `instance_id` as optional filters when those labels are present. The AutoMQ Version value should be discovered from the node metadata gauges (`kafka_node_metadata`, `kafka_node_info`, or `kafka_node`) and rendered by the dashboard generation/import workflow. Do not derive it from the Helm Chart/App version label because that label can differ from the runtime image version.
</Info>

<Warning>
  AutoMQ metrics use Prometheus naming conventions with underscores (e.g., `kafka_broker_active_count`). Datadog's built-in Apache Kafka dashboards rely on the Datadog Kafka integration, which uses dot-separated names (e.g., `kafka.request.time`). As a result, AutoMQ metrics do not appear in those built-in dashboards. Use Datadog Metrics Explorer to query AutoMQ metrics directly, or build custom dashboards using the underscore-separated names. For available metrics, see [Prometheus Metrics](/automq/observability/prometheus-metrics).
</Warning>

The screenshots below illustrate this naming difference — Datadog's built-in dashboard uses dot-separated format, while AutoMQ metrics use underscore-separated format:

<img alt="Datadog built-in dashboard showing dot-separated metric naming convention" />

<img alt="AutoMQ metrics showing underscore-separated Prometheus naming convention" />

## Option 2: Linux with Gateway OTEL Collector

In this approach, one OTEL Collector or Datadog DDOT Collector runs as a Gateway on Linux and scrapes all AutoMQ Prometheus metrics endpoints with static targets. Use the same standard-minimal metric reduction profile as the Kubernetes approach. The Linux-specific difference is service discovery: use a static target list instead of Kubernetes service discovery.

```text theme={null}
AutoMQ Prometheus metrics endpoints
    ↓  Prometheus scrape from /metrics
Gateway OTEL Collector / DDOT Collector on Linux
    ↓  Forward to Datadog
Datadog
```

### Step 1: Confirm AutoMQ metrics export

<Info>
  AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. No `server.properties` changes are required in the standard BYOC flow.
</Info>

### Step 2: Install the Collector

Choose one Collector distribution and keep the configuration path and lifecycle commands aligned with that choice.

<Tabs>
  <Tab title="Datadog DDOT Collector">
    Install the Datadog Agent with the embedded DDOT Collector. The official Linux install flow manages the Collector through `datadog-agent` and uses `/etc/datadog-agent/otel-config.yaml` as the OpenTelemetry Collector configuration file.

    ```bash theme={null}
    DD_API_KEY="<your-api-key>" \
    DD_SITE="<your-datadog-site>" \
    DD_OTELCOLLECTOR_ENABLED=true \
    DD_AGENT_MAJOR_VERSION=7 \
    bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

    sudo datadog-agent status
    ```

    If the Datadog Agent is already installed, follow the [Datadog DDOT Collector documentation](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/linux/) for the matching Agent version.
  </Tab>

  <Tab title="Upstream otelcol-contrib">
    Install an OpenTelemetry Collector distribution that includes the Prometheus receiver, Datadog exporter, transform processor, filter processor, cumulative-to-delta processor, memory limiter processor, batch processor, and health check extension. The examples below use `/etc/otelcol/config.yaml` and a custom `otelcol` systemd service.

    Confirm the Collector binary is available:

    ```bash theme={null}
    <otel-collector-binary> --version
    ```
  </Tab>
</Tabs>

### Step 3: Configure the Collector

Create the Collector configuration file. The following configuration is the Linux version of the standard-minimal profile: it uses static Prometheus scrape targets, keeps only the raw metrics needed by the standard dashboards or rollup processor, creates lower-cardinality `automq_*` rollup metrics, and drops the raw source metrics after the rollups are created.

Run this configuration once per AutoMQ cluster from a Collector host that can reach every AutoMQ metrics endpoint. Do not deploy the same profile on every AutoMQ node with only `127.0.0.1:9090` as the target, because local collectors would roll up counters independently before exporting to Datadog.

Use the configuration path for your selected Collector:

* Datadog DDOT Collector: `/etc/datadog-agent/otel-config.yaml`
* Upstream otelcol-contrib: `/etc/otelcol/config.yaml`

```bash theme={null}
COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector
# COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml"          # Upstream otelcol-contrib

sudo mkdir -p "$(dirname "$COLLECTOR_CONFIG_PATH")"
```

```bash theme={null}
COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector
# COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml"          # Upstream otelcol-contrib

sudo tee "$COLLECTOR_CONFIG_PATH" > /dev/null << 'EOF'
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "<your-cluster-id>"
          scrape_interval: 15s
          honor_labels: true
          static_configs:
            - targets:
                - "<automq-node-1-host>:9090"
                - "<automq-node-2-host>:9090"
                - "<automq-node-3-host>:9090"
              labels:
                env_id: "<your-env-id>"
                instance_id: "<your-instance-id>"
          metric_relabel_configs:
            - source_labels: [__name__]
              action: keep
              regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$"
            - action: labeldrop
              regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$"
exporters:
  datadog:
    api:
      # Upstream otelcol-contrib: keep these environment references and set DD_API_KEY/DD_SITE before starting the service.
      # Datadog DDOT Collector: either set these environment variables for datadog-agent, or remove this api block to use the core Agent configuration.
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
  batch:
    timeout: 10s
    send_batch_size: 10000
  cumulativetodelta/automq_counters:
    initial_value: auto
    include:
      match_type: strict
      metrics:
        - kafka_network_io_bytes_total
        - kafka_message_count_total
        - kafka_topic_request_count_total
        - kafka_broker_network_io_bytes_total
        - kafka_request_error_count_total
  transform/automq_rollups:
    error_mode: ignore
    metric_statements:
      - context: metric
        statements:
          - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size"
          - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset"
          - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total"
          - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total"
          - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total"
          - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total"
          - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total"
          - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count"
          - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count"
          - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count"
          - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count"
          - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count"
          - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count"
          - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count"
          - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count"
          - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset"
          - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count"
          - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds"
          - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds"
      - context: datapoint
        statements:
          - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil
          - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil
      - context: metric
        statements:
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag"
  filter/drop_raw_replaced_metrics:
    metrics:
      metric:
        - 'name == "kafka_log_size"'
        - 'name == "kafka_log_end_offset"'
        - 'name == "kafka_network_io_bytes_total"'
        - 'name == "kafka_message_count_total"'
        - 'name == "kafka_broker_network_io_bytes_total"'
        - 'name == "kafka_topic_request_count_total"'
        - 'name == "kafka_request_error_count_total"'
        - 'name == "kafka_stream_topic_partition_count"'
        - 'name == "kafka_group_count"'
        - 'name == "kafka_group_stable_count"'
        - 'name == "kafka_group_empty_count"'
        - 'name == "kafka_group_dead_count"'
        - 'name == "kafka_group_preparing_rebalance_count"'
        - 'name == "kafka_group_completing_rebalance_count"'
        - 'name == "kafka_group_commit_offset"'
        - 'name == "consumer_offset_lag_count"'
        - 'name == "consumer_estimated_time_lag_milliseconds"'
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, batch]
      exporters: [datadog]
EOF
```

<Info>
  With `honor_labels: true`, if the scraped AutoMQ endpoint already exposes a Prometheus `job` label, that target-provided value is preserved and `job_name` is not used for that label. Standard AutoMQ metrics normally expose `job` as the cluster identifier, and Datadog maps that label to the `service` tag used by the dashboard `cluster_id` filter. If the endpoint does not expose `job`, `job_name` becomes the fallback value. The static `env_id` and `instance_id` labels are optional; set them only when you want those Datadog dashboard filters to have values for this Linux collection path.
</Info>

<Info>
  This Gateway profile performs rollups in the Collector to reduce Datadog custom metrics cardinality. It is not equivalent to a Prometheus recording rule and cannot express `max by (..., partition)` across scrape targets followed by `sum by (...)`. If you need strict deduplication when the same partition gauge is temporarily exposed by multiple AutoMQ nodes during partition migration, add a Prometheus recording-rule / federation layer before Datadog, or keep the partition dimension and handle it explicitly in Datadog queries.
</Info>

### Step 4: Start the Collector

<Tabs>
  <Tab title="Datadog DDOT Collector">
    Restart the Datadog Agent and verify the embedded Collector status:

    ```bash theme={null}
    sudo systemctl restart datadog-agent
    sudo datadog-agent status
    ```
  </Tab>

  <Tab title="Upstream otelcol-contrib">
    For a foreground smoke test, run the Collector directly:

    ```bash theme={null}
    DD_API_KEY="<your-api-key>" \
    DD_SITE="<your-datadog-site>" \
    <otel-collector-binary> --config /etc/otelcol/config.yaml
    ```

    For a systemd deployment, create a dedicated service user and environment file:

    ```bash theme={null}
    sudo useradd --system --home /var/lib/otelcol --shell /usr/sbin/nologin otelcol || true
    sudo install -d -o otelcol -g otelcol /var/lib/otelcol

    sudo tee /etc/default/otelcol > /dev/null << 'EOF'
    DD_API_KEY=<your-api-key>
    DD_SITE=<your-datadog-site>
    EOF
    ```

    Limit access to this file because it contains your Datadog API key. Also avoid pasting real keys into shared shell history or logs.

    ```bash theme={null}
    sudo chmod 600 /etc/default/otelcol
    ```

    ```bash theme={null}
    sudo tee /etc/systemd/system/otelcol.service > /dev/null << 'EOF'
    [Unit]
    Description=OpenTelemetry Collector for AutoMQ Datadog metrics
    After=network-online.target
    Requires=network-online.target

    [Service]
    User=otelcol
    Group=otelcol
    EnvironmentFile=/etc/default/otelcol
    ExecStart=<otel-collector-binary-path> --config /etc/otelcol/config.yaml
    Restart=always
    RestartSec=5
    NoNewPrivileges=true
    ProtectSystem=full
    ProtectHome=true
    PrivateTmp=true
    StateDirectory=otelcol

    [Install]
    WantedBy=multi-user.target
    EOF

    sudo systemctl daemon-reload
    sudo systemctl enable otelcol
    sudo systemctl start otelcol
    ```
  </Tab>
</Tabs>

### Step 5: Verify

Open Datadog **Metrics > Explorer**, search for AutoMQ metrics such as `kafka_broker_active_count`, and filter by `service:<expected-cluster-id>`. In the imported dashboards, the filter is still named `cluster_id`, but it reads values from the Datadog `service` tag. If the expected service value does not appear, check one AutoMQ `/metrics` endpoint and confirm the actual `job` label because `honor_labels: true` preserves the target-provided `job` value. If your AutoMQ environment exposes native `env_id`, `instance_id`, or `broker_id` labels, you can use them as additional filters. New metrics may take 1–3 minutes to appear.

<img alt="AutoMQ metrics visible in Datadog Metrics Explorer after Linux Gateway OTEL Collector integration" />

## Appendix: Manually enable AutoMQ metrics export for Kubernetes Software

AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. Use this appendix only when you manage an AutoMQ Software deployment on Kubernetes and need to enable the endpoint manually.

For AutoMQ Software deployments on Kubernetes via Helm, set the metrics exporter to Prometheus mode in your Helm values:

```yaml theme={null}
global:
  config: |
    s3.telemetry.metrics.exporter.uri=prometheus://?host=0.0.0.0&port=9090
```

This exposes Prometheus-format metrics on port 9090 of each AutoMQ pod.

The Datadog Agent Gateway profile in this guide uses selector-based Kubernetes discovery, so it does not require Prometheus scrape annotations to discover AutoMQ pods. If you also use an annotation-based Prometheus or Datadog collector outside of this profile, add the following annotations to the AutoMQ Helm values:

```yaml theme={null}
controller:
  annotations:
    prometheus.io/automq-scrape: "true"
    prometheus.io/automq-port: "9090"
    prometheus.io/automq-path: "/metrics"

broker:
  annotations:
    prometheus.io/automq-scrape: "true"
    prometheus.io/automq-port: "9090"
    prometheus.io/automq-path: "/metrics"
```

## References

* [Datadog Agent Helm Chart](https://github.com/DataDog/helm-charts/tree/main/charts/datadog)
* [Datadog OTEL Collector on Kubernetes](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes/?tab=helm)
* [Datadog DDOT Collector on Linux](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/linux/)
* [OpenTelemetry Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver)


# Monitoring & Alert Via Prometheus
Source: https://docs.automq.com/automq-cloud/monitoring-alert/monitoring-alert-via-prometheus

AutoMQ offers cloud-native scalability and cost efficiency with Kafka compatibility. Monitor Kafka metrics effortlessly using Prometheus and Grafana, integrating custom alerts seamlessly.

Metrics are crucial analytical data for system observability. AutoMQ supports exposing various native Apache Kafka® Metrics data through Prometheus. This article outlines the details of the Metrics exposed by AutoMQ.

## Metrics collection methods

AutoMQ supports Prometheus-compatible monitoring through the following methods:

1. **Remote Write:** AutoMQ pushes metrics directly to a user-provided Prometheus Remote Write endpoint. Configure the endpoint and authentication when you create or update the instance.
2. **Exporter:** AutoMQ exposes a Prometheus-compatible scrape endpoint that your monitoring system pulls.

Exporter service discovery depends on the deployment platform:

* **Kubernetes deployments:** Discover the scrape endpoint by using standard Prometheus annotations on Kubernetes resources.
* **Virtual machine deployments:** Discover the hosts by using the host tags provided by the cloud provider.

Both methods provide metrics collection and delivery. Configure dashboards and alerts in your Prometheus-compatible monitoring system. AutoMQ also provides [Grafana dashboard templates](https://download.automq.com/automq-dashboard/dashboard-20260203.zip) and [Prometheus alert rule templates](https://github.com/AutoMQ/automq-for-kafka/blob/main/docker/telemetry/prometheus/rules/alert_rules_template.yml).

## Prometheus Metrics Definitions

For metric names, labels, and definitions, see [Prometheus Metrics in the AutoMQ open-source documentation](/automq/observability/prometheus-metrics).

## Grafana Dashboard Example

As mentioned above, AutoMQ Cloud does not offer managed Grafana dashboard services. Users can refer to the Grafana templates provided by AutoMQ to quickly configure dashboards. The related dashboard templates can be downloaded from [this link](https://download.automq.com/automq-dashboard/dashboard-20260203.zip).

The pre-configured Grafana dashboard templates offer metric monitoring across different dimensions:

* **Cluster Overview:** Provides cluster-level monitoring, including node count, data size, cluster traffic, etc. It also includes metric overviews for Topic, Group, and Broker dimensions and offers drill-down capabilities for detailed monitoring.

<img />

* **Broker Metrics:** Offers broker-level monitoring, including connection count, partition count, node traffic, node requests, and more.

<img />

* **Topic Metrics:** Provides topic-level metric monitoring, including message throughput, total data volume, partition count, and consumption latency.

<img />

* **Group Metrics:** Delivers group-level metric monitoring, including consumption rate and consumption latency.

<img />

## Business Monitoring and Alerts

AutoMQ integrates with Prometheus to push metrics data to Prometheus. Users can then configure custom alert rules in Prometheus to monitor for anomalies such as business stress levels.

### Kafka Cluster Alert Templates

AutoMQ provides a series of alert templates based on high-frequency metrics usage in production environments. Users can choose and configure these alert rules according to their actual needs.

The list of alert rule templates is as follows:

| **Alert Rule Item**<br />                   | **Function and Scenario**<br />                                                                                                                                                                                                                                                                                     |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| HighTopicBytesInPerSec<br />                | <ul><li>Alert Rule: High Topic Data Write Throughput Alert</li><li>Function and Scenario: Monitors if the data write rate per second for a specified topic exceeds a threshold, typically used for monitoring sudden write traffic.</li></ul>                                                                       |
| HighTopicBytesInPerSecDayToDayChange<br />  | <ul><li>Alert Rule: High Day-to-Day Increase in Topic Data Write Throughput Alert</li><li>Function and Scenario: Monitors if the day-to-day increase rate of data writes per second for a specified topic exceeds a threshold, generally used for monitoring sudden write traffic.</li></ul>                        |
| LowTopicBytesInPerSec<br />                 | <ul><li>Alert Rule: Low Topic Data Ingestion Throughput</li><li>Use Case and Scenario: Monitor if the data ingestion rate for a specified topic falls below a threshold, typically to detect if the ingestion pipeline is damaged or zeroed out.</li></ul>                                                          |
| LowTopicBytesInPerSecDayToDayChange<br />   | <ul><li>Alert Rule: Daily Drop in Topic Data Ingestion Throughput</li><li>Use Case and Scenario: Monitor if the daily data ingestion rate for a specified topic drops below a threshold compared to the previous day, generally used to detect damage in the ingestion pipeline or business fluctuations.</li></ul> |
| HighTopicBytesOutPerSec<br />               | <ul><li>Alert Rule: High Topic Data Retrieval Throughput</li><li>Use Case and Scenario: Monitor if the data retrieval rate for a specified topic exceeds a threshold, typically to monitor burst reads and Fanout traffic.</li></ul>                                                                                |
| HighTopicBytesOutPerSecDayToDayChange<br /> | <ul><li>Alert Rule: Daily Surge in Topic Data Retrieval Throughput</li><li>Use Case and Scenario: Monitor if the daily data retrieval rate for a specified topic exceeds a threshold compared to the previous day, generally used to monitor burst read traffic.</li></ul>                                          |
| LowTopicBytesOutPerSec<br />                | <ul><li>Alert Rule: Low Topic Data Retrieval Throughput</li><li>Use Case and Scenario: Monitor if the data retrieval rate for a specified topic falls below a threshold, typically to detect if the retrieval pipeline is damaged or zeroed out.</li></ul>                                                          |
| LowTopicBytesOutPerSecDayToDayChange<br />  | <ul><li>Alert Rule: Daily Drop in Topic Data Retrieval Throughput</li><li>Use Case and Scenario: Monitor if the daily data retrieval rate for a specified topic drops below a threshold compared to the previous day, generally used to detect damage in the retrieval pipeline or business fluctuations.</li></ul> |
| HighGroupConsumeRatePerTopic<br />          | <ul><li>Alert Rule: High Consumer Group Consumption Rate</li><li>Use Case and Scenario: Monitor if the consumption rate for a specified Consumer Group exceeds a threshold, typically used to detect abnormal read behavior.</li></ul>                                                                              |
| LowGroupConsumeRatePerTopic<br />           | <ul><li>Alert Rule: Low Consumer Group Consumption Rate</li><li>Use Case and Scenario: Monitor if the consumption rate for a specified Consumer Group falls below a threshold, typically used to detect abnormal read behavior.</li></ul>                                                                           |
| HighGroupConsumerLag<br />                  | <ul><li>Alert Rule: Consumer Group lag alert</li><li>Purpose and Scenario: Monitors the lag for a specified Consumer Group to check if it exceeds the threshold, typically used to detect read anomalies.</li></ul>                                                                                                 |

A complete alert template YAML file can be found below for copying and importing.

```yaml theme={null}
# This Is the Alert Rules Template for AutoMQ, Please Modify the Alert Thresholds and Period Per Your Needs
# Before Applying It to Your Production Environment.
groups:
  - name: kafka_alerts
    rules:
      - alert: HighTopicBytesInPerSec
        expr: sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic) > 50 * 1024 * 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High inbound network throughput {{ printf \"%0.2f\" $value }} Bytes/s for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The inbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} is exceeding threshold."

      - alert: LowTopicBytesInPerSec
        expr: sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic) < 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low inbound network throughput {{ printf \"%0.2f\" $value }} Bytes/s for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The inbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} is below threshold."

      - alert: HighTopicBytesOutPerSec
        expr: sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic) > 50 * 1024 * 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High outbound network throughput {{ printf \"%0.2f\" $value }} Bytes/s for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The outbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} is exceeding threshold."

      - alert: LowTopicBytesOutPerSec
        expr: sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic) < 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low outbound network throughput {{ printf \"%0.2f\" $value }} Bytes/s for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The outbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} is below threshold."

      - alert: HighGroupConsumeRatePerTopic
        expr: sum(max(rate(kafka_group_commit_offset{consumer_group="example_group", topic="example_topic"}[1m])) by (job, consumer_group, topic, partition)) by (job, consumer_group, topic) > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High group consume rate {{ printf \"%0.2f\" $value }} msg/s for consumer group {{ $labels.consumer_group }} on topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The consume rate of consumer group {{ $labels.consumer_group }} on topic {{ $labels.topic }} in cluster {{ $labels.job }} is exceeding threshold."

      - alert: LowGroupConsumeRatePerTopic
        expr: sum(max(rate(kafka_group_commit_offset{consumer_group="example_group", topic="example_topic"}[1m])) by (job, consumer_group, topic, partition)) by (job, consumer_group, topic) < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low group consume rate {{ printf \"%0.2f\" $value }} msg/s for consumer group {{ $labels.consumer_group }} on topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The consume rate of consumer group {{ $labels.consumer_group }} on topic {{ $labels.topic }} in cluster {{ $labels.job }} is below threshold."

      - alert: HighTopicBytesInPerSecDayToDayChange
        expr: (sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic) 
          - sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic))
          / sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic) > 0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High inbound network throughput change {{ printf \"%0.2f\" $value }} for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The increase of inbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} compared to 24h ago is exceeding threshold"

      - alert: LowTopicBytesInPerSecDayToDayChange
        expr: (sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic)
          - sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic))
          / sum(max(rate(kafka_network_io_bytes_total{direction="in", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic) < -0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low inbound network throughput change {{ printf \"%0.2f\" $value }} for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The decrease of inbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} compared to 24h ago is exceeding threshold"

      - alert: HighTopicBytesOutPerSecDayToDayChange
        expr: (sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic)
          - sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic))
          / sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic) > 0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High outbound network throughput change {{ printf \"%0.2f\" $value }} for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The increase of outbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} compared to 24h ago is exceeding threshold"

      - alert: LowTopicBytesOutPerSecDayToDayChange
        expr: (sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m])) by (job, topic, partition)) by (job, topic)
          - sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic))
          / sum(max(rate(kafka_network_io_bytes_total{direction="out", topic="example_topic"}[1m] offset 24h)) by (job, topic, partition)) by (job, topic) < -0.2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low outbound network throughput change {{ printf \"%0.2f\" $value }} for topic {{ $labels.topic }} in cluster {{ $labels.job }}"
          description: "The decrease of outbound bytes per second produced by topic {{ $labels.topic }} in cluster {{ $labels.job }} compared to 24h ago is exceeding threshold"

      - alert: HighGroupConsumerLag
        expr: (sum(max(kafka_log_end_offset{topic="example_topic"}) by (job, topic, partition)) by (job, topic)
          - on (topic) group_left (consumer_group) sum(max(kafka_group_commit_offset{consumer_group="example_group", topic="example_topic"}) by (job, consumer_group, topic, partition)) by (job, consumer_group, topic) > 10000)
          and (count(max(kafka_log_end_offset{topic="example_topic"}) by (job, topic, partition)) by (job, topic) == count(max(kafka_group_commit_offset{consumer_group="example_group", topic="example_topic"}) by (job, consumer_group, topic, partition)) by (job, consumer_group, topic))
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High group consumer lag {{ printf \"%0.f\" $value }} for consumer group {{ $labels.consumer_group }} in cluster {{ $labels.job }} on topic {{ $labels.topic }}."
          description: "The consumer lag of consumer group {{ $labels.consumer_group }} in cluster {{ $labels.job }} on topic {{ $labels.topic }} is exceeding threshold."
```

### Managed Connector Alert Templates

AutoMQ also provides a series of alert templates for Managed Connector scenarios to monitor Connector task status, resource usage, and data processing anomalies.

The list of alert rule templates is as follows:

| **Alert Rule Item**             | **Function and Scenario**                                                                                                                                                                                                                          |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| TasksFailed                     | <ul><li>Alert Rule: Connector Task Failure Alert</li><li>Function and Scenario: Monitors if the number of failed Connector tasks is greater than 0, used to detect task anomalies promptly.</li></ul>                                              |
| CPUUsage                        | <ul><li>Alert Rule: High CPU Usage Alert</li><li>Function and Scenario: Monitors if the CPU usage of a Connector instance exceeds 80%, used to monitor resource pressure.</li></ul>                                                                |
| TotalRecordFailures             | <ul><li>Alert Rule: Record Processing Failure Alert</li><li>Function and Scenario: Monitors if the total number of failed records processed by Connector tasks is greater than 0, used to detect data processing anomalies.</li></ul>              |
| FailedAuthenticationConnections | <ul><li>Alert Rule: Authentication Failure Connection Alert</li><li>Function and Scenario: Monitors the number of failed authentication connections between Connector and Kafka cluster, used to detect connection configuration issues.</li></ul> |

A complete alert template YAML file can be found below for copying and importing.

```yaml theme={null}
groups:
  - name: connect_alerts
    rules:
      - alert: TasksFailed
        expr: sum(kafka_connect_worker_metrics_connector_failed_task_count{connect_instance_id=~"<your-connect-instance-id>", connector=~"<your-connector-name>"}) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Connector tasks failed for {{ $labels.connector }} in connect instance {{ $labels.connect_instance_id }}"
          description: "The number of failed tasks for connector {{ $labels.connector }} is greater than 0."

      - alert: CPUUsage
        expr: rate(jvm_cpu_time_seconds_total{connect_instance_id=~"<your-connect-instance-id>"}[5m]) > 0.8
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage {{ printf \"%0.2f\" $value }} for connect instance {{ $labels.connect_instance_id }}"
          description: "The CPU usage of connect instance {{ $labels.connect_instance_id }} is exceeding 80%."

      - alert: TotalRecordFailures
        expr: kafka_task_error_metrics_total_record_failures_total{connect_instance_id=~"<your-connect-instance-id>", connector=~"<your-connector-name>"} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Record failures detected for connector {{ $labels.connector }} task {{ $labels.task }}"
          description: "The total record failures for connector {{ $labels.connector }} task {{ $labels.task }} is greater than 0."

      - alert: FailedAuthenticationConnections
        expr: kafka_connect_metrics_failed_authentication_total{connect_instance_id=~"<your-connect-instance-id>", client_id!=""} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Failed authentication connections for client {{ $labels.client_id }} in connect instance {{ $labels.connect_instance_id }}"
          description: "The number of failed authentication connections for client {{ $labels.client_id }} is greater than 0."
```

<Tip>
  When using the Managed Connector alert templates, replace `<your-connect-instance-id>` and `<your-connector-name>` with your actual Connect instance ID and Connector name.
</Tip>

### Configuration Steps

AutoMQ provides the above Prometheus alert template. Users can import this alert template into their current Prometheus cluster (instance) and then configure custom alert rules based on the template.

The following example uses Alibaba Cloud Prometheus to demonstrate the configuration method. If you are using a self-hosted Prometheus, adjust accordingly.

#### Configuring Alerts in Alibaba Cloud Prometheus

1. **Import the alert template:** Open the [Alibaba Cloud Prometheus Console](https://prometheus.console.aliyun.com/), go to **Alert Rule Templates**, and select **Batch Import Template**.

<img />

2. **Copy the content of the above template file and import it.**

<img />

3. **Apply Alert Template:** After importing the template, select the specific alert template and click **Apply Template** to apply the desired alert template to the corresponding Prometheus instance.

<img />

4. **Configure Monitoring and Alert Rules:** Take the HighGroupConsumerLag alert as an example. After clicking "Apply Template," select the corresponding Prometheus instance. Once applied successfully, you can see the enabled alert rules in the "Alert Rules List" on the left side.

<img />

5. Click "Edit" to enter the alert rule editing page. Change "example\_topic" and "example\_group" to the topic and consumer group you want to monitor, and modify the alert threshold (10000 in the image below) to your desired value.

<img />

6. **Configure Notification Policy:** After editing, select an existing notification policy or click "Create New Notification Policy" to create one.

7. **Quickly Copy Alert Policies (Optional):** If you want to monitor multiple topics or consumer groups, you can click **"Copy"** to create multiple alert rules.


# Release Notes
Source: https://docs.automq.com/automq-cloud/release-notes

Experience seamless cloud-native scalability and full Kafka API compatibility with AutoMQ, cutting costs while ensuring robust data infrastructure management across various cloud environments.

# Version Numbers

AutoMQ Cloud employs a versioning format of `{major version}.{minor version}.{patch version}` for its releases.

* **Major version:** Indicates major upgrades or changes and is incremented when significant updates are made. This version may be backward incompatible.

* **Minor version:** Represents the addition of new features that maintain backward compatibility.

* **Patch version:** Represents fixes for issues while maintaining backward compatibility.

# Historical Versions

Each official release of AutoMQ Cloud corresponds to a specific [automq](https://github.com/AutoMQ/automq-for-kafka) kernel version and a compatible range of Apache Kafka versions. The current historical version information for AutoMQ Cloud is as follows:

| **Environment Console Version**<br /> | **Data Plane Cluster Version**<br /> | **Compatible Apache Kafka Versions**<br /> |
| ------------------------------------- | ------------------------------------ | ------------------------------------------ |
| 8.3.1<br />                           | 1.0.0 \~ 5.5.x<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 8.1.2<br />                           | 1.0.0 \~ 5.3.5<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 8.0.0<br />                           | 1.0.0 \~ 5.3.1<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.8.7<br />                           | 1.0.0 \~ 5.3.1<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.7.4<br />                           | 1.0.0 \~ 5.2.0<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.6.0<br />                           | 1.0.0 \~ 5.1.0<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.5.1<br />                           | 1.0.0 \~ 5.0.1<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.4.0<br />                           | 1.0.0 \~ 5.0.0<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.3.1<br />                           | 1.0.0 \~ 1.4.1<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.2.0<br />                           | 1.0.0 \~ 1.4.0<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 7.1.0<br />                           | 1.0.0 \~ 1.4.0-beta1<br />           | 0.9.0.x \~ 3.9.x<br />                     |
| 7.0.0<br />                           | 1.0.0 \~ 1.4.0-beta1<br />           | 0.9.0.x \~ 3.9.x<br />                     |
| 1.6.5<br />                           | 1.0.0 \~ 1.4.0-beta1<br />           | 0.9.0.x \~ 3.9.x<br />                     |
| 1.5.10<br />                          | 1.0.0 \~ 1.4.0-beta1<br />           | 0.9.0.x \~ 3.9.x<br />                     |
| 1.5.4<br />                           | 1.0.0 \~ 1.4.0-beta<br />            | 0.9.0.x \~ 3.9.x<br />                     |
| 1.5.3<br />                           | 1.0.0 \~ 1.3.5<br />                 | 0.9.0.x \~ 3.9.x<br />                     |
| 1.3.7<br />                           | 1.0.0 \~ 1.2.5<br />                 | 0.9.0.x \~ 3.8.x<br />                     |
| 1.3.2<br />                           | 1.0.0 \~ 1.2.2<br />                 | 0.9.0.x \~ 3.8.x<br />                     |
| 1.2.8<br />                           | 1.0.0 \~ 1.2.0<br />                 | 0.9.0.x \~ 3.8.x<br />                     |
| 1.2.1<br />                           | 1.0.0 \~ 1.1.1<br />                 | 0.9.0.x \~ 3.7.x<br />                     |
| 1.1.3<br />                           | 1.0.0 \~ 1.1.1<br />                 | 0.9.0.x \~ 3.7.x<br />                     |
| 1.1.1<br />                           | 1.0.0 \~ 1.1.0<br />                 | 0.9.0.x \~ 3.7.x<br />                     |
| 1.1.0<br />                           | 1.0.0 \~ 1.1.0<br />                 | 0.9.0.x \~ 3.7.x<br />                     |
| 1.0.0<br />                           | 1.0.0 \~ 1.0.4<br />                 | 0.9.0.x \~ 3.4.x<br />                     |

## Historical Versions of Environment Console

### 8.3.1

* Supports modifying cloud resource tags in AWS environments.
* Supports custom security groups.
* Supports dynamic configuration changes for NFS WAL instances that use EFS or FSx for NetApp ONTAP, so you can adjust throughput specifications as needed.
* Supports small instance specifications and vertical EC2 instance type changes in AWS environments.
* Adds consumer lag metrics that do not require user-side aggregation, and adds metrics for consumer delay time.
* Supports independent Schema Registry configuration management.
* Refactors the Managed Connector model to support independent Worker cluster management and sharing one Worker cluster across multiple Connectors.

### 8.1.2

* Optimized the ops channel: a secure and controllable ops channel can be achieved based on ngrok.
* Managed Connector supports custom node pools, custom K8S scheduling policies, Kafka multi-protocol selection control, and version upgrades, optimizing the experience of Plugin management and log query.
* Support custom node pools when deploying clusters on EKS, removing the limitation of the AL2 system.
* Support for FSx WAL instances in AWS environments.
* Support for viewing the creation status of instances being created, exposing necessary error messages.
* Optimized several defects in KafkaLinking to support more accurate automatic Group promotion.
* Support for setting custom security groups when creating instances.
* The installation and deployment console supports custom IAM policies for the console.

### 8.0.0

* Simplified the console installation and initialization process with Docker.
* Provided a managed Kafka Connect service with built-in managed plugins for MongoDB, Clickhouse, and ClickZetta, supporting one-click data ETL.
* Supported Pod Identity authentication for AWS EKS mode deployments.
* Enabled Kafka Linking to access source clusters via SASL\_SSL and mTLS protocols.
* Enhanced security by supporting certificate domain validation for deployments using SASL\_SSL and mTLS.
* Supported cloud resource tags, allowing custom tags to be set for cloud resources used by AutoMQ.
* Fixed other known defects and optimizations.

### 7.8.7

* Support for data plane version 5.3.1, with FSxWAL support in AWS environments.

* Support for setting custom Tags in AWS environments.

### 7.7.4

* Support for Prometheus RemoteWrite integration with custom Labels.

* Support for custom Placeholders in Kubernetes deployment.

### 7.6.0

* Support reducing cross-availability zone traffic for producers and consumers in AWS environments.

* Support disaster recovery for object storage in Tencent Cloud environments.

### 7.5.1

* Support for extended custom Listener and AdvertisedListener mapping.

### 7.4.0

* Supports Kafka Linking for zero-downtime cluster migrations.

* Supports deployment to Azure.

### 7.3.1

* Supports data plane version 1.4.1.

* Supports the maintenance channel, enabling secure and controllable remote operation and maintenance.

* Supports metric integration into the Kafka Topic.

* TableTopic functionality integrates with Hive Metastore Catalog.

* Supports instance lifecycle OpenAPI.

### 7.2.0

* Supports data plane version 1.4.0.

* Optimize the failover mechanism in Kubernetes deployment to achieve faster recovery time.

### 7.1.0

* Supports data plane version 1.3.10.

* Supports storage encryption on AWS cloud environments.

* Enhance the rate-limiting mechanism of object storage to ensure smoother read and write traffic during rate-limiting triggers.

### 7.0.0

* Change the version number sequence starting from 7.0 to avoid confusion with the data plane cluster version.

* Each BYOC environment supports multiple DeployProfiles.

* Each DeployProfile supports multiple Buckets.

* Support for SASL\_SSL and mTLS.

* Support for SAML SSO and fine-grained RBAC control.

* Supports deployment to Baidu Cloud.

### 1.6.5

* Optimize access point display in load balancing mode.

* Support instance creation ranging from 60MiB/s to 10GiB/s.

* Add a new customer information registration page.

* Fix various issues from previous versions.

### 1.5.10

* Supports deployment on AWS EKS.

* Update the supported data plane version to 1.4.0-beta1.

* Fixed vulnerabilities in AWS base images (including critical vulnerabilities).

### 1.5.4

* New Table Topic feature, integrating Kafka messages into Iceberg tables and supporting AWS S3 Table Bucket.

### 1.5.3

* Added tag management feature, supporting the passthrough of cloud resource tags used by AutoMQ.

* Introduced scheduled elasticity rules, supporting one-time and recurring scheduled elasticity rules for planned scaling.

* Added inbound rule control for access points, allowing custom source CIDR allowlist settings.

### 1.3.7

* Fixed an issue where AWS CloudFormation could not install using an existing VPC subnet.

### 1.3.2

* Supports accessing the environment console using a fixed EIP in the AWS environment.

* Support for creating 3AKU specification instances. On AWS, small specifications support Graviton-based EC2 instances.

* Supports creating instances with S3 WAL type.

* Fixed upgrade issues when using AWS CloudFormation installation method.

### 1.2.8

* Supports Prometheus RemoteWrite integration.

* Supports querying offline Consumer Group lag data.

* Supports custom version selection when upgrading instance versions.

* Supports instance version 1.2.0, compatible with Apache Kafka 3.8.x.

### 1.2.1

* Adds service account functionality.

* Supports Terraform Provider, allowing management of instances, Topics, and other resources through Terraform.

* Supports CloudWatch integration, forwarding metrics data to CloudWatch.

### 1.1.3

* Data plane cluster upgrade supports version 1.1.1.

### 1.1.1

* Fixed an issue where migration task metrics were displayed inaccurately.

### 1.1.0

* Upgraded the AutoMQ environment console to CloudScape UI style.

* AutoMQ clusters support ACL management and custom authorization.

### 1.0.0

* Available for purchase on Alibaba Cloud Marketplace.

* Supports AutoMQ Kafka instance lifecycle management.

* Supports Topic lifecycle management.

* Supports Consumer Group lifecycle management.

* Supports Metrics integration.

* Supports user management and role authorization.

## Historical Versions of Data Plane Cluster

### 5.5.0

* Adds Consumer Lag time-lag collection and offset-timestamp query support.
* Publishes structured Kafka Linking and Prometheus RemoteWrite failure events to the cluster events Topic.
* Improves S3 WAL failover target node selection, Kubernetes scale-in handling, and cloud storage delete/detach error classification.

### 5.4.3

* Fixed an issue where ByteBuf could be released multiple times.

* Fixed an issue where partitions could not be expanded after Linking Promoted.

* Upgraded the Linux base image to fix CVE-2026-31431.

### 5.4.0

* ZeroZone supports EFS as a low-latency storage medium.

* Fixed an issue where linking Topics with no data caused ZeroZone to fail cross-AZ consumption.

* Fixed a memory leak issue in ZeroZone mode.

### 5.3.5

* Optimized Kafka Linking related functions:
  * Support for more accurate Group promotion.
  * Optimized performance.
* Support for AWS FSx WAL instances.
* Support for deployment based on EKS custom node pools, removing the limitation of the AL2 system.

### 5.3.1

* Support for FSWAL clusters in AWS environments.

* Optimized several known kernel issues.

### 5.2.0

* Optimized Kubernetes deployment with support for custom Placeholders.

* Fixed various kernel issues.

### 5.1.0

* Support reducing cross-availability zone traffic for producers and consumers in AWS environments.

* Support disaster recovery for object storage in Tencent Cloud environments.

### 5.0.1

* Support for reporting the expiration date of data plane cluster TLS certificates through Metrics.

* Support for extended custom Listener and AdvertisedListener mapping.

### 5.0.0

* Supports Kafka Linking for zero-downtime cluster migrations.

* Supports Azure deployment.

### 1.4.1

* TableTopic supports the Hive Metastore Catalog.

* Supports metric integration into the Kafka Topic.

### 1.4.0

* Optimize the failover mechanism in Kubernetes deployment to achieve faster recovery time.

* Enhance the rate-limiting mechanism of object storage to ensure smoother read and write traffic during rate-limiting triggers.

### 1.4.0-beta1

* Fixed known issues with Table Topic.

### 1.4.0-beta

* Support for Table Topic feature, integrating Kafka messages into Iceberg tables and supporting AWS S3 Table Bucket.

### 1.3.10

* Added fast-fail retry optimization for object storage requests.

* Fixed the issue with writing large messages.

### 1.3.9

* Supports deployment on AWS EKS.

### 1.3.6

* Fix the issue of reading Compact Topics resulting in a StackOverflow in specific scenarios.

### 1.3.5

* Optimized server-side metadata management, supporting scheduled elasticity rules.

### 1.3.0

* Upgraded version, compatible with Apache Kafka 3.9 version.

* Support multi-point writing capability to eliminate cross-AZ traffic.

* AutoBalancer performance optimization significantly reduces scheduling frequency.

### 1.2.6

* Fix the issue of consumer stalling in scenarios with extreme concurrent cold reads.

### 1.2.5

* Fixed server-side SASL authentication errors.

### 1.2.4

* Optimized operating system memory fragmentation.

### 1.2.3

* Optimized Kraft's performance in large-scale clusters.

### 1.2.2

* Fixed a problem in AWS environments where mount path conflicts could lead to failover failures.

### 1.2.1

* Fixed an issue where upgrading from 1.1.x to 1.2.0 could result in end offset loss.

* Fixed an issue in AWS environments where machines without mounted public IPs could not start properly.

### 1.2.0

* Upgraded version to be compatible with Apache Kafka 3.8.

### 1.1.7

* Optimized operating system memory fragmentation

### 1.1.6

* Fixed resource cleanup issue when client connection is dropped

* Resolved compatibility issue with fetch.min.bytes parameter

### 1.1.1

* Optimized self-balancing strategy to reduce the probability of commit offset failure for client versions (\<3.x)

### 1.1.0

* Upgraded version to be compatible with Apache Kafka 3.7.

### 1.0.4

* Fixed several defects from version 1.0.0 upgrade

* Enhanced user experience

## Additional Notes:

* Maintain a consistent release cadence across processor architectures.


# Obtaining Services
Source: https://docs.automq.com/automq-cloud/support/obtaining-services

AutoMQ Cloud provides comprehensive technical support through one-on-one expert services via online tickets, Slack, and service email, ensuring effective assistance for enterprise users.

This article introduces how to obtain technical support and services when using the AutoMQ Cloud product.

## AutoMQ Cloud Technical Support

After enterprise users purchase a paid subscription to AutoMQ Cloud, AutoMQ technical personnel will establish a one-on-one expert service guarantee system for enterprise customers, providing comprehensive technical consulting, communication, emergency response, and other support.

Customers can choose the following channels to contact us based on their current communication preferences.

<Tabs>
  <Tab title="Online Ticket">
    The AutoMQ official website offers a ticket system. Customers can visit the official [Contact Us](https://www.automq.com/contact) page to submit relevant information.
  </Tab>

  <Tab title="Slack">
    AutoMQ offers a public consultation channel for Slack users. Customers using Slack can contact us by joining the [AutoMQ Slack Channel](https://go.automq.com/slack).
  </Tab>

  <Tab title="Service Email">
    AutoMQ Cloud provides a public email for technical services. Customers can reach us at [support@automq.com](mailto:support@automq.com) for support.
  </Tab>
</Tabs>

## Other Media Channels

🌟GitHub: [https://github.com/AutoMQ](https://github.com/AutoMQ)

💻 Official Website: [https://www.automq.com](https://www.automq.com)

🔍 Video Channel: AutoMQ


# Service Level Agreement
Source: https://docs.automq.com/automq-cloud/support/service-level-agreement

Experience superior cloud-native scalability and Kafka compatibility with AutoMQ. Ensure 99.95% service availability, cost-efficiently managing data streaming needs.

**AutoMQ Cloud Service Level Agreement**

**Effective Date: December 16, 2023**

This Service Level Agreement (hereinafter referred to as "SLA") defines the service availability levels, shared responsibility boundaries, and compensation schemes provided by AutoMQ HK Limited to customers for the AutoMQ Cloud (hereinafter referred to as "AutoMQ"). Please note, unless otherwise agreed, this agreement does not apply to public beta, invite-only beta, and free test environments and instances.

## **Definitions**

**Service Period** : A service period is one calendar month.

**Total Minutes in a Single Instance Service Period** : Calculated as the total number of days in the service period for a single instance × 24 (hours) × 60 (minutes).

**Instance Unavailability** : When an AutoMQ instance configured as expected fails to produce messages or encounters message production failures, and this condition lasts for more than one minute, the AutoMQ instance is considered unavailable for that minute.

**Single Instance Unavailable Minutes** : The sum of minutes a single AutoMQ instance is unavailable within a service period.

**Monthly Service Fee:**  The total service fee paid by the customer for a single AutoMQ instance within a service cycle (i.e., a calendar month). If the customer pays for multiple months at once, the monthly service fee will be prorated according to the number of months purchased.

## **Service Availability**

**2.1 Service Availability Calculation Method**

The service availability of AutoMQ will be calculated for each AutoMQ instance based on the service cycle, using the following two dimensions:

Service Availability = (Total minutes in the service cycle for a single instance - Unavailable minutes for a single instance) / Total minutes in the service cycle for a single instance × 100%

**2.2 Service Availability Commitment**

AutoMQ commits that the service availability for an instance within a service cycle will be no less than 99.95%.

**2.3 If the above availability commitment is not met, the customer can receive compensation as stipulated in Section 3 of this agreement.**

**Compensation does not cover service unavailability caused by the following reasons:**

(1) Attacks on the customer's application by hackers;

(2) Loss or leakage of data, credentials, passwords, etc., due to improper maintenance or confidentiality by the customer;

(3) Customer's negligence or actions authorized by the customer;

(4) Failure to follow AutoMQ product usage documentation or recommendations, such as incorrect changes or releases to AutoMQ instances via the console, API, or CLI;

(5) Errors in AutoMQ instances caused by installing third-party software or configurations not provided by AutoMQ;

(6) Brief service interruptions due to normal maintenance or upgrades of instances as described in the "AutoMQ Cloud Service Terms";

(7) Force majeure events.

## **Compensation Plan**

**3.1 Compensation Standards**

For AutoMQ instances, if the service availability falls below 99.95%, compensation can be obtained according to the standards in the table below. The compensation method is limited to vouchers or other deduction tools for purchasing AutoMQ instances. The total compensation amount shall not exceed the single-instance monthly service fee (excluding fees deducted using vouchers) paid by the customer for the AutoMQ instance in the month when the service availability commitment was not met.

| **Service Availability**<br />               | **Compensation Voucher Amount**<br /> |
| -------------------------------------------- | ------------------------------------- |
| Below 99.95% but equal to or above 99%<br /> | 15% of the monthly service fee<br />  |
| Below 99% but equal to or above 95%<br />    | 30% of the monthly service fee<br />  |
| Below 95%<br />                              | 100% of the monthly service fee<br /> |

**3.2 Compensation Application Deadline**

Customers may submit compensation claims for AutoMQ instances that did not meet availability requirements within five (5) business days after the end of the month. **Compensation claims must be submitted within two (2) months following the month in which AutoMQ did not meet service availability. Claims submitted beyond this timeframe will not be processed.**

## **Others**

**4.1 Service Agreement Amendments**

This Service Level Agreement (SLA) takes effect on December 16, 2023. AutoMQ reserves the right to modify the terms of this SLA. If there are any changes to the SLA terms, AutoMQ will notify you 30 days in advance via website announcement or email. If you do not agree with the modifications to the SLA, you have the right to stop using AutoMQ services. Continued use of AutoMQ services will be deemed as acceptance of the modified SLA.

**4.2 Responsibility Sharing Model**

AutoMQ Cloud offers both SaaS and BYOC deployment models. Based on the deployment model, there are shared responsibilities between AutoMQ and the customer. Refer to the table below for details:

**SaaS Model**

In the SaaS model, AutoMQ instances are deployed in the service provider's account, and customers do not need to manage the underlying cloud resources of the instances. The related responsibility model is as follows:

| **Responsibility**<br />             | **AutoMQ Service Provider**<br /> | **Customer**<br /> |
| ------------------------------------ | --------------------------------- | ------------------ |
| Cluster Deployment<br />             | Responsible<br />                 | -<br />            |
| System Vulnerability Patching<br />  | Responsible<br />                 | -<br />            |
| Software Version Upgrades<br />      | Responsible<br />                 | -<br />            |
| Instance Capacity Expansion<br />    | Responsible<br />                 | -<br />            |
| Cloud Account Authorization<br />    | Responsible<br />                 | -<br />            |
| Network Connectivity Assurance<br /> | Responsible<br />                 | -<br />            |
| Service Availability SLA<br />       | Responsible<br />                 | -<br />            |

**BYOC Deployment Model**

In the BYOC deployment model, AutoMQ instances are deployed in the customer's cloud account, allowing customers to manage and monitor the underlying cloud resources of the instances. The responsibility model is as follows:

| **Responsibility**<br />             | **AutoMQ Service Provider**<br /> | **Customer**<br />                                     |
| ------------------------------------ | --------------------------------- | ------------------------------------------------------ |
| Cluster Deployment<br />             | Responsible<br />                 | -<br />                                                |
| System Vulnerability Patching<br />  | Responsible<br />                 | -<br />                                                |
| Software Version Upgrades<br />      | Responsible<br />                 | -<br />                                                |
| Instance Capacity Expansion<br />    | Responsible<br />                 | Responsible (ensure cloud resource availability)<br /> |
| Cloud Account Authorization<br />    | Responsible<br />                 | Responsible<br />                                      |
| Network Connectivity Assurance<br /> | Responsible<br />                 | Responsible<br />                                      |
| Service Availability SLA<br />       | Responsible<br />                 | -<br />                                                |


# Terms of Service
Source: https://docs.automq.com/automq-cloud/support/terms-of-service

AutoMQ Cloud offers seamless Kafka compatibility, cloud-native scalability, and cost efficiency. Discover our comprehensive terms of service and user agreement.

**User Agreement**

Welcome to AutoMQ Cloud services.

This AutoMQ Cloud Service End User Agreement ("this Agreement") is a legally binding agreement between **You ("User")**  and **AutoMQ HK Limited ("Service Provider")** , regarding the Service Provider's provision of AutoMQ Cloud services ("the Service").

I. Effectiveness of the Agreement

1. User Confirmation: This Agreement is provided online by the Service Provider. Users should carefully read this Agreement before purchasing the Service and must keep the Agreement checked to proceed to the next purchase step. Once the User confirms this Agreement, it becomes legally binding and enforceable against both the Service Provider and the User.

2. User Activation: Users can activate the Service through Public Cloud marketplace products or other means. The Service Provider will provide an online version of this Agreement for the User's review at the relevant stage. **Once the User uses the Service in any of the aforementioned ways, the User agrees to all terms of this Agreement. If the User does not accept the terms of this Agreement, please do not use the Service.**

II. Definitions

1. AutoMQ Cloud services refer to using the Service Provider's image to install AutoMQ Cloud software and provide technical support services, including fault response, upgrades, and migrations.

III. License of Use

1. This service is protected by copyright law, international copyright treaties, and other relevant intellectual property laws, regulations, and treaties. This software product/service is licensed for use by authorized users and is not sold.

2. The ownership and all intellectual property rights (including but not limited to patent rights and their applications, trademark rights and their applications, copyrights, proprietary technology ownership, and usage rights, etc.) related to the source code, object code, and technical documentation of the software/service under this agreement belong to the service provider (AutoMQ HK Limited).

3. According to the terms of this agreement, the service provider grants the user a non-exclusive, non-transferable license to use this service. Users may not resell or otherwise transfer this license right for commercial gain.

IV. Fees

1. Users must pay all fees in accordance with the service's pricing structure to use this service. The service provider reserves the right not to provide services and/or technical support to users who have not paid all fees as agreed, or to terminate services and/or technical support.

2. If the user renews the service and the name, specifications, or price of the service have been adjusted, the user agrees to comply with the new service name, specifications, or price effective at the time. If the user does not agree to the new service name, specifications, or price, they may choose not to renew, and the service will automatically terminate upon expiration.

V. Rights and Obligations of the User

1. The user guarantees that all actions taken in utilizing this service comply with national laws and regulations, are legal and truthful, and do not infringe on the legal rights of any third party.

2. The user shall pay the fees for this service in full and on time; otherwise, the service provider reserves the right to terminate the user's access to this service at any time. The user shall bear full responsibility for any damages that may arise from the service provider's termination of the service.

3. The user guarantees that, except as explicitly permitted by law, they shall not reverse engineer, decompile, or disassemble this service.

4. The user is obligated to keep the service provider's trade secrets confidential, including but not limited to intellectual property, pricing, information technology, market strategies, and other related commercial secrets, and shall not disclose them to any third party.

5. The user understands and agrees that, due to the limitations of existing technology, the services provided by the service provider may have defects and cannot guarantee normal execution or the achievement of the user's expected results under all circumstances.

VI. Rights and Obligations of the Service Provider

1. The service provider guarantees that it has the legal right to grant the user a license to use this service and ensures the legality of its copyright.

2. The service provider commits that its provision of this service to the user does not infringe on any third party's legal rights, including but not limited to third-party intellectual property rights. If its actions result in any legal proceedings or administrative procedures initiated by a third party against the user, the service provider shall bear the legal responsibility and consequences.

3. The provider is responsible for timely upgrading, maintaining, and managing this service. They will offer free consultation and technical support services to users via phone support and online customer service.

4. The provider commits to ensuring the stability and continuity of the services provided to users. If the provider's actions lead to an early termination of the user's service license, the provider must fulfill the corresponding refund obligations to the user (if any). If such actions result in losses for the user, the provider shall be fully liable for compensation.

5. The provider commits that the services provided to users shall not contain:

   1. Any viruses, trojan horses, worms, time bombs, or other computer programs that are intended to damage, maliciously interfere with, secretly intercept or expropriate any system, data, or personal information;

   2. Any known vulnerabilities, backdoors, malware;

   3. Other unsafe content that may harm the security of the user's system.

6. **Limitation of Liability: To the maximum extent permitted by applicable law, the provider shall not be liable for any indirect, incidental, direct, special, punitive, or other damages (including but not limited to damages for personal injury or property damage, loss of profits, data loss, business interruption, computer crash or malfunction, loss of business information, damages for privacy disclosure due to failure to perform any duty including good faith or reasonable care, damages for negligence, or any financial losses or any other damages) arising out of or in connection with the use or inability to use the service, even if the provider has been advised of the possibility of such damages.**

VII. Use and Disclosure of User Data

1. To serve users better, the service provider may use user data to offer services, including but not limited to sending users information about products and services.

2. Without user permission, the service provider will not disclose user data without authorization. However, user data may be partially or fully disclosed in the following situations:

(1) With user consent, disclosed to third parties;

(2) Disclosed to third parties or administrative or judicial institutions as required by relevant laws, or by administrative or judicial institutions;

(3) Disclosed to third parties if the user violates relevant Chinese laws and regulations;

(4) Sharing user data with third parties as necessary to provide the software or services requested by the user.

VIII. Term and Termination

1. The term of the service shall be based on the actual term when the user submits the order and completes the purchase.

2. The service provider reserves the right to terminate the provision of this service without any liability under any of the following circumstances:

(1) The user's purchased service has expired and has not been renewed;

(2) The user has seriously violated the terms of this agreement and has not corrected such violations after being notified by the service provider.

IX. Amendment of the Agreement

1. If any content of this agreement changes, the service provider should inform the user of the modified content through appropriate means.

2. If the user does not agree with the modifications to the relevant terms of this agreement made by the service provider, the user has the right to stop using this service. In such cases, the service provider should refund the remaining fees (if any) to the user and provide a reasonable period before terminating the service for the user to back up, download, save, reassign data, and perform other necessary tasks. If the user continues to use this service, it will be deemed that the user has accepted the modifications to the relevant terms of this agreement made by the service provider.

X. Law and Dispute Resolution

1. The formation, execution, interpretation, and resolution of disputes regarding this agreement shall be governed by the laws of the People's Republic of China.

2. In the event of any dispute arising from the execution, performance, or interpretation of this Agreement, both parties shall strive to resolve the matter amicably through friendly consultation. If the consultation fails, either party may file a lawsuit with the People's Court at the location of the defendant.

XI. Supplementary Provisions

1. If any term of this Agreement is deemed invalid or unenforceable, such term may be severed, and the remaining parts shall continue to be in full force and effect.

2. The titles of this Agreement are provided for convenience of reading only and shall not define, limit, interpret, or describe the scope or boundaries of the terms.

3. The service provider may, based on its own operational conditions and upon giving the user a prior notice of \[30] working days, transfer all its rights and obligations under this Agreement to a third party without obtaining the user's prior consent. The assignee of the service provider shall be bound by this Agreement, and the service provider and its assignee shall be jointly and severally liable for the performance of this Agreement.


# Overview
Source: https://docs.automq.com/automq-cloud/table-topic/overview

AutoMQ Table Topic offers real-time data lake ingestion with Kafka compatibility, eliminating traditional ETL processes and providing elastic scaling and AWS integration. Enable with one click.

AutoMQ Table Topic integrates with Iceberg, allowing streaming data to be ingested into data lakes for analysis and querying. This article introduces the technical architecture, principles, and core concepts of Table Topic functionality.

## Architecture and Advantages

AutoMQ Table Topic enables real-time data lake ingestion and query analysis through its embedded streaming table architecture. The technical architecture is outlined below:

<img alt="Table Topic has several advantages over traditional ETL data lake ingestion solutions:" />

Table Topic has several advantages over traditional ETL data lake ingestion solutions:

* **Out of the Box:** With just one click, you can activate AutoMQ Table Topic to stream data into Iceberg tables for continuous, real-time analytics.

* **Built-in Schema Registry:** The built-in Kafka Schema Registry is ready to use. Table Topic leverages registered schemas to automatically create Iceberg tables in your catalog services (e.g., AWS Glue) and supports automatic schema evolution.

* **ETL-Free (Extract, Transform, Load):** Traditional data lake ingestion methods often require intermediary tools such as Kafka Connect or Flink. Table Topic eliminates such ETL pipelines, significantly reducing costs and operational complexity.

* **Auto Scaling:** AutoMQ features a stateless and elastic architecture that allows brokers to scale up or down with dynamic partition reassignment. Table Topic leverages this framework to handle ingestion rates from hundreds of MiB/s to several GiB/s.

* **AWS S3 Table Integration:** Table Topic integrates with S3 Table, leveraging its Data Catalog and maintenance features such as compression, snapshot management, and unreferenced file deletion. This integration also facilitates large-scale data analytics through AWS Athena.

## Constraints and Limitations

Using the AutoMQ Table Topic feature requires meeting the following conditions:

* **Version Constraint:** The AutoMQ instance version must be >= 1.4.1.

* **Catalog Requirement:** To use Table Topic, users must provide an external, available Data Catalog service. Currently, AutoMQ supports the following Catalog types:

  * **AWS S3Table Catalog:** AWS S3 offers a new Table Bucket, equipped with built-in catalog management and data lake storage.

  * **AWS Glue Catalog:** AWS Glue provides unified catalog management in the cloud and supports integration with query tools such as Athena.

  * **Hive Catalog:** Customers can provide catalog support themselves based on the Hadoop ecosystem's Hive Metastore, or they can purchase cloud provider-hosted EMR HMS services.

## Workflow

Users need to follow the configuration workflow when using the AutoMQ Table Topic feature as outlined below:

1. **Configure Data Catalog**: Select and configure your preferred Data Catalog service (e.g., AWS S3Table Catalog, AWS Glue Catalog, or Hive Catalog).
2. **Create Instance, Enable Table Topic, and Set Target Catalog**: When creating an AutoMQ instance, enable the Table Topic feature and specify the configured Data Catalog.
3. **Configure Topic to Enable Stream Table Rotation**: In the Topic configuration, enable the stream table rotation feature.

<Tip>
  **Note:**

  When configuring a Topic for stream table processing, you can freely modify the default Topic parameter configurations. For detailed Table Topic configuration information, refer to [Table Topic Configuration▸](/automq/table-topic/table-topic-configuration).
</Tip>

## Practices Tutorial

* [Table Topic Integration with AWS S3 Table▸](/automq-cloud/table-topic/table-topic-integration-with-aws-s3-table)

* [Table Topic Integration with AWS Glue▸](/automq-cloud/table-topic/table-topic-integration-with-aws-glue)


# Table Topic Integration with AWS Glue
Source: https://docs.automq.com/automq-cloud/table-topic/table-topic-integration-with-aws-glue

Integrate AutoMQ's Kafka-compatible Table Topic with AWS Glue for data lake ingestion, analytics, and querying without ETL. Scale with cloud-native and cost-efficient solutions.

AutoMQ Table Topic supports integration with Iceberg, allowing for streaming data into the lake for analysis and queries, while eliminating the need for ETL configuration and maintenance. This document explains how to configure the integration of Table Topic with Glue in an AWS environment.

## Prerequisites

To use the AutoMQ Table Topic feature in an AWS environment, the following conditions must be met:

* **Version Constraints:** The AutoMQ instance must be version >= 1.4.1.

* **Instance Constraints:** The Table Topic feature must be enabled at the time of creating the AutoMQ instance. It can only be used if enabled during creation, as it cannot be activated later.

* **Resource Requirements:** On AWS, Table Topic allows you to use AWS Glue as a Data Catalog or AWS S3 Tablebucket as a Data Catalog.

## Operational Steps

### Step 1: Prepare a Data Lake Bucket

When using Glue Catalog, data lake tables need to be stored in an S3 Bucket. We recommend creating an available Bucket in the target region in advance to store Iceberg table data for the long term.

### Step 2: Create an AutoMQ Instance and Enable the Table Topic Feature

To use the AutoMQ Table Topic feature, it must be enabled during instance creation to facilitate streaming data into the lake. Refer to the following configuration when creating the instance:

1. Enable Table Topic.

2. Select **Glue** as the Catalog type.

3. Set the S3 Bucket used by Glue Catalog. We recommend selecting the data lake Bucket prepared in Step 1.

<Tip>
  **Note:**

  After enabling the Table Topic for an AutoMQ instance, not all Topics will automatically activate the table transformation feature. Configuration is required at the individual Topic level to stream data into the lake as needed.

  To utilize Table Topics, you must enable them during the instance creation process. After an instance is created, this setting cannot be modified.
</Tip>

<img alt="AutoMQ instance creation page with Table Topic feature enabled" />

### Step 3: Create a Topic and Configure a Materialized View Table

Once the Table Topic feature is enabled in the AutoMQ instance, you can set up a materialized view table as necessary when creating a Topic. Follow these steps:

1. Navigate to the instance from Step 2, find the Topic list, and click **Create Topic**.

2. In the configuration for creating the Topic, enable Table Topic conversion and configure the following parameters:

   1. **Namespace:** A namespace serves to segregate different Iceberg tables, analogous to a Database in the Data Catalog. It's advisable to assign parameter values based on business ownership.

   2. **Schema Constraint Type:** This setting determines whether a Topic message complies with Schema constraints. Selecting **Schema** enables these constraints, requiring message Schemas to be registered in AutoMQ's built-in SchemaRegistry. Messages must then be strictly formatted according to the Schema, and upcoming Table Topics will use this Schema's fields to populate the Iceberg table. Choosing **Schemaless** signifies that the message content lacks explicit Schema constraints; in this scenario, the message Key and Value are used as a single field to populate the Iceberg table.

<img alt="Topic creation form with Table Topic conversion enabled, showing Namespace and Schema Constraint Type fields" />

3. **Click Confirm** to create a Topic that supports streaming tables.

### Step 4: Produce Messages and Query Iceberg Table Data in Real-time

After configuring the AutoMQ instance and creating the Table Topic, you can proceed with testing data production and real-time querying of data in Iceberg tables.

1. Click to enter the Topic details, navigate to the Produce Message tab, input the test message Key and message Value, and send the message.

2. Go to the [AWS Glue Console](https://ap-southeast-1.console.aws.amazon.com/glue/home) to view the Iceberg database and table created by AutoMQ.

<img alt="AWS Glue Console showing the Iceberg database and table created by AutoMQ" />

1. **Click View Data** to enable AWS Athena to query the table data in Glue, allowing you to see how AutoMQ transforms Kafka messages into corresponding data records in real-time. Users can also utilize other query engines for analysis and computation.

<img alt="AWS Athena query results showing AutoMQ Kafka messages converted to Iceberg table records in real time" />


# Table Topic Integration with AWS S3 Table
Source: https://docs.automq.com/automq-cloud/table-topic/table-topic-integration-with-aws-s3-table

Integrate AutoMQ with AWS S3 Table for streaming data lake analysis without ETL. Achieve cloud-native scalability and Kafka compatibility with AutoMQ's Table Topic feature.

AutoMQ Table Topic integrates with Iceberg for streaming data analytics in lakes, removing the necessity for ETL configuration and upkeep. This article guides you through configuring the integration of Table Topic with the AWS S3 Table Bucket in an AWS environment.

## Prerequisites

To utilize the AutoMQ Table Topic feature in an AWS environment, the following conditions must be fulfilled:

* **Version Constraint:** The AutoMQ instance version must be >= 1.4.1.

* **Instance Constraint:** The Table Topic feature needs to be enabled during the creation of the AutoMQ instance, as it cannot be enabled retrospectively once the instance is established.

* **Resource Requirements:** On AWS, when using the Table Topic feature, you can leverage AWS Glue as the Data Catalog or utilize an AWS S3 Table Bucket for the Data Catalog.

## Operation Steps

### Step 1: Create an S3 Table Bucket

To integrate AutoMQ with the S3 Table Bucket, you need to first create a Table Bucket through the AWS S3 console. Ensure it's in the same deployment region as AutoMQ. This Table Bucket will be configured when you create the AutoMQ instance.

<img alt="AWS S3 Console showing the Table Bucket creation page" />

### Step 2: Create an AutoMQ Instance and Enable the Table Topic Feature

The AutoMQ Table Topic feature must be enabled during the creation of the instance to support streaming data into the lake. Therefore, follow the instructions below when configuring the instance:

1. Enable Table Topic.

2. Select **S3 Table** as the Catalog type.

3. Set the S3 Table Bucket used by the Catalog. We recommend selecting the Table Bucket created in Step 1.

<Tip>
  **Note:**

  After enabling Table Topic in an AutoMQ instance, not all Topics will automatically activate stream tables. You still need to configure each Topic individually to facilitate streaming data into the lake.

  To use Table Topic, you must enable it while creating the instance. Once the instance is created, this configuration cannot be altered.
</Tip>

<img alt="AutoMQ instance creation page with Table Topic feature enabled" />

### Step 3: Create Topic and Configure Stream Table

Once Table Topic functionality is enabled in the AutoMQ instance, you can configure stream tables as needed during the Topic creation process. The specific steps are as follows:

1. Access the instance in Step 2, find the Topic list, and click **Create Topic**.

2. In the Topic creation configuration, enable Table Topic conversion and configure the following parameters:

   3. **Namespace:** The namespace is used to isolate different Iceberg tables and corresponds to the Database in the Data Catalog. It is recommended to set this parameter based on business affiliation.

   4. **Schema Constraint Type:** Specifies whether Topic messages comply with schema constraints. By selecting **Schema**, you activate schema constraints, which necessitate that message schemas be registered with the AutoMQ built-in SchemaRegistry. Any subsequent message sent must strictly conform to the schema, and the Table Topic will then utilize the fields from this schema to populate the Iceberg table. If **Schemaless** is chosen, it signifies that the message content lacks explicit schema constraints; in this scenario, the message Key and Value are collectively used to populate the Iceberg table.

<img alt="Topic creation form with Table Topic conversion enabled, showing Namespace and Schema Constraint Type fields" />

5. **Click Confirm** to create a Topic that supports streaming to tables.

### Step 4: Produce Messages and Query Iceberg Table Data in Real-Time

After configuring the AutoMQ instance and creating Table Topics, you can test data production and query data in the Iceberg table.

1. Click to enter the Topic details, navigate to the Produce Messages tab, input the test message Key and message Value, and send the message.

2. Visit the [AWS](https://ap-southeast-1.console.aws.amazon.com/s3/table-buckets/)[S3 Console](https://ap-southeast-1.console.aws.amazon.com/s3/table-buckets/) to view the Iceberg database and tables written by AutoMQ.

<img alt="AWS S3 Console showing the Iceberg database and tables written by AutoMQ" />

3. **Click Query Table from Athena** to enable AWS Athena to query the table data in the Table Bucket. You can observe how AutoMQ converts Kafka messages into corresponding data table records in real-time. Users can also utilize other query engines for analysis and computation.

<img alt="AWS Athena query results showing AutoMQ Kafka messages converted to S3 Table Bucket records in real time" />