Skip to main content
As described in Overview▸, AutoMQ supports deployment on Kubernetes. This article outlines the installation process for deploying an AutoMQ cluster on the Google Cloud GKE platform.
In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.

​
Install the Environment Console

As described in Overview▸, AutoMQ supports deployment on GKE clusters. In the GKE deployment mode, you still need to first install the AutoMQ console, and then use the console interface to manage GKE for deploying the cluster on GKE. On Google Cloud, the 8.x AutoMQ console is installed and started with a Docker image. For console installation instructions, see Install AutoMQ on Google Cloud▸. After the AutoMQ console installation is complete, use the console address, initial username, and password to log in to the console, and follow the page instructions to complete permission initialization.

​
Prepare the GKE Cluster and Required Node Pools

If you want to run AutoMQ on Kubernetes, prepare a GKE cluster for AutoMQ. If you only want to deploy AutoMQ in IaaS mode, you do not need to prepare a GKE cluster.

​
Step 1: Create a GKE Standard cluster

For instructions on creating a GKE Standard cluster, see the Google Cloud documentation Creating a regional cluster. This document does not repeat the full GKE creation procedure. It only lists the key configuration requirements for AutoMQ deployment. AutoMQ currently supports only GKE Standard clusters. GKE Autopilot is not supported. When you create the cluster, confirm the following settings:
  • Cluster type: Select GKE Standard.
  • Cluster location: A Regional cluster is recommended so that AutoMQ instances can span multiple zones.
  • Network mode: Enable VPC-native traffic routing and prepare secondary IP ranges for Pods and Services.
  • Dataplane: Dataplane V2 is recommended.
  • Identity model: Enable Workload Identity Federation for GKE.
  • Node metadata: Do not disable the GKE Metadata Server. AutoMQ Pods depend on it when a Kubernetes ServiceAccount is bound to a Google Service Account (GSA).
If you use Shared VPC, use the following project boundary:
  • Service project: Owns the GKE cluster, Console VM, AutoMQ instances, data buckets, and instance GSAs.
  • Host project: Owns the VPC, subnets, routes, NAT, firewall rules, and other network resources.
In Shared VPC mode, the GKE service agent also needs permissions in the host project to use the shared network. See the Google Cloud documentation Setting up clusters with Shared VPC.

​
Step 2: Configure network and firewall rules

The AutoMQ console needs to access the GKE API Server and operate Kubernetes resources during instance creation, upgrades, and scaling. The AutoMQ console also needs to access the service ports exposed by the AutoMQ data plane for lifecycle management, status checks, and operations. Whether Kafka clients need to access these ports depends on your application access pattern. If the AutoMQ console, GKE cluster, and Kafka clients are not in the same subnet or cannot communicate by default, add the required VPC firewall rules. At minimum, confirm the following access paths:
  • The AutoMQ console can access the GKE API Server.
  • The AutoMQ console can access the AutoMQ Pod or Service ports. This access path is required.
  • Kafka client networks can access the AutoMQ service ports required by your applications, such as Kafka bootstrap, broker, or related proxy ports.
  • If the GKE nodes don’t use external IPs and need to access Google APIs such as Cloud Storage, IAM, and Cloud DNS, the node subnet has Private Google Access enabled and the required routes and DNS are configured for Google APIs. Access to external image registries or other non-Google endpoints still requires Cloud NAT or another egress path.
Common AutoMQ service ports include: 8083, 9090, 9092, 9093, 9102, 9103, 9112, and 9113. The console subnet is a required source. Add Kafka client source CIDR ranges only when those clients need application access. Restrict the source CIDR ranges based on the console subnet and client networks. If the AutoMQ workload node pool uses a dedicated node VM service account, you can restrict the target to that node pool by service account. Otherwise, use network tags or another targeting method to narrow the target scope. The following command shows how to allow common AutoMQ ports from the console source CIDR range and optional client source CIDR ranges, while narrowing the target when a dedicated node VM service account is used. In Shared VPC mode, create firewall rules in the host project.

​
Step 3: Prepare the AutoMQ workload node pool

When you create a GKE Standard cluster, it has a node pool that can run GKE system components. Prepare a separate AutoMQ workload node pool, and add the dedicated=automq:NoSchedule taint only to the AutoMQ workload node pool.
AutoMQ Brokers should run on a dedicated node pool. Prepare this node pool in advance, and select or enter the node pool name during instance creation. When you prepare the AutoMQ workload node pool, confirm the following settings:
  • GKE Metadata Server: Must be enabled.
  • Taint: Set dedicated=automq:NoSchedule so that only AutoMQ workloads are scheduled to this node pool.
  • Machine type and size: Select machine types based on the node pool requirements in Overview▸, and set the minimum, desired, and maximum node counts based on workload size.
  • Autoscaling: Cluster Autoscaler with the Balanced policy is recommended.
AutoMQ on GKE uses Workload Identity to map a Kubernetes ServiceAccount to a Google Service Account (GSA). Make sure the AutoMQ workload node pool has GKE Metadata Server enabled. The node pool VM Service Account and access scopes affect the node itself, but they no longer carry AutoMQ data plane permissions.

​
Step 4: Confirm the AutoMQ workload identity

AutoMQ data plane Pods need to access Google Cloud resources such as GCS, Cloud DNS, and Compute. When you create an instance, choose one of the following identity modes: If you use an existing identity, also prepare the Kubernetes namespace and Kubernetes ServiceAccount, and make sure the ServiceAccount can use the GSA through Workload Identity. See Appendix: Configure GKE Workload Identity for AutoMQ workloads.

​
Create an AutoMQ Instance and Select Kubernetes Deployment

Log in to the AutoMQ console, create an instance, select Kubernetes as the deployment type, and configure the following settings.
  1. Deployment type: Select Kubernetes.
  2. Kubernetes cluster: Select the target GKE cluster. If you use an API or automation tool, the GKE cluster ID must be the full resource name: projects/<project-id>/locations/<location>/clusters/<cluster-name>.
  3. Node pool: Select or enter the node pool name for AutoMQ Brokers. The node pool must meet the node pool requirements in Overview▸.
  4. Data bucket: Let AutoMQ create a managed bucket, or enter an existing GCS bucket.
  5. Private DNS Zone: Let AutoMQ create a managed Cloud DNS private managed zone, or enter an existing zone.
  6. Instance identity:
    • For managed identity, the AutoMQ console creates the instance GSA and configures the required IAM bindings.
    • For existing identity, enter the full resource name of the existing GSA and confirm that the permissions shown in the console have been granted.
  7. Namespace and ServiceAccount: Required only when you manage the Kubernetes identity yourself. The ServiceAccount must be bound to the instance GSA through Workload Identity.
  8. Preview the configuration and complete the creation.

​
Appendix: Configure GKE Workload Identity for AutoMQ workloads

For complete instructions, see the Google Cloud documentation Authenticate to Google Cloud APIs from GKE workloads. If you use an existing GSA, make sure the Kubernetes ServiceAccount can impersonate the GSA. Example steps:
  1. Create the namespace and Kubernetes ServiceAccount.
  2. Add a roles/iam.workloadIdentityUser binding to the GSA.
  3. Annotate the Kubernetes ServiceAccount with the GSA.
When you create the AutoMQ instance, enter the namespace, ServiceAccount, and GSA resource name prepared above.

​
Appendix: Deploy AutoMQ Placeholder Deployment (optional)

If you want to reserve failover capacity in the AutoMQ workload node pool, you can deploy a Placeholder Deployment for the AutoMQ node pool. It runs low-priority placeholder Pods that pre-allocate resources. When a node that hosts an AutoMQ Broker fails, the low-priority Pods can be preempted so that the Broker can be scheduled and recovered faster. You can deploy the Placeholder Deployment by using kubectl or the Kubernetes console.
  1. Download and create the low-priority class declaration.
  2. Download the GKE Placeholder Deployment example.
  3. Modify the key parameters in automq-gke-placeholder.yaml based on the actual AutoMQ workload node pool:
    • metadata.name: Use a meaningful name, such as placeholder-for-automq-nodepool-a.
    • replicas: The number of Placeholder Pods to reserve. For multi-zone deployments, evaluate whether to reserve one Pod per zone.
    • affinity.nodeAffinity: Match the AutoMQ workload node pool. On GKE, use cloud.google.com/gke-nodepool to match the node pool name, or use node.kubernetes.io/instance-type to match the node machine type.
    • resources: Set limits based on the node size. Set requests slightly lower than the node size so that the Placeholder Pods can be scheduled.
  4. Apply the Placeholder Deployment.
  5. Check whether the Placeholder Pods are running on the expected node pool.