Deploy to Google Cloud GKE

As described in Overview▸, AutoMQ supports deployment on Kubernetes. This article outlines the installation process for deploying an AutoMQ cluster on the Google Cloud GKE platform.

In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.

Install the Environment Console

As described in Overview▸, AutoMQ supports deployment on GKE clusters. In the GKE deployment mode, you still need to first install the AutoMQ console, and then use the console interface to manage GKE for deploying the cluster on GKE. On Google Cloud, the 8.x AutoMQ console is installed and started with a Docker image. For console installation instructions, see Install AutoMQ on Google Cloud▸. After the AutoMQ console installation is complete, use the console address, initial username, and password to log in to the console, and follow the page instructions to complete permission initialization.

Prepare the GKE Cluster and Required Node Pools

If you want to run AutoMQ on Kubernetes, prepare a GKE cluster for AutoMQ. If you only want to deploy AutoMQ in IaaS mode, you do not need to prepare a GKE cluster.

Step 1: Create a GKE Standard cluster

For instructions on creating a GKE Standard cluster, see the Google Cloud documentation Creating a regional cluster. This document does not repeat the full GKE creation procedure. It only lists the key configuration requirements for AutoMQ deployment. AutoMQ currently supports only GKE Standard clusters. GKE Autopilot is not supported. When you create the cluster, confirm the following settings:

Cluster type: Select GKE Standard.
Cluster location: A Regional cluster is recommended so that AutoMQ instances can span multiple zones.
Network mode: Enable VPC-native traffic routing and prepare secondary IP ranges for Pods and Services.
Dataplane: Dataplane V2 is recommended.
Identity model: Enable Workload Identity Federation for GKE.
Node metadata: Do not disable the GKE Metadata Server. AutoMQ Pods depend on it when a Kubernetes ServiceAccount is bound to a Google Service Account (GSA).

If you use Shared VPC, use the following project boundary:

Service project: Owns the GKE cluster, Console VM, AutoMQ instances, data buckets, and instance GSAs.
Host project: Owns the VPC, subnets, routes, NAT, firewall rules, and other network resources.

In Shared VPC mode, the GKE service agent also needs permissions in the host project to use the shared network. See the Google Cloud documentation Setting up clusters with Shared VPC.

Step 2: Configure network and firewall rules

The AutoMQ console needs to access the GKE API Server and operate Kubernetes resources during instance creation, upgrades, and scaling. The AutoMQ console also needs to access the service ports exposed by the AutoMQ data plane for lifecycle management, status checks, and operations. Whether Kafka clients need to access these ports depends on your application access pattern. If the AutoMQ console, GKE cluster, and Kafka clients are not in the same subnet or cannot communicate by default, add the required VPC firewall rules. At minimum, confirm the following access paths:

The AutoMQ console can access the GKE API Server.
The AutoMQ console can access the AutoMQ Pod or Service ports. This access path is required.
Kafka client networks can access the AutoMQ service ports required by your applications, such as Kafka bootstrap, broker, or related proxy ports.
If the GKE nodes don’t use external IPs and need to access Google APIs such as Cloud Storage, IAM, and Cloud DNS, the node subnet has Private Google Access enabled and the required routes and DNS are configured for Google APIs. Access to external image registries or other non-Google endpoints still requires Cloud NAT or another egress path.

Common AutoMQ service ports include: 8083, 9090, 9092, 9093, 9102, 9103, 9112, and 9113. The console subnet is a required source. Add Kafka client source CIDR ranges only when those clients need application access. Restrict the source CIDR ranges based on the console subnet and client networks. If the AutoMQ workload node pool uses a dedicated node VM service account, you can restrict the target to that node pool by service account. Otherwise, use network tags or another targeting method to narrow the target scope. The following command shows how to allow common AutoMQ ports from the console source CIDR range and optional client source CIDR ranges, while narrowing the target when a dedicated node VM service account is used. In Shared VPC mode, create firewall rules in the host project.

HOST_PROJECT_ID="<host-project-id>"
VPC_NAME="<vpc-name>"
AUTOMQ_NODE_SA_EMAIL="<dedicated-automq-node-service-account-email>"
# If you don't need to allow Kafka client CIDR ranges yet, keep only the console subnet CIDR.
SOURCE_CIDRS="<console-subnet-cidr>,<client-cidr>"

gcloud compute firewall-rules create "automq-gke-allow-automq-service" \
  --project="${HOST_PROJECT_ID}" \
  --network="${VPC_NAME}" \
  --direction=INGRESS \
  --action=ALLOW \
  --source-ranges="${SOURCE_CIDRS}" \
  --target-service-accounts="${AUTOMQ_NODE_SA_EMAIL}" \
  --rules=tcp:8083,tcp:9090,tcp:9092,tcp:9093,tcp:9102,tcp:9103,tcp:9112,tcp:9113

Step 3: Prepare the AutoMQ workload node pool

When you create a GKE Standard cluster, it has a node pool that can run GKE system components. Prepare a separate AutoMQ workload node pool, and add the dedicated=automq:NoSchedule taint only to the AutoMQ workload node pool.

AutoMQ Brokers should run on a dedicated node pool. Prepare this node pool in advance, and select or enter the node pool name during instance creation. When you prepare the AutoMQ workload node pool, confirm the following settings:

GKE Metadata Server: Must be enabled.
Taint: Set dedicated=automq:NoSchedule so that only AutoMQ workloads are scheduled to this node pool.
Machine type and size: Select machine types based on the node pool requirements in Overview▸, and set the minimum, desired, and maximum node counts based on workload size.
Autoscaling: Cluster Autoscaler with the Balanced policy is recommended.

AutoMQ on GKE uses Workload Identity to map a Kubernetes ServiceAccount to a Google Service Account (GSA). Make sure the AutoMQ workload node pool has GKE Metadata Server enabled. The node pool VM Service Account and access scopes affect the node itself, but they no longer carry AutoMQ data plane permissions.

Step 4: Confirm the AutoMQ workload identity

AutoMQ data plane Pods need to access Google Cloud resources such as GCS, Cloud DNS, and Compute. When you create an instance, choose one of the following identity modes:

Identity mode	When to use	What you prepare	What the AutoMQ console does
Managed identity	Recommended for new environments where the console should create and maintain the instance identity.	Make sure control-plane permission initialization has passed.	Creates the instance GSA and configures the required IAM bindings for GCS, Cloud DNS, Workload Identity, and related resources.
Existing identity	Use when your organization already has GSA naming, audit, or approval workflows.	Create the GSA in advance and grant the permissions shown in the console. The GSA resource name uses `projects/<project-id>/serviceAccounts/<service-account-email>`.	Verifies the GSA is readable and uses it in the instance configuration.

If you use an existing identity, also prepare the Kubernetes namespace and Kubernetes ServiceAccount, and make sure the ServiceAccount can use the GSA through Workload Identity. See Appendix: Configure GKE Workload Identity for AutoMQ workloads.

Create an AutoMQ Instance and Select Kubernetes Deployment

Step 1: Create the instance in the console

Log in to the AutoMQ console, create an instance, select Kubernetes as the deployment type, and configure the following settings.

Deployment type: Select Kubernetes.
Kubernetes cluster: Select the target GKE cluster. If you use an API or automation tool, the GKE cluster ID must be the full resource name: projects/<project-id>/locations/<location>/clusters/<cluster-name>.
Node pool: Select or enter the node pool name for AutoMQ Brokers. The node pool must meet the node pool requirements in Overview▸.
Data bucket: Let AutoMQ create a managed bucket, or enter an existing GCS bucket.
Private DNS Zone: Let AutoMQ create a managed Cloud DNS private managed zone, or enter an existing zone.
Instance identity:
- For managed identity, the AutoMQ console creates the instance GSA and configures the required IAM bindings.
- For existing identity, enter the full resource name of the existing GSA and confirm that the permissions shown in the console have been granted.
Namespace and ServiceAccount: Required only when you manage the Kubernetes identity yourself. The ServiceAccount must be bound to the instance GSA through Workload Identity.
Preview the configuration and complete the creation.

Step 2: Check and configure Placeholders

After creating the instance, use the Placeholder check script to prepare Placeholders for failure recovery. For production, complete this step before sending application traffic to the instance. You need Python 3.9+ and kubectl. Your identity needs read access to Controller StatefulSets and Deployments across all namespaces and cluster-scoped PriorityClasses, permission to create or update the Namespaces, PriorityClasses, and Deployments in the generated file, and read access to Pods in automq-placeholders. Run the commands step by step, following the comments:

# 1. Confirm that the current kubeconfig points to the cluster hosting the instance
kubectl config current-context

# 2. Install the dependency and download the script
python3 -m pip install pyyaml
curl -fL https://www.automq.com/downloads/check-placeholder.py \
  -o check-placeholder.py

# 3. Enter the instance ID from the AutoMQ console and check Placeholder Deployments
INSTANCE_ID="<instance-id>"
python3 check-placeholder.py --instance-id "$INSTANCE_ID"

# If the output file exists, choose a new filename with --output <new-file.yaml>
# 4. If this check generated YAML, review it, then run the kubectl apply command printed by the script
# If every zone shows EXISTS, continue to step 5. For ERROR, resolve the error and retry
# kubectl apply -f placeholder.yaml

# 5. Check Pod status; wait for Running and all containers ready in the READY column
kubectl get pods -n automq-placeholders \
  -l app=low-priority-placeholder -o wide

# More options (context, output filename, image, and others)
# python3 check-placeholder.py --help

In the same Kubernetes cluster, the script reuses Placeholders per Availability Zone. Instances can share an existing Deployment when their configuration for that zone matches, including node matching conditions, instance type, tolerations, and Controller requests, even if they use different additional zones.

Appendix: Configure GKE Workload Identity for AutoMQ workloads

For complete instructions, see the Google Cloud documentation Authenticate to Google Cloud APIs from GKE workloads. If you use an existing GSA, make sure the Kubernetes ServiceAccount can impersonate the GSA. Example steps:

Create the namespace and Kubernetes ServiceAccount.

kubectl create namespace <automq-namespace>
kubectl create serviceaccount <automq-service-account> -n <automq-namespace>

Add a roles/iam.workloadIdentityUser binding to the GSA.

gcloud iam service-accounts add-iam-policy-binding <gsa-email> \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:<project-id>.svc.id.goog[<automq-namespace>/<automq-service-account>]"

Annotate the Kubernetes ServiceAccount with the GSA.

kubectl annotate serviceaccount <automq-service-account> \
  -n <automq-namespace> \
  iam.gke.io/gcp-service-account=<gsa-email>

When you create the AutoMQ instance, enter the namespace, ServiceAccount, and GSA resource name prepared above.

Overview

Getting Started

Deploy on Kubernetes

Subscriptions and Billings

Manage Environments

Manage Identities and Access

Managing AutoMQ

Migrate to AutoMQ

Manage Security

Kafka Connect

Monitoring & Alert

Table Topic

Best Practice

Eliminate Inter-Zone Traffics

Release Notes

Support

Appendix

Install the Environment Console

Prepare the GKE Cluster and Required Node Pools

Step 1: Create a GKE Standard cluster

Step 2: Configure network and firewall rules

Step 3: Prepare the AutoMQ workload node pool

Step 4: Confirm the AutoMQ workload identity

Create an AutoMQ Instance and Select Kubernetes Deployment

Step 1: Create the instance in the console

Step 2: Check and configure Placeholders

Appendix: Configure GKE Workload Identity for AutoMQ workloads

​Install the Environment Console

​Prepare the GKE Cluster and Required Node Pools

​Step 1: Create a GKE Standard cluster

​Step 2: Configure network and firewall rules

​Step 3: Prepare the AutoMQ workload node pool

​Step 4: Confirm the AutoMQ workload identity

​Create an AutoMQ Instance and Select Kubernetes Deployment

​Step 1: Create the instance in the console

​Step 2: Check and configure Placeholders

​Appendix: Configure GKE Workload Identity for AutoMQ workloads

Install the Environment Console

Prepare the GKE Cluster and Required Node Pools

Step 1: Create a GKE Standard cluster

Step 2: Configure network and firewall rules

Step 3: Prepare the AutoMQ workload node pool

Step 4: Confirm the AutoMQ workload identity

Create an AutoMQ Instance and Select Kubernetes Deployment

Step 1: Create the instance in the console

Step 2: Check and configure Placeholders

Appendix: Configure GKE Workload Identity for AutoMQ workloads