Skip to Main Content

Deploy to Google Cloud GKE

Refer to Overview▸, AutoMQ supports deployment on K8s. This article details the installation process of deploying AutoMQ on the Google Cloud GKE platform.

In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.

Prerequisites: Prepare VPC Network​

The AutoMQ BYOC environment is deployed within the user's VPC to ensure data privacy and security. Before installing the AutoMQ environment, a usable VPC must be prepared. The VPC must meet the following conditions:

  • Zone requirements: Depending on business needs, if you need to create instances across three zones later, you must allocate at least three zones and subnets.

  • Subnet CIDR Requirements: AutoMQ data plane clusters are deployed using the user's VPC and will consume the VPC subnet's IPs. It is recommended to allocate a sufficient number of available IPs in each subnet (at least 1000 available IPs per subnet) to avoid issues during future deployments and scaling of clusters.

  • Public Internet Access Requirements: Deploying the AutoMQ data plane cluster requires pulling Docker images and Helm Chart artifacts from the public internet. Therefore, the VPC environment needs to support public internet access via SNAT or similar methods.

Operation Procedure​

Step 1: Install AutoMQ Environment Console​

On Google Cloud, you can install the AutoMQ environment console using Terraform. Users can follow the instructions in the document Install Env via Terraform Module▸.

After installing AutoMQ, obtain the environment console address, initial username, password, and the service account required by AutoMQ from the Terraform output menu. This service account will be used in Step 2 to create the GKE node pool.

Step 2: Create GKE Cluster​

Refer to Overview▸, users need to create a dedicated GKE cluster assigned for AutoMQ usage in advance. Users can follow the steps on the Google Cloud GKE product console or refer to the Terraform template provided by AutoMQ for quick creation.

Creation Method 1: Create GKE Cluster Via GKE Console​

  1. Log in to the Google Cloud GKE Console. Click Create Cluster.
  1. Select the cluster type as Standard: You manage your cluster.

  1. Fill in the cluster name and region information:

    • Location Type: Choose Regional so you can create an AutoMQ instance with three availability zones later.

    • Target Release Channel: It is recommended to choose the stable version.

  1. Switch to the Networking Configuration, and select the target VPC and subnet.

Note:

The default Firewall policy between multiple subnets in Google Cloud does not allow intercommunication by default. Therefore, it is recommended to install both GKE and the AutoMQ environment console in the same subnet.

If the GKE, AutoMQ environment console, and Kafka client applications are not in the same subnet, you need to go to VPC Firewall Policies.

If you need to manually add firewall policies, it is recommended to allow all IP ranges of the subnet where the AutoMQ environment console resides. The port list includes: 9090, 9092-9093, 9102-9103, 22.

  1. Modify the network configuration. Ensure the following options are set:

    1. Enable VPC-native traffic routing.

    2. Enable Dataplane V2.

  1. Navigate to the NODE POOLS Configuration and modify the default node pool configuration with the following parameters.

Enable Cluster AutoScaler. Configure it as Balanced type for load balancing. Set the size limit to Total limits and set an appropriate upper limit (it is recommended to set the upper limit to 50-100).

In the Nodes tab, refer to Overview▸ to configure the list of machine types that meet AutoMQ requirements.

In the Security tab, bind the Service Account returned during the installation of the BYOC environment in step 1.

  1. After verifying all configurations, click Create at the bottom to create the GKE cluster.

  2. Once the GKE cluster is created, copy the Cluster Name for use in step 3.

  1. In the GKE cluster list, locate the cluster and add a tag to it to define the subsequent management permissions for the AutoMQ console.

AutoMQ Console only supports managing GKE clusters that include the automqAssigned: automq tag. Therefore, you need to add the aforementioned tag to your GKE cluster.

Step 3: Access the Environment Console and Configure the K8s Cluster Information​

When first accessing the AutoMQ BYOC console, you need to configure the K8s cluster information before it can be used normally. Follow the instructions on the console guidance page to fill in the GKE Cluster Name created in Step 2 and complete the environment initialization.

Note:

If the AutoMQ environment console (Kafka client application) and GKE cluster are deployed in different VPC subnets, users need to go to the VPC firewall page to add traffic rules. Otherwise, network issues may impact the creation and usage of instances.

It is recommended to allow all IP sources from the subnet where the environment console is located in the added firewall policy. The port list is: 9090, 9092-9093, 9102-9103, 22.