Skip to Main Content

Deploy to Google Cloud GKE

Refer to Overview▸, AutoMQ supports deployment on Kubernetes. This article introduces the installation process of deploying AutoMQ clusters on the Google Cloud GKE platform.

In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.

Operation Procedure​

Step 1: Install Environment Console​

Refer to Overview▸, AutoMQ supports deployment on GKE clusters. In the GKE deployment mode, you still need to install the AutoMQ console first, then use the console interface to operate GKE and deploy the cluster to GKE.

On Google Cloud, both Marketplace and Terraform are supported for installing the environment console.

After the AutoMQ console installation is complete, you need to obtain the environment console address, initial username, password, and the service account required by AutoMQ from the console interface or Terraform output menu. This service account will be used in Step 2 to create the GKE node pool.

Step 2: Create GKE Cluster​

Refer to Overview▸, users need to create a separate GKE cluster in advance for AutoMQ use. Users can access the Google Cloud GKE product console and follow the steps below.

Configure GKE Cluster Type and Region​

  1. Log in to the Google Cloud GKE Console. Click Create Cluster.
  1. Select the cluster type as Standard: You manage your cluster.

  1. Fill in the cluster name and region information:

    • Location Type: Choose Regional so you can create an AutoMQ instance with three availability zones later.

    • Target Release Channel: It is recommended to choose the stable version.

Configure GKE Cluster Network​

  1. Switch to Networking Configuration and select the Target VPC and subnet.

Note:

The default Firewall policy between multiple subnets in Google Cloud does not allow intercommunication by default. Therefore, it is recommended to install both GKE and the AutoMQ environment console in the same subnet.

If the GKE, AutoMQ environment console, and Kafka client applications are not in the same subnet, you need to go to VPC Firewall Policies.

If you need to manually add firewall policies, it is recommended to allow all IP ranges of the subnet where the AutoMQ environment console resides. The port list includes: 9090, 9092-9093, 9102-9103, 22.

  1. Modify the network configuration. Ensure the following options are set:

    1. Enable VPC-native traffic routing.

    2. Enable Dataplane V2.

Set up the GKE Cluster Public Node Pool​

Switch to the NODE POOLS configuration item, configure the default node pool for deploying GKE system components. Adjust the number of nodes to 1 per available zone, ensuring the total number of nodes is at least 3. Apart from the configuration shown in the figure, other options for the GKE system components node pool can use the default configuration.

Add an AutoMQ Dedicated Node Pool​

Refer to Overview▸, users need to create a dedicated node pool for AutoMQ to apply for machines for subsequent deployment instances. Therefore, when creating a GKE cluster, you need to create 2 node pools: one is a public node pool for deploying GKE system components, and the other is an AutoMQ dedicated node pool for deploying the AutoMQ cluster.

  1. Click the Add Node Pool button above to configure the AutoMQ dedicated node pool.
  1. Enable Cluster AutoScaler and configure it to Balanced type, with the size limit set to Total limits. Set an appropriate upper limit (recommended upper limit is 50-100).
  1. In the Nodes tab, refer to Overview▸ to configure the list of machine types that meet the AutoMQ requirements.
  1. In the Security tab, bind the Service Account returned by the Output when installing the BYOC environment. If using the Marketplace installation environment, you can find the service account in the Output information from Solution Deployments.

The node security configuration is as follows:

  1. In the Metadata tab, set the taint for AutoMQ exclusive use. The key of the taint is dedicated, the value is automq, and the effect is NO_SCHEDULE.

Click to Create the GKE Cluster​

  1. After verifying all configurations are correct, click the Create button below to create the GKE cluster.

  2. Once the GKE cluster is created, copy the cluster Name for use in step 3 configuration.

  1. Locate the GKE cluster in the cluster list and add a tag to it. This tag will be used to manage permissions in the subsequent AutoMQ console.

AutoMQ Console only supports managing GKE clusters that include the automqAssigned: automq tag. Therefore, you need to add the aforementioned tag to your GKE cluster.

Step 3: Access the Environment Console and Configure the Kubernetes Cluster Information.​

When first entering the AutoMQ BYOC console, you need to configure the Kubernetes cluster information before normal usage. Follow the console guide page to fill in the GKE Cluster Name created in step 2 to complete the environment initialization.

Note:

If the AutoMQ environment console (Kafka client application) and GKE cluster are deployed in different VPC subnets, users need to go to the VPC firewall page to add traffic rules. Otherwise, network issues may impact the creation and usage of instances.

It is recommended to allow all IP sources from the subnet where the environment console is located in the added firewall policy. The port list is: 9090, 9092-9093, 9102-9103, 22.