Skip to Main Content

Prepare VPC

All components of the AutoMQ Cloud BYOC environment are deployed within the user's VPC to ensure data privacy and security. Preparing the VPC beforehand is necessary before installing AutoMQ. This article explains how to create the VPC required for AutoMQ on AWS Public Cloud.

In this article, any mention of AutoMQ product service provider, AutoMQ service provider, or AutoMQ refers specifically to AutoMQ HK Limited and its subsidiaries.

VPC Requirements​

For POC testing scenarios, it is recommended to create a new VPC and deploy AutoMQ along with the test programs within this new VPC. For production scenarios, an existing VPC is often already in place, in which case it is necessary to check if the VPC meets AutoMQ's installation requirements and make the necessary changes and adjustments.

Installing the AutoMQ BYOC environment requires providing a VPC on AWS that meets the following conditions:

VPC Checklist
Description
Public Subnet: 1
  • The public subnet is used to deploy the AutoMQ BYOC environment console. The environment console provides a WebUI, which users need to access via the public network.
Private Subnet: 1 or 3
  • The private subnet is used to deploy the AutoMQ data plane cluster.
  • AutoMQ supports single-zone deployment and three-zone deployment, with each availability zone requiring a subnet.
S3 Endpoint: 1
  • All AutoMQ components access S3 through the S3 endpoint using the internal network, avoiding public network transmission.
  • The S3 endpoint is created at the VPC level, and the endpoint route table needs to cover all internal subnets.
EC2 Endpoint: 1
  • All AutoMQ components access EC2 and ASG APIs through the EC2 endpoint using the internal network.
  • The EC2 endpoint is created at the VPC level, and the effect range needs to cover all internal subnets.
DNS Hostnames and Resolution: Enabled
  • AutoMQ uses Route 53 to allocate VPC internal DNS resolution. Therefore, the VPC needs to enable DNS hostnames and DNS resolution.

Prerequisites​

To create a BYOC environment, the cloud account used must be the primary account or an IAM sub-account that has been granted relevant permissions. If you are currently using an IAM sub-account on the AWS console, you need to authorize it before enabling the service. The relevant authorization policies and cloud product list are as follows:

  • AmazonVPCFullAccess: Permissions to manage the VPC.

  • AmazonS3FullAccess: Permissions to manage S3.

  • AmazonRoute53FullAccess: Permissions to manage Route 53.

The above permission policies are AWS's recommended default system policy templates. When granting permissions, users can narrow the scope based on their management habits to achieve fine-grained authorization.

Scenario 1: Create a New VPC​

If there are no available VPC resources, or in a POC testing scenario, you want to use a brand-new VPC, you can refer to the documentation below to create a new VPC.

Operation Process​

Step 1: Create a VPC and Enable the S3 Endpoint​

Log in to the VPC console using your AWS cloud account. Taking the Singapore region as an example, visit the VPC console, and click "Create VPC."

When using the quick create option for VPC, you can choose to create the VPC and other resources simultaneously. Confirm the following options and click "Create":

  1. VPC Name: Set the VPC name to a custom name.

  2. CIDR: Set an appropriate CIDR subnet.

  3. Availability Zones: It is recommended to select 1-3 zones based on your business deployment needs.

  4. Public Network Configuration: Configure this if you need to access the AutoMQ environment console from the public network.

  5. VPC Endpoint: Ensure configuration is enabled and create an S3 gateway.

  6. DNS Options: Make sure to enable DNS hostnames and DNS resolution, as subsequent Kafka cluster access points will rely on DNS services for domain name resolution.

Step 2: Create EC2 Endpoint​

Create an EC2 Endpoint within the current VPC to facilitate subsequent AutoMQ data nodes accessing the EC2 API via the VPC private network.

When creating an EC2 Endpoint, pay attention to the following configuration items:

  1. Service Type: Select AWS Services.

  2. VPC: Select the VPC where AutoMQ needs to be deployed.

  3. Availability Zone and Subnet: Select all the availability zones and subnets where AutoMQ needs to be deployed.

  4. Security Group: It is recommended to create a new security group, and ensure to set inbound rules to allow access from 0.0.0.0/0 on port 443.

    AutoMQ clusters require access to the EC2 API via the VPC internal network. Therefore, when creating an EC2 Endpoint, ensure that the subnet configuration and security group configuration for the availability zones are complete. Otherwise, the AutoMQ cluster will fail to start.

    All subnets and availability zones planned for AutoMQ cluster deployment must be covered; no possible subnet should be omitted.

    Security group rules must ensure access to port 443 is smooth and allow sources from 0.0.0.0/0.

Scenario 2: Configuring an Existing VPC​

If there is already an available VPC network in the current production environment and the application has been deployed in the current VPC, you need to install the AutoMQ environment into the existing VPC network. At this time, you still need to check whether the current VPC configuration meets the constraints.

Operation Process​

Step 1: Check and Enable DNS Configuration​

Access the existing VPC network, click on details, and review the VPC configuration.

Ensure that the DNS configuration has been enabled in the current VPC network.

Step 2: Check and Create S3 Endpoints​

Confirm whether the current VPC has an S3 gateway endpoint created. If there is no S3 endpoint, you need to create one.

Create an S3 endpoint (if it was not configured in the previous step).

When creating the S3 endpoint, make sure to add the route table that covers all subnets in the current VPC where Kafka clusters might be deployed. Failure to do so will prevent the Kafka clusters from accessing the S3 service through the internal network.

Step 3: Check and Create EC2 Endpoints​

Check and create EC2 Endpoints within the current VPC to facilitate subsequent data nodes in accessing the EC2 API through the private network.

When creating an EC2 Endpoint, pay attention to the following configuration items:

  1. Service Type: Select AWS Services.

  2. VPC: Select the VPC where AutoMQ needs to be deployed.

  3. Availability Zone and Subnet: Select all the availability zones and subnets where AutoMQ needs to be deployed.

  4. Security Group: It is recommended to create a new security group, and ensure to set inbound rules to allow access from 0.0.0.0/0 on port 443.

Next Steps​

After setting up the VPC network, you can proceed to install the AutoMQ BYOC environment. On AWS, AutoMQ supports installation through the following two methods:

  • Install AutoMQ Environment via Marketplace: Users can install the AutoMQ BYOC environment by selecting the appropriate product link on AWS Marketplace. Install Env from Marketplace▸

  • Install AutoMQ Environment via Terraform Module: Users can install the AutoMQ BYOC environment using the AutoMQ Terraform Module with Terraform. Install Env via Terraform Module▸