Skip to main content
All components of the AutoMQ Cloud BYOC environment are deployed within your AWS account to ensure data privacy and security. Before installing AutoMQ, you need to prepare various AWS resources in advance. This article describes how to prepare the necessary cloud resources for AutoMQ on the AWS public cloud.
In this document, “AutoMQ,” “AutoMQ service provider,” and “the AutoMQ service” all refer to AutoMQ HK Limited and its affiliates.

​
Prepare VPC

You can choose to use an existing VPC when installing AutoMQ, or create a new one for it. For production environments, we recommend checking and adjusting your existing VPC configuration to meet AutoMQ’s requirements. For POC testing, creating a new VPC is a quicker option.

​
VPC Requirements

To deploy AutoMQ BYOC environment on AWS, your VPC must meet the following conditions:

​
Scenario 1: Create a New VPC

If you need to create a new VPC for AutoMQ, we recommend using the VPC and more feature in the AWS VPC console.
  1. Log in to the AWS VPC console.
  2. In the left navigation pane, choose Your VPCs, then click Create VPC.
  3. In VPC settings, select VPC and more.
  4. In the Name tag auto-generation section, set a clear name for your VPC.
  5. Configure the IPv4 CIDR block, Number of Availability Zones, Number of public subnets, and Number of private subnets.
  6. Depending on your deployment needs, choose whether to create NAT gateway(s) and VPC endpoints.
    • NAT Gateway: If you plan to deploy on EKS, select at least 1 per AZ.
    • VPC Endpoints: Select S3 Gateway, which will automatically create a gateway-type S3 endpoint.
  7. Ensure that both Enable DNS hostnames and Enable DNS resolution under DNS options are checked.
  8. Click Create VPC.
After the VPC is created, you also need to manually create an EC2 interface endpoint for it.
  1. In the AWS VPC console, choose Endpoints from the left navigation pane.
  2. Click Create endpoint.
  3. Name tag: Name the endpoint.
  4. Service category: Select AWS services.
  5. Services: Search for and select the com.amazonaws.<region>.ec2 interface service.
  6. VPC: Select the VPC you just created.
  7. Subnets: Select all the private subnets intended for AutoMQ deployment.
  8. Security groups: Select or create a security group and add an inbound rule to allow HTTPS (443) traffic from your VPC’s CIDR.
  9. Click Create endpoint.

​
Scenario 2: Configure an Existing VPC

If you are deploying AutoMQ in an existing VPC, verify that the following configurations meet the requirements.

​
1. Check DNS Configuration

  1. In the AWS VPC console, choose Your VPCs from the left navigation pane.
  2. Select your target VPC and view the Details tab.
  3. Confirm that the values for DNS hostnames and DNS resolution are both Enabled. If not, click Actions -> Edit VPC settings to modify them.

​
2. Check S3 Endpoint

  1. In the AWS VPC console, choose Endpoints from the left navigation pane.
  2. Search for and check if there is an S3 endpoint of type Gateway (service name com.amazonaws.<region>.s3) associated with your VPC.
  3. If it doesn’t exist, click Create endpoint to create one.
    • When creating it, ensure its route table (Route tables) is associated with all private subnets that will be used for deploying AutoMQ.

​
3. Check EC2 Endpoint

  1. On the Endpoints page, continue to search for and check if there is an EC2 endpoint of type Interface (service name com.amazonaws.<region>.ec2) associated with your VPC.
  2. If it doesn’t exist, click Create endpoint to create one.
    • Subnets: Ensure you select all private subnets that will be used for deploying AutoMQ.
    • Security groups: Select or create a security group and add an inbound rule to allow HTTPS (443) traffic from your VPC’s CIDR.
    • Enable DNS name: Make sure to check Enable DNS name to enable Private DNS resolution, ensuring that AutoMQ components can correctly access the EC2 API via private DNS names.

​
Prepare Private DNS (Optional)

The AutoMQ cluster exposes service addresses to applications through Private DNS, which relies on private domain name resolution within the VPC. Therefore, a Private DNS must be configured to provide reliable domain name resolution services. You can choose to have the AutoMQ console create the Private DNS Zone automatically or prepare it yourself by following the tutorial below.

​
Private DNS Zone Requirements

When configuring Private DNS, the following two key conditions must be met:

​
Private DNS Configuration Steps

​
Step 1: Check Existing Private DNS

  1. Log in to the AWS console and visit the Route53 console.
  2. In the left navigation bar, select Hosted zones.
  3. Check if a Private DNS associated with the target VPC already exists.

​
Step 2: Create or Configure Private DNS

If a Private DNS already exists:
  1. Select the target Private DNS and click Details.
  2. Confirm that the Type is Private hosted zone.
  3. Check the VPCs section to ensure it includes the VPC where AutoMQ is deployed.
  4. Record the domain name information for subsequent TLS feature configuration.
If you need to create a new Private DNS:
  1. Click Create hosted zone.
  2. Configure the basic information:
    • Domain name: Enter a private domain name (e.g., internal.example.com).
    • Type: Select Private hosted zone.
    • VPC: Select the VPC where AutoMQ is deployed and the corresponding region.
  3. Click Create hosted zone.

​
Step 3: Verify the Configuration

  1. On the Private DNS details page, confirm the VPC association status.
  2. Check if the domain name configuration meets the planning requirements for the TLS feature.
  3. Record the ID and domain name of the Private DNS, which will be provided to the AutoMQ components during subsequent installation.

​
Prepare Cluster Security Group (Optional)

During the cluster deployment process, you can opt to use a pre-configured security group for precise network access control. If you choose this method, you must create a security group in advance and configure its inbound rules according to the table below. If you prefer to simplify network configuration, you can choose to have AutoMQ automatically create a new security group with all the necessary rules during cluster creation, in which case you can skip this section.
Notes on the access sources mentioned in the table above:
  • Console: It is recommended to configure this as the Subnet CIDR where the AutoMQ console is located.
  • Internal cluster components: Select the Subnet CIDR chosen when creating the cluster instances.
  • Applications: Refers to the source from which applications access the Kafka service. Configure this based on your actual business requirements.

​
Configure cross-VPC access

An AutoMQ instance that uses a Route 53 private hosted zone can be resolved only from VPCs associated with that hosted zone. To access the instance from an application in another VPC, complete both of the following tasks:
  1. Establish network connectivity between the AutoMQ VPC and the application VPC, for example by using VPC peering or AWS Transit Gateway. Update the relevant route tables and security groups to allow Kafka traffic.
  2. Associate the Route 53 private hosted zone used by the AutoMQ instance with the application VPC.

​
Find the private hosted zone

Copy the instance endpoint from the AutoMQ Console and identify its domain suffix. In the Route 53 console, choose Hosted zones, open the matching private hosted zone, and record its hosted zone ID. Find the Route 53 private hosted zone ID used by the AutoMQ instance.

​
Associate a VPC in the same AWS account

On the private hosted zone details page, edit the VPC associations and add the application VPC and its AWS Region. Associate a VPC in the same AWS account with the private hosted zone.

​
Associate a VPC in a different AWS account

For a cross-account association, first use credentials for the account that owns the private hosted zone to authorize the application VPC:
Then use credentials for the account that owns the application VPC to create the association:
Allow several minutes for the association and DNS records to propagate. Then verify that a host in the application VPC can resolve the AutoMQ endpoint and connect to the configured Kafka listener.

​
Prepare FSx File System Security Group (Optional)

If you plan to use AutoMQ in an AWS environment and leverage FSx for NetApp ONTAP as the storage backend for the WAL (Write-Ahead Log), you need to configure a dedicated security group for the FSx file system. This section details the required inbound rules for that security group. If you prefer AutoMQ to automatically create and configure this security group during deployment, you can skip this section.
Notes on the access sources mentioned in the table above:
  • Console: It is recommended to configure this as the Subnet CIDR where the AutoMQ Console is located.
  • Internal cluster components: Select the Subnet CIDR chosen when creating the cluster instances.

​
Prepare EFS File System Security Group (Optional)

If you plan to use AutoMQ in an AWS environment and leverage EFS (Elastic File System) as the storage backend for NFS WAL, you need to configure a dedicated security group for the EFS file system. This section details the required inbound rules for that security group. If you prefer AutoMQ to automatically create and configure this security group during deployment, you can skip this section.
Notes on the access sources mentioned in the table above:
  • Internal cluster components: Select the Subnet CIDR chosen when creating the cluster instances.