Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.automq.com/llms.txt

Use this file to discover all available pages before exploring further.

In cloud environments like AWS and GCP, cross-AZ traffic incurs data transfer fees. AutoMQ can help avoid the traffic generated by Kafka’s production, consumption, and server-side replica replication. To configure AutoMQ to save on cross-AZ data transfer costs, the following conditions must be met:
  1. Cloud Provider Environment Constraints: It is only recommended to configure AutoMQ’s multi-AZ write feature when deploying in a multi-AZ setup on AWS or GCP. If your application architecture is single-AZ or you are using other cloud providers (where cross-AZ traffic may be free or lower-cost depending on the cloud provider, region, and pricing policy), this configuration is not recommended.
  2. Multi-AZ Deployment Constraints: To use AutoMQ’s multi-AZ write architecture to save on cross-AZ traffic, both the application and AutoMQ must be deployed in a multi-AZ setup.
    1. AZ Quantity Constraint: Currently, only three-AZ deployments for Producers, Consumers, and AutoMQ are supported. If Producers and Consumers are in more than three AZs, some cross-AZ traffic will still be generated.
    2. AZ Capacity Balance Constraint: Currently, only balanced AZ deployments are supported, meaning the capacity of Producer and Consumer applications is the same in each AZ. If the capacity is unbalanced, some cross-AZ traffic will still be generated.

​
Create an AutoMQ Instance

To use AutoMQ to save on cross-AZ data transfer costs, you first need to ensure that you create a compliant AutoMQ instance. The specific steps are as follows:
  1. Log in to the AutoMQ BYOC environment.
  2. Create an instance, selecting three availability zones in the network configuration. Choose S3WAL or FSWAL as the WAL type.
Note:When creating an AutoMQ instance, the AZs must be consistent with the deployment regions of the Kafka producers and consumers, and the capacity within all AZs must be balanced. If the capacity is not balanced, a certain proportion of cross-AZ transfer traffic may still be generated.For example, if Kafka producers and consumers are deployed in three regions, A, B, and C, and the application capacity in these three regions is the same, then when creating the AutoMQ instance, you should also select regions A, B, and C.

​
Configure Mapping Rules to Identify Client’s Source CIDR Rack

Due to the limitations of the Kafka protocol, the server cannot perceive the Rack to which the Client belongs. To perceive the Client’s Rack and avoid cross-AZ production and consumption, AutoMQ provides two Rack configuration methods: client-side configuration▸ and server-side dynamic configuration. This chapter introduces the server-side dynamic configuration method using automq.zone.cidr.blocks.
In a cloud environment, Rack is usually set to ZoneId. The concepts of Rack and Zone mentioned later can be used interchangeably.
The format for automq.zone.cidr.blocks is as follows: 
# ex. use1-az1@10.0.0.0/19,10.0.32.0/19<>use1-az2@10.0.64.0/19<>use1-az3@10.0.96.0/19
$zone1@$CIDRBlock1,$CIDRBlock2<>$zone2@$CIDRBlock3<>$zone3@$CIDRBlock4,$CIDRBlock5
  • Configurations for different Zones are separated by <>.
  • Each Zone’s configuration starts with the Zone name, followed by @, and then a list of CIDRs for that Zone. CIDRs in the list are separated by ,.
AutoMQ will match the Client’s connecting IP with the CIDRs sequentially to determine the Client’s Zone. The server-side configuration automq.zone.cidr.blocks is suitable for the following scenarios:
  • There are clear CIDR segments between the Zones where the Clients are located.
  • It is inconvenient to set different Client configurations based on the load’s Zone for computing engines like Flink, Spark, or Presto.
If both server-side and client-side configurations are set, the priority is as follows: client.id > automq.zone.cidr.blocks > Consumer client.rack.