Runs on Cloud
AutoMQ is a cloud-native reimagining of the Kafka release, making full use of cloud services such as EBS and S3. The following sections will introduce the support for AutoMQ across major cloud providers.
Dependency on EBS Services
AutoMQ leverages EBS as Write-Ahead-Log (WAL) storage, accessing it as a raw device without the need to mount a file system. EBS is a core product of cloud providers and offers extremely high data durability, often utilizing triple replication or erasure coding for multiple replicas. Services like AWS Nitro and Alibaba Cloud ECS Bare Metal Instance use local hardware acceleration, and EBS server clusters leverage RDMA technology, making EBS latency comparable to local NVME SSD. Additionally, EBS uses dedicated network channels that do not consume the host machine’s network bandwidth. When used appropriately, EBS is highly cost-effective, such as AWS EBS GP3 at 10GB/3000IOPS/125MBps costing less than $1 per month.
Therefore, EBS is an excellent alternative to traditional data storage technologies for WAL.
AutoMQ accesses EBS using standard POSIX APIs, ensuring great compatibility with all mainstream cloud providers' EBS services.
Dependency on Object Storage Services
AutoMQ accesses object storage services using the AWS S3 SDK. Most cloud providers support the standard S3 protocol, with the only exception being Azure. The AutoMQ team will soon release a separate adaptation for Azure, so stay tuned.
The Community Edition of AutoMQ uses the following AWS S3 SDK to access object storage.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.20.127</version>
</dependency>
Dependency on Access Control Policies
When accessing object storage, to ensure security, you can restrict access by setting minimal permissions. The configuration methods are similar across major cloud providers. Refer to Cluster Deployment on Linux▸ for detailed steps.
Testing Object Storage Compatibility
After creating a storage bucket, you can use the following commands to test if the object storage is compatible with AutoMQ.
bin/automq-kafka-admin.sh generate-s3-url \
--s3-access-key=xxx \
--s3-secret-key=yyy \
--s3-region=cn-northwest-1 \
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \
--s3-data-bucket=automq-test \
--s3-ops-bucket=automq-test
List of Supported Cloud Providers
| Cloud Providers | EBS Compatibility | Object Storage Compatibility | Description |
|---|---|---|---|
| AWS | Compatible | Compatible | - |
| Azure | Compatible | Not Compatible | Azure's object storage network protocol is entirely different from AWS S3, requiring separate adaptation. |
| Google Cloud | Compatible | Not Compatible | The deleteObjects and uploadPartCopy interfaces of Google Cloud are not compatible with AWS S3 and require separate adaptation. |
| Alibaba Cloud | Compatible | Compatible | - |
| Oracle Cloud | Compatible | Compatible | - |
| DigitalOcean | Compatible | Compatible | - |
| Cloudflare | Compatible | Compatible | - |
| Tencent Cloud | Compatible | Compatible | - |
| Huawei Cloud | Compatible | Compatible | - |
| Baidu Cloud | Compatible | Compatible | - |
| Volcanic Cloud | Compatible | Pending Testing | - |
| China Mobile Cloud | Compatible | Pending Testing | - |
| Chinatelecom Cloud | Compatible | Pending Testing | - |
Deploy AutoMQ
Please refer to Cluster Deployment on Linux▸