Runs on Cloud
AutoMQ is a Kafka distribution redesigned with cloud-native principles, fully leveraging cloud services such as EBS and S3. The following text outlines AutoMQ's support across various major cloud providers.
Depends on EBS Service
AutoMQ utilizes EBS for Write-Ahead-Log (WAL) storage, accessing it directly as a bare device, bypassing traditional file system mounting. EBS, a fundamental service provided by cloud vendors, ensures high data durability, typically employing triple-replica technology or erasure coding. Solutions like AWS Nitro and Alibaba Cloud ECS Bare Metal Instance utilize local hardware acceleration, and EBS server clusters also integrate RDMA technology, allowing EBS latency to compete with local NVME SSDs. Additionally, EBS uses dedicated network channels, ensuring it does not consume bandwidth from the host machine. Cost-wise, EBS proves very economical—for instance, an AWS EBS GP3 configuration with 10GB/3000IOPS/125MBps costs under $1 per month.
Therefore, EBS represents an outstanding alternative to traditional data storage technologies for WAL.
AutoMQ employs the standard POSIX API to access EBS, ensuring broad compatibility across various cloud service providers and supporting all major cloud providers' EBS services.
Depends on Object Storage Service
AutoMQ accesses object storage services using the AWS S3 SDK, which most cloud providers support through the standard S3 protocol, except for Azure. The AutoMQ team is currently developing specific adaptations for Azure, so stay tuned.
AutoMQ Community Edition accesses object storage using the following AWS S3 SDK.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.20.127</version>
</dependency>
Based on Access Control Policies
When utilizing Object storage to bolster security, you can limit access by implementing the least privilege policy. The operational procedures across various Cloud providers are quite similar, and you can find specific settings in the Cluster Deployment on Linux▸.
Testing Object Storage Compatibility
Once you've set up a storage bucket, use the command below to check if the Object storage is compatible with AutoMQ.
bin/automq-kafka-admin.sh generate-s3-url \
--s3-access-key=xxx \
--s3-secret-key=yyy \
--s3-region=cn-northwest-1 \
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \
--s3-data-bucket=automq-test \
--s3-ops-bucket=automq-test
List of Cloud Providers Support
| Cloud providers | EBS Compatibility | Object Storage Compatibility | Description |
|---|---|---|---|
| AWS | Compatible | Compatible | - |
| Azure | Compatible | Incompatible | Azure's object storage network protocol is completely different from AWS S3 and requires separate adaptation. |
| GCP | Compatible | Incompatible | GCP's deleteObjects and uploadPartCopy interfaces are incompatible with AWS S3 and require individual adaptation. |
| Alibaba Cloud | Compatible | Compatible | - |
| Oracle Cloud | Compatible | Compatible | - |
| DigitalOcean | Compatible | Compatible | - |
| Cloudflare | Compatible | Compatible | - |
| Tencent Cloud | Compatible | Compatible | - |
| Huawei Cloud | Compatible | Compatible | - |
| Baidu Cloud | Compatible | Compatible | - |
| Volcanic Cloud | Compatible | To be tested | - |
| China Mobile Cloud | Compatible | To be tested | - |
| Chinatelecom Cloud | Compatible | To be tested | - |
Deploy AutoMQ
Please refer to Cluster Deployment on Linux▸