Optimize your data storage with AutoMQ, the cloud-native Kafka-compatible solution utilizing leading cloud object storage for superior cost efficiency and scalability.
AutoMQ is a cloud-native, redesigned distribution of Kafka that fully utilizes cloud object storage services. It can build the storage layer using object storage from major cloud providers or storage software.
The URI is a unique identifier used by AutoMQ to represent writes to object storage, adhering to established standards to reconcile differences among various object storage service providers. The URI components are as follows: s3 specifies the protocol; xxx_bucket indicates the specific bucket; region and endpoint are additional parameters used by different vendors to locate the bucket.Ultimately, replace the variables in the configuration templates global.config and global.envs from Deploy Multi-Nodes Cluster on Linux▸ to generate the final configuration files.
If the s3.wal.path is set, you can also include the batchInterval and maxBytesInBatch parameters in the URI to control the submit interval and the maximum size of bytes per batch, respectively. The default settings are batchInterval=250 milliseconds and maxBytesInBatch=8388608 bytes.
It is recommended to refer to the Policy and Endpoint & Region links in the AWS documentation to get detailed configuration instructions for IAM and S3.
In a production environment, it is advisable to scope IAM Policy permissions to specific buckets to avoid unintended authorizations.
In a production environment, it is advisable to scope RAM policy permissions to specific buckets (OSS storage spaces) to avoid overly broad authorization and reduce security risks.
Since the AWS S3 SDK cannot directly access GCP Cloud Storage, AutoMQ Open Source cannot run directly on Google Cloud Storage. If you need to run AutoMQ on GCP, contact the AutoMQ team for support. AutoMQ BYOC supports GCP Cloud Storage.
Azure Blob Storage is not compatible with the AWS S3 network protocol; therefore, AutoMQ Open Source cannot run directly on Azure Blob Storage. Users can address this as follows:
We recommend checking the official website for more detailed information.It is advisable to consult the Policy, Detailed Policy, and Endpoint & Region links, and review OCI documentation for more detailed configuration instructions on IAM and Object Storage.
Allow group 'AutoMQ PoC' to manage objects in tenancy where any {target.bucket.name='automq-data', target.bucket.name='automq-ops'}
To deploy AutoMQ in other public cloud environments, make sure that the cloud provider’s object storage service is compatible with the S3 protocol. You can contact the AutoMQ team via this form to obtain the necessary assistance and best practices.
If you wish to deploy AutoMQ in a private data center, you can also use storage software like MinIO, Ceph, or CubeFS that supports the S3 protocol to provide object storage services.
