Skip to Main Content

Runs on CubeFS

AutoMQ leverages both EBS and S3 storage, and CubeFS offers support for both POSIX and S3 access protocols, which makes it an excellent storage backend for AutoMQ. Here's a guide on how to deploy AutoMQ using CubeFS.

Prerequisites​

Configure WAL​

  1. For instructions on mounting raw devices, please consult the official CubeFS documentation: https://www.cubefs.io/docs/master/user-guide/file.html

  2. It is advisable to set the raw device path to /dev/vdb.

  3. Discover how to use FUSE to bypass the file system and write directly to raw devices by consulting the official CubeFS documentation. For comprehensive details, refer to the official CubeFS documentation.

  4. AutoMQ utilizes raw devices to store WAL data in a specified path. This configuration can be set using the startup parameter --override s3.wal.path=/dev/vdb.

Configuring S3URL​

Creating a User for CubeFS​


curl -H "Content-Type:application/json" -X POST --data '{"id":"automq","pwd":"12345","type":3}' "http://172.16.1.101:17010/user/create"

By default, users are granted full permissions necessary for AutoMQ. For minimal permissions, refer to the official CubeFS documentation for custom settings. Here are the results of the executed commands:


{
    "code":0,
    "msg":"success",
    "data":{
        "user_id":"automq",
        "access_key":"AEv7EVirKDJtfyK5",
        "secret_key":"fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6",
        "policy":
            {
                "own_vols":[],
                "authorized_vols":{}
            },
        "user_type":3,
        "create_time":"2024-05-16 16:56:13",
        "description":"",
        "EMPTY":false
    }
}

Creating a Bucket for CubeFS​

  1. The required Access Key and Secret Key for AWS CLI can be configured via environment variables.

export AWS_ACCESS_KEY_ID=AEv7EVirKDJtfyK5
export AWS_SECRET_ACCESS_KEY=fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6

  1. Create an S3 bucket using AWS CLI.

aws s3api create-bucket --bucket automq-data --endpoint=http://10.1.0.240:17410

Deploying AutoMQ​

This article provides you with the necessary parameters for generating an S3URL, as listed below:

Parameter Name
Default Value in This Example
Description
--s3-access-key
AEv7EVirKDJtfyK5
Remember to replace this after creating a CubeFS user, according to actual circumstances
--s3-secret-key
fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6
Remember to replace this after creating a CubeFS user, according to actual circumstances
--s3-region
us-west-2
This parameter is irrelevant in CubeFS and can be set to any value, such as us-west-2
--s3-endpoint
http://10.1.0.240:17410
This parameter is the address of the CubeFS object gateway's external service. If there are multiple machines, it is recommended to use a load balancer (SLB) to consolidate into a single IP address.
--s3-data-bucket
automq-data
-
--s3-ops-bucket
automq-ops
-

After configuring WAL and S3URL, you are ready to deploy AutoMQ. Please follow the detailed steps in Cluster Deployment on Linux▸.