Runs on CEPH

AutoMQ leverages EBS and S3 storage, while CEPH supports both POSIX and S3 access protocols, making it an ideal storage backend for AutoMQ. Here's a guide to deploying AutoMQ with CEPH.

Prerequisites

To install CEPH, please refer to: https://docs.ceph.com/en/latest/install/
To install CEPH's S3 compatible component RGW, please refer to: https://docs.ceph.com/en/latest/cephadm/services/rgw/

Configure WAL

Consult the Ceph official documentation for instructions on how to mount raw devices on Linux hosts: https://docs.ceph.com/en/latest/rbd/
Set the raw device path to /dev/vdb.
AutoMQ utilizes raw devices to store WAL data at a specified path. Configure this using the startup parameter --override s3.wal.path=/dev/vdb.

Configure S3URL

Create a User for CEPH

radosgw-admin user create --uid="automq" --display-name="automq"

The default user has all the necessary permissions for AutoMQ. For limited permissions, refer to the CEPH official documentation for custom configurations. Here is what happens after running the commands:

{
    "user_id": "automq",
    "display_name": "automq",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "automq",
            "access_key": "X1J0E1EC3KZMQUZCVHED",
            "secret_key": "Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

Create a Bucket for CEPH

Set environment variables to configure the Access Key and Secret Key required by the AWS CLI.

export AWS_ACCESS_KEY_ID=X1J0E1EC3KZMQUZCVHED
export AWS_SECRET_ACCESS_KEY=Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD

Use the AWS CLI to create an S3 bucket.

aws s3api create-bucket --bucket automq-data --endpoint=http://127.0.0.1:80

Deploy AutoMQ

Here are the necessary parameters required to generate an S3 URL:

Parameter Name	Default Value in This Example	Description
--s3-access-key	X1J0E1EC3KZMQUZCVHED	Replace with actual details after creating a Ceph user
--s3-secret-key	Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD	Replace with actual details after creating a Ceph user
--s3-region	us-west-2	This parameter is ineffective in Ceph and can be set to any value, such as us-west-2
--s3-endpoint	http://127.0.0.1:80	This is the address of Ceph's S3-compatible component RGW. If there are multiple machines, it is recommended to use a load balancer (SLB) to aggregate into a single IP address.
--s3-data-bucket	automq-data	-
--s3-ops-bucket	automq-ops	-

Once the WAL and S3URL configurations are complete, you are ready to deploy AutoMQ. Please refer to the instructions in Cluster Deployment on Linux▸.

Runs on CEPH

Prerequisites​

Configure WAL​

Configure S3URL​

Create a User for CEPH​

Create a Bucket for CEPH​

Deploy AutoMQ​