Ceph
Preface
Ceph[1] is an open-source, distributed storage system that supports object, block, and file storage. Initially developed in 2003 from Sage Weil's doctoral dissertation, Ceph was officially released under the LGPL 2.1 license in 2006. Integrated with the Linux kernel's KVM, Ceph is a standard feature in numerous GNU/Linux distributions. Its versatility in handling different storage types—object, block, and file—makes it ideal for various storage requirements.
AutoMQ[2] features an innovative shared storage architecture that leverages low-latency block devices and cost-effective object storage. With its support for both POSIX and S3 access protocols, Ceph is highly compatible with AutoMQ. This compatibility allows for the deployment of AutoMQ clusters in private data centers, creating a stream system that not only aligns with Kafka but also provides enhanced cost-efficiency, scalability, and sub-millisecond latency. This article will guide you through deploying the AutoMQ cluster on Ceph in your private data center.
Prerequisites
Ensure a functional Ceph environment; for installation details, consult the official documentation.
Install the S3-compatible component RGW of Ceph by following the official documentation.
Prepare five hosts for the AutoMQ cluster deployment. It is recommended to use Linux amd64 hosts equipped with 2 CPUs and 16GB of memory, and to prepare two virtual storage volumes. See the example below:
Role IP Node ID System Volume Data Volume CONTROLLER 192.168.0.1 0 EBS 20GB EBS 20GB CONTROLLER 192.168.0.2 1 EBS 20GB EBS 20GB CONTROLLER 192.168.0.3 2 EBS 20GB EBS 20GB BROKER 192.168.0.4 3 EBS 20GB EBS 20GB BROKER 192.168.0.5 4 EBS 20GB EBS 20GB Tips:
Please ensure that these machines are in the same subnet and can communicate with each other
In non-production environments, it's also feasible to deploy a single Controller; this Controller, by default, doubles as a Broker
Download the latest official binary installation package from AutoMQ Github Releases to install AutoMQ.
Create a Bucket for Ceph
- Set environment variables to configure the Access Key and Secret Key required by the AWS CLI.
export AWS_ACCESS_KEY_ID=X1J0E1EC3KZMQUZCVHED
export AWS_SECRET_ACCESS_KEY=Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD- Use the AWS CLI to create an S3 bucket.
aws s3api create-bucket --bucket automq-data --endpoint=http://127.0.0.1:80
aws s3api create-bucket --bucket automq-ops --endpoint=http://127.0.0.1:80Create a user for Ceph
radosgw-admin user create --uid="automq" --display-name="automq"
The user created by default has full permissions required by AutoMQ. For configuring minimal permissions, please refer to the official CEPH documentation for customization. The results of the commands executed are as follows:
{
"user_id": "automq",
"display_name": "automq",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "automq",
"access_key": "X1J0E1EC3KZMQUZCVHED",
"secret_key": "Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
Install and Start the AutoMQ Cluster
Configure the S3URL
Step 1: Generate the S3 URL
AutoMQ provides the automq-kafka-admin.sh tool for rapid deployment of AutoMQ. Just supply an S3 URL with the required access point and authentication details to launch AutoMQ using a single command, bypassing the need to manually create a cluster ID or format storage.
### Command Line Usage Example
bin/automq-kafka-admin.sh generate-s3-url \
--s3-access-key=xxx \
--s3-secret-key=yyy \
--s3-region=cn-northwest-1 \
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \
--s3-data-bucket=automq-data \
--s3-ops-bucket=automq-ops
When configuring with Ceph, use the following settings to generate the specific S3URL.
| Parameter Name | Default Value in This Example | Description |
|---|---|---|
| --sincere-access-key | X1J0E1EC3KZMQUZCVHED | Remember to replace this after creating a Ceph user, based on the actual situation |
| --s3-secret-key | Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD | After creating a Ceph user, remember to replace this according to actual circumstances |
| --s3-region | us-west-2 | This parameter is ineffective in Ceph; you can set it to any value, such as us-west-2 |
| --s3-endpoint | http://127.0.0.1:80 | This parameter is the address of Ceph's S3-compatible component RGW. If there are multiple machines, it is recommended to use a load balancer (SLB) to consolidate into one IP address. |
| --s3-data-bucket | automq-data | - |
| --s3-ops-bucket | automq-ops | - |
Output Results
Following the execution of this command, the process will automatically progress through the following steps:
Check the basic functionalities of S3 using the provided accessKey and secretKey to ensure compatibility with AutoMQ.
Create an s3url using the identity details and endpoint information.
Retrieve the startup command example for AutoMQ using the s3url, updating --controller-list and --broker-list with the actual CONTROLLER and BROKER addresses to be deployed.
The execution results are as follows:
############ Ping S3 ########################
[ OK ] Write s3 object
[ OK ] Read s3 object
[ OK ] Delete s3 object
[ OK ] Write s3 object
[ OK ] Upload s3 multipart object
[ OK ] Read s3 multipart object
[ OK ] Delete s3 object
############ String of S3url ################
Your s3url is:
s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=xxx&s3-secret-key=yyy&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA
############ Usage of S3url ################
To start AutoMQ, generate the start commandline using s3url.
bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093" \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"
TIPS: Please replace the controller-list and broker-list with your actual IP addresses.
Step 2: Generate the Startup Command List
Replace the --controller-list and --broker-list in the commands generated in the previous step with your host information, specifically, replace them with the IP addresses of the 3 CONTROLLERs and 2 BROKERs mentioned in the environment preparation, using the default ports 9092 and 9093.
bin/automq-kafka-admin.sh generate-start-command \
--s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" \
--controller-list="192.168.0.1:9093;192.168.0.2:9093;192.168.0.3:9093" \
--broker-list="192.168.0.4:9092;192.168.0.5:9092"
Parameter Description
| Parameter Name | Mandatory | Description |
|---|---|---|
| --s3-url | Yes | Generated by the command line tool bin/automq-kafka-admin.sh generate-s3-url, includes authentication, cluster ID, etc. |
| --controller-list | Yes | At least one address is required, used as the IP and port list for CONTROLLER hosts. Format: IP1:PORT1; IP2:PORT2; IP3:PORT3 |
| --broker-list | Yes | At least one address is required, used as the IP and port list for BROKER hosts. Format: IP1:PORT1; IP2:PORT2; IP3:PORT3 |
| --controller-only-mode | No | Determines whether the CONTROLLER node assumes only the CONTROLLER role. Default is false, meaning the deployed CONTROLLER node also serves as a BROKER role. |
Output Result
Upon executing the command, it will produce a command to initiate AutoMQ.
############ Start Commandline ##############
To start an AutoMQ Kafka server, please navigate to the directory where your AutoMQ tgz file is located and run the following command.
Before running the command, make sure that Java 17 is installed on your host. You can verify the Java version by executing 'java -version'.
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=1 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.2:9092,CONTROLLER://192.168.0.2:9093 --override advertised.listeners=PLAINTEXT://192.168.0.2:9092
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=2 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.3:9092,CONTROLLER://192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.3:9092
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=3 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.4:9092 --override advertised.listeners=PLAINTEXT://192.168.0.4:9092
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker --override node.id=4 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.5:9092 --override advertised.listeners=PLAINTEXT://192.168.0.5:9092
TIPS: Start controllers first and then the brokers.
node.id is automatically generated starting from 0 by default.
Step 3: Start AutoMQ
To activate the cluster, execute the commands sequentially from the provided list on the designated CONTROLLER or BROKER host. For instance, to initiate the first CONTROLLER process at 192.168.0.1, run the first command template from the generated startup command list.
bin/kafka-server-start.sh --s3-url="s3://s3.cn-northwest-1.amazonaws.com.cn?s3-access-key=XXX&s3-secret-key=YYY&s3-region=cn-northwest-1&s3-endpoint-protocol=https&s3-data-bucket=automq-data&s3-path-style=false&s3-ops-bucket=automq-ops&cluster-id=40ErA_nGQ_qNPDz0uodTEA" --override process.roles=broker,controller --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override listeners=PLAINTEXT://192.168.0.1:9092,CONTROLLER://192.168.0.1:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092
Parameter Description
When deploying the startup command, any unspecified parameters will default to Apache Kafka's default configurations. For parameters newly introduced by AutoMQ, AutoMQ's default settings will apply. To alter these defaults, you can append --override key=value parameters at the end of the command.
| Parameter Name | Mandatory | Description |
|---|---|---|
| s3-url | Yes | Generated by the command line tool bin/automq-kafka-admin.sh generate-s3-url, containing authentication details, cluster ID, etc. |
| process.roles | Yes | Options include CONTROLLER or BROKER. If a host serves as both CONTROLLER and BROKER, the configuration value should be CONTROLLER, BROKER. |
| node.id | Yes | An integer that uniquely identifies a BROKER or CONTROLLER within a Kafka cluster; must be unique within the cluster. |
| controller.quorum.voters | Yes | Information about the hosts participating in the KRAFT election, including nodeid, ip, and port, for example: 0@192.168.0.1:9093, 1@192.168.0.2:9093, 2@192.168.0.3:9093 |
| listeners | Yes | The IP and port that are being listened to. |
| advertised.listeners | Yes | The access address provided by the BROKER for Clients. |
| log.dirs | No | Directory for storing KRAFT, BROKER metadata. |
| s3.wal.path | No | In production environments, it is recommended to store AutoMQ WAL data on a newly mounted standalone raw device. This can achieve better performance as AutoMQ supports writing data directly to raw devices, thereby reducing latency. Please ensure the correct path is configured for storing WALs. |
| autobalancer.controller.enable | No | The default value is false, meaning self-balancing is not enabled. Once traffic self-balancing is activated, AutoMQ's auto balancer component will automatically reassign partitions to ensure overall traffic is balanced. |
Tips: If continuous self-balancing is required or if you need to run Example: Self-Balancing When Cluster Nodes Change, it is advisable to explicitly set the parameter --override autobalancer.controller.enable=true when launching the Controller.
Background Execution
To execute in the background, append the following code at the end of your command:
command > /dev/null 2>&1 &
Preparing Raw Device Volumes
AutoMQ utilizes raw devices as the data volume for WALs to optimize the efficiency of writing to WALs. On Ceph, you can prepare raw devices by following these steps:
Consult the Ceph official documentation for instructions on how to mount raw devices on a Linux host.
Set the bare device path to
/dev/vdb.
Data Volume Path
Use the lsblk command on Linux to view local data volumes; unpartitioned block devices will appear as data volumes. In this example, vdb is shown as an unpartitioned bare block device.
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 2M 0 part
├─vda2 253:2 0 200M 0 part /boot/efi
└─vda3 253:3 0 19.8G 0 part /
vdb 253:16 0 20G 0 disk
By default, AutoMQ stores metadata and WAL data in the /tmp directory. However, it is critical to note that if the /tmp directory is mounted on tmpfs, it is unsuitable for production use.
For enhanced suitability in production or formal testing environments, adjust the configuration as follows: set the metadata directory log.dirs and WAL data directory s3.wal.path (write data disk's bare device) to the path of the bare device.
bin/kafka-server-start.sh ...\
--override s3.telemetry.metrics.exporter.type=prometheus \
--override s3.metrics.exporter.prom.host=0.0.0.0 \
--override s3.metrics.exporter.prom.port=9090 \
--override log.dirs=/root/kraft-logs \
--override s3.wal.path=/dev/vdb \
> /dev/null 2>&1 &
Tips:
/dev/vdbis the bare device path we configured using Ceph
At this stage, you have successfully deployed an AutoMQ cluster using Ceph, creating a cost-effective, low-latency, and scalable Kafka cluster. To further explore features of AutoMQ such as second-level partition reassignment and self-balancing, please refer to the official example.
Reference Materials
[1] Ceph: https://ceph.io/en/
[2] What is ceph: https://ubuntu.com/ceph/what-is-ceph