Databend - AutoMQ Documentation

Databend is a next-generation cloud-native data warehouse developed in Rust. It is specifically designed for cloud architecture. Built on object storage, it offers businesses integration with data lakes and warehouses while providing a big data analytics platform that separates compute from storage. This document will guide you on how to import data from AutoMQ into Databend using bend-ingest-kafka.

Environment Preparation

Prepare Databend Cloud and Test Data

Firstly, visit Databend Cloud to initiate a Warehouse. Then, create databases and test tables within the worksheet.

create database automq_db;
create table users (
    id bigint NOT NULL,
    name string NOT NULL,
    ts timestamp,
    status string
)

Prepare AutoMQ and Test Data

Refer to Deploy Multi-Nodes Cluster on Linux▸ to set up AutoMQ, making sure there’s network connectivity between AutoMQ and Databend. Quickly create a topic named example_topic in AutoMQ and write test JSON data into it by following these steps.

Create Topic

Use Apache Kafka command-line tools to create a topic. Ensure you have access to the Kafka environment and that the Kafka service is running. Below is an example command for creating a topic:

./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092  --partitions 1 --replication-factor 1

When executing commands, replace topic and bootstrap-server with the actual Kafka server addresses.

After creating the topic, you can use the following command to verify whether the topic has been successfully created.

./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092

Generate Test Data

Generate test data in JSON format that corresponds to the previously mentioned table.

{
  "id": 1,
  "name": "Test User",
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

Write Test Data

Use Kafka’s command-line tools or programming methods to write test data into a topic named example_topic. Here is an example using command-line tools:

echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic

When executing the command, replace the topic and bootstrap-server with the actual Kafka server address being used.

You can view the topic data just written using the following command:

sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning

Create a Bend-ingest-databend Job

bend-ingest-kafka is designed to monitor Kafka and batch write data into a Databend Table. After deploying bend-ingest-kafka, the data import job can be initiated.

bend-ingest-kafka --kafka-bootstrap-servers="localhost:9094" --kafka-topic="example_topic" --kafka-consumer-group="Consumer Group" --databend-dsn="https://cloudapp:password@host:443" --databend-table="automq_db.users" --data-format="json" --batch-size=5 --batch-max-interval=30s

When executing the command, replace kafka-bootstrap-servers with the actual Kafka server address you are using.

Parameter Description

databend-dsn

The DSN provided by Databend Cloud for connecting to the warehouse can be found in this documentation.

batch-size

bend-ingest-kafka will gather data until it reaches the specified batch size, then initiate a data synchronization process.

Verify Data Import

Navigate to the Databend Cloud worksheet and query the automq_db.users table. You should confirm that the data has been successfully synchronized from AutoMQ to the Databend Table.

​Environment Preparation

​Prepare Databend Cloud and Test Data

​Prepare AutoMQ and Test Data

​Create Topic

​Generate Test Data

​Write Test Data

​Create a Bend-ingest-databend Job

​Parameter Description

​databend-dsn

​batch-size

​Verify Data Import

Environment Preparation

Prepare Databend Cloud and Test Data

Prepare AutoMQ and Test Data

Create Topic

Generate Test Data

Write Test Data

Create a Bend-ingest-databend Job

Parameter Description

databend-dsn

batch-size

Verify Data Import