Discover seamless data integration with AutoMQ’s Table Topic, enhancing data lakes through cloud-native scalability and Kafka compatibility. Perfect for swift trials.
Table Topic is a core feature of AutoMQ, designed for modern data lake architectures. Its primary purpose is to create direct integration between streaming data and static data lakes through architectural innovation, addressing traditional issues like stream-batch separation, complex ETL processes, and high costs. This document is aimed at new users experiencing AutoMQ for the first time, focusing on how to deploy an AutoMQ cluster with Table Topic support in a local development environment (by deploying MinIO to provide object storage services), enabling users to quickly experience the features of Table Topic.
This document is intended solely for testing and learning scenarios. For production environments, please use the Table Topic feature and refer to deployment documentation such as Deploy Multi-Nodes Cluster on Linux▸.
Run the following command to download the Docker Compose configuration file for a single-node deployment and start the AutoMQ cluster using Docker Compose.
curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/table_topic/docker-compose.yml && docker-compose up -d
Once started successfully, we will send Table Topic messages and perform queries using Spark SQL to experience automatic table creation, table querying and updating, schema changes, data partitioning, and CDC capabilities. Users can open http://localhost:8888 in their browser and interactively execute commands using Jupyter Notebook.
The first three steps in the Notebook involve the creation of a Table Topic, writing a single message, and triggering the creation of an Iceberg table.
Note:There may be a delay between writing the message and the data being entered into the table. If an error occurs during the query, we recommend retrying after a short wait.
