Redpanda Console

Manage Kafka/AutoMQ Clusters More Conveniently Through Kafka Web UI

With the rapid development of big data technology, Kafka has become a core component for real-time data processing in enterprises due to its high throughput and low latency. However, managing and monitoring Kafka clusters is not an easy task. Traditional command-line tools and scripts, although powerful, are complex and not intuitive for developers and operations personnel. To address these challenges, the Kafka Web UI was created, providing users with a more convenient and efficient way to manage Kafka clusters.

Over more than a decade of development, Apache Kafka has built a rich ecosystem within the industry. As the successor to Apache Kafka, AutoMQ can fully leverage the products in this ecosystem due to its complete compatibility with Kafka. The AutoMQ Business Edition offers robust control capabilities. If you are using AutoMQ, you can also use products like Kafdrop and Redpanda Console [1] to manage AutoMQ clusters.

Today’s content shares how to monitor the status of AutoMQ clusters through Redpanda Console [1] to improve system maintainability and stability.

Integrating AutoMQ with Redpanda Console

Redpanda Console, provided by Redpanda [2], is a Kafka Web UI interface used for monitoring and managing Redpanda or Kafka clusters. It offers an intuitive user interface through which users can easily view cluster status, monitor performance metrics, and manage Topics and partitions. This console aims to simplify the daily operations of data stream systems, allowing users to maintain and monitor their clusters more effectively.

Thanks to AutoMQ’s complete compatibility with Kafka, it can seamlessly integrate with Redpanda Console. By utilizing Redpanda Console, AutoMQ users can also enjoy an intuitive user interface to monitor the status of AutoMQ clusters in real-time, including Topics, partitions, Consumer Groups, and their offsets. This monitoring capability not only enhances problem diagnosis efficiency but also helps optimize cluster performance and resource utilization.

This tutorial will teach you how to start the Redpanda Console service and use it in conjunction with AutoMQ clusters to achieve cluster status monitoring and management.

Prerequisites

Deploy AutoMQ Cluster
Prepare Redpanda Console Environment

Deploy AutoMQ Cluster

Please refer to the official AutoMQ documentation: Cluster Deployment | AutoMQ [3].

Deploy Redpanda Console

There are two ways to deploy Redpanda Console: Docker deployment and release version deployment. Docker deployment is relatively simple. If you want a quick and easy way to integrate AutoMQ with Redpanda Console, it is recommended to choose Docker deployment. If you have specific requirements such as login verification, SASL authentication, TLS configuration, and log level settings, you should deploy according to the release version method. Below, I will introduce both configuration methods.

Docker Deployment

Redpanda Console can be deployed using Docker. Refer to Quick Start [4]. After setting up the AutoMQ cluster in the above process, you will already know the addresses and ports of all Broker nodes. Therefore, you can establish an association between Redpanda Console and the AutoMQ cluster by specifying the KAFKA_BROKERS parameter in the Docker startup command. The Docker container startup command is as follows:

docker run -p 8080:8080 -e KAFKA_BROKERS=192.168.0.4:9092,192.168.0.5:9092,192.168.0.6:9092 docker.redpanda.com/redpandadata/console:latest

-p 8080:8080 : Specifies the service access mapping port for Redpanda Console.
KAFKA_BROKERS : Needs to be specified as the Broker addresses of your AutoMQ cluster.

Release Deployment

You need to download and extract the suitable version from the Redpanda Console's GitHub Releases page: Release Redpanda Console [5], to a specified folder such as /opt. The operation command is as follows:

# Ubuntu Linux
cd /opt
sudo curl -L -o redpanda_console.tar.gz https://github.com/redpanda-data/console/releases/download/v2.6.0/redpanda_console_2.6.0_linux_amd64.tar.gz
# Unzip, Get Redpanda_console
sudo tar -xzf redpanda_console.tar.gz
# Config Set
sudo mkdir -p /etc/redpanda
# Write Config
sudo vim /etc/redpanda/redpanda-console-config.yaml

An example of the redpanda-console-config.yaml configuration file content is as follows:

kafka:
  #Brokers is a list of bootstrap servers with
  #port (for example "localhost:9092").
  brokers:
    - broker-0.mycompany.com:19092
    - broker-1.mycompany.com:19092
    - broker-2.mycompany.com:19092

Note: Ensure that the server where you are currently installing Redpanda Console can access the servers where the Broker nodes in the configuration file are located.

For more detailed settings, refer to the official documentation: Redpanda Console Configuration [6]. After the configuration is completed, you need to set environment variables so that the Redpanda Console executable can obtain the configuration file information and start the Redpanda Console:

# Set Env
export CONFIG_FILEPATH="/etc/redpanda/redpanda-console-config.yaml"
# /opt/ Run Console
./redpanda-console

./redpanda-console
{"level":"info","ts":"2024-07-10T09:52:52.958+0800","msg":"started Redpanda Console","version":"2.6.0","built_at":"1717083695"}
{"level":"info","ts":"2024-07-10T09:52:52.963+0800","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2024-07-10T09:52:54.780+0800","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":2,"controller_id":0,"kafka_version":"at least v3.6"}
{"level":"info","ts":"2024-07-10T09:53:05.620+0800","msg":"Server listening on address","address":"[::]:8080","port":8080}

Access the Console Page

After completing the deployment steps above, you can access the console service by entering the address in your browser (e.g., http://{console_ip}:8080). The display will look like this:

Cluster Overview

The Cluster Overview page provides users with a macro view, displaying core information about the AutoMQ cluster, including but not limited to:

Cluster Running Status: Shows the current health status of the cluster, helping users quickly identify issues.
Storage Usage: Displays data storage usage within the cluster, facilitating storage management and planning for users.
Version Information: Shows the current version of the AutoMQ cluster, making it easier to track and upgrade.
Online Broker Count: Displays the real-time count of online Brokers, a key metric.
Number of Topics and Replicas: Provides information on the number of Topics and Replicas, helping users understand the cluster's scale and data replication status.

Monitoring the cluster status is crucial for ensuring the stability and performance of the message queue system. By monitoring the cluster's running status, storage usage, version information, online Broker count, and the number of Topics and Replicas in real-time, operations personnel can quickly identify and resolve potential issues, preventing system failures from impacting business operations. Additionally, these metrics aid in capacity planning and resource management, ensuring the system can handle future data growth. Furthermore, understanding the cluster's version information helps users perform timely software upgrades, leveraging the latest features and security fixes to enhance the overall reliability and efficiency of the system.

Topic Overview

On the Topic list page, users can see a list of all Topics in the current AutoMQ cluster, including key information for each Topic such as the number of partitions and replication strategy. Users can quickly browse and manage Topics through this page.

Topic Details

Clicking on a specific Topic, users will be taken to the detailed page of that Topic, where they can gain in-depth understanding and perform operations:

Message List: Browse and search messages within the Topic, which is very useful for message tracking and debugging.
Consumer Information: Displays current consumers and Consumer Group information subscribing to the Topic, facilitating monitoring of consumption status.
Partition Status: Shows detailed information for each partition, including key metrics such as Leader and ISR.
Configuration Information: Lists the configuration details of the Topic, with support for modifications to optimize performance or behavior.
ACL (Access Control List): Manages access permissions to the Topic, ensuring data security.

Additionally, Redpanda Console allows users to manually create and publish messages, which is highly valuable for testing or injecting messages in specific scenarios.

Monitoring Topic details allows us to gain in-depth insights into the operational status of the message queue, track and debug by browsing the message list, monitor consumer information to understand consumption status, review partition status to ensure data distribution and high availability, manage configuration details to optimize performance, and set access controls to safeguard data security. These features help in the timely identification and resolution of issues, improving the overall efficiency and reliability of the system.

Summary

This article introduced the integration process of Redpanda Console with AutoMQ, demonstrating how this powerful tool simplifies and enhances the management of AutoMQ clusters. It is hoped that this article provides a practical reference for users aiming to improve the efficiency and functionality of message queue management.