Redpanda Console

Managing Kafka/AutoMQ Clusters with Greater Convenience Through the Kafka Web UI

With the rapid advancement of big data technologies, Kafka has emerged as a core component for real-time data processing in enterprises due to its high throughput and low latency distributed messaging system. However, managing and monitoring Kafka clusters is no easy task. Traditional command-line tools and scripts, while powerful, can be complex and unintuitive for developers and operations personnel. To address these challenges, the Kafka Web UI was developed, providing users with a more convenient and efficient way to manage Kafka clusters.

Over more than a decade, Apache Kafka® has built a rich ecosystem within the industry. As the successor to Apache Kafka, AutoMQ, being fully compatible with Kafka, can leverage products within this ecosystem. The AutoMQ Business Edition already offers a robust control plane. If you are using AutoMQ, you can also utilize products like Kafdrop and Redpanda Console [1] to manage AutoMQ clusters.

Today’s content focuses on how to monitor the status of AutoMQ clusters using Redpanda Console [1] to improve system maintainability and stability.

Integrating AutoMQ with Redpanda Console

Redpanda Console, provided by Redpanda [2], is a Kafka Web UI for monitoring and managing Redpanda or Kafka clusters. It offers an intuitive user interface through which users can easily view cluster statuses, monitor performance metrics, and manage Topics and partitions. This console aims to simplify daily operations of data flow systems, enabling users to more effectively maintain and monitor their clusters.

Thanks to AutoMQ's full compatibility with Kafka, it can seamlessly integrate with Redpanda Console. By leveraging Redpanda Console, AutoMQ users can also enjoy an intuitive user interface and real-time monitoring of the AutoMQ cluster status, including Topics, partitions, Consumer Groups, and their offsets. This monitoring capability not only enhances issue diagnosis efficiency but also aids in optimizing cluster performance and resource utilization.

This tutorial will teach you how to start the Redpanda Console service and pair it with your AutoMQ cluster to achieve cluster status monitoring and management.

Prerequisites

Deploy AutoMQ Cluster
Prepare Redpanda Console Environment

Deploy AutoMQ Cluster

Please refer to the official AutoMQ documentation: Cluster Deployment | AutoMQ [3].

Deploy Redpanda Console

There are two ways to deploy the Redpanda Console: using Docker or the release version. The Docker deployment is simpler. If you want a quick and easy way to experience the integration of AutoMQ and Redpanda Console, we recommend choosing Docker. If you have specific requirements such as login authentication, SASL authentication, TLS configuration, or log level settings, you can deploy it using the release version. Below, I will introduce both configuration methods.

Docker Deployment

Redpanda Console can be deployed via Docker. For reference: Quick Start [4]. After setting up the AutoMQ cluster, we already know the addresses and ports of all the Broker nodes. Therefore, we can establish a connection between the Redpanda Console and the AutoMQ cluster by specifying the KAFKA_BROKERS parameter in the Docker startup command. The Docker container startup command is as follows:

docker run -p 8080:8080 -e KAFKA_BROKERS=192.168.0.4:9092,192.168.0.5:9092,192.168.0.6:9092 docker.redpanda.com/redpandadata/console:latest

-p 8080:8080: Specifies the port mapping for accessing the Redpanda Console service.
KAFKA_BROKERS: Needs to be set to the Broker address of your own AutoMQ cluster.

Distribution Deployment

You need to visit the GitHub Releases page of Redpanda Console: Release Redpanda Console [5], Select the appropriate version to download and extract it to a specified folder, such as /opt. The commands are as follows:

# Ubuntu Linux
cd /opt
sudo curl -L -o redpanda_console.tar.gz https://github.com/redpanda-data/console/releases/download/v2.6.0/redpanda_console_2.6.0_linux_amd64.tar.gz
# Unzip, Get Redpanda_console
sudo tar -xzf redpanda_console.tar.gz
# Config Set
sudo mkdir -p /etc/redpanda
# Write Config
sudo vim /etc/redpanda/redpanda-console-config.yaml

redpanda-console-config.yaml Here is an example of the configuration file content:

kafka:
  #Brokers is a list of bootstrap servers with
  #port (for example "localhost:9092").
  brokers:
    - broker-0.mycompany.com:19092
    - broker-1.mycompany.com:19092
    - broker-2.mycompany.com:19092

Note: Please ensure that the server where you are currently installing Redpanda Console can access the servers where the Broker nodes specified in the configuration file are located.

For more detailed settings, refer to the official documentation: Redpanda Console Configuration [6]. After completing the configuration, you need to set environment variables so that the Redpanda Console executable can access the configuration file information and start the Redpanda Console:

# Set Env
export CONFIG_FILEPATH="/etc/redpanda/redpanda-console-config.yaml"
# /opt/ Run Console
./redpanda-console

./redpanda-console
{"level":"info","ts":"2024-07-10T09:52:52.958+0800","msg":"started Redpanda Console","version":"2.6.0","built_at":"1717083695"}
{"level":"info","ts":"2024-07-10T09:52:52.963+0800","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2024-07-10T09:52:54.780+0800","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":2,"controller_id":0,"kafka_version":"at least v3.6"}
{"level":"info","ts":"2024-07-10T09:53:05.620+0800","msg":"Server listening on address","address":"[::]:8080","port":8080}

Access the Console Page

After completing the deployment steps, you can access the console service by entering the address in your browser (for example: http://{console_ip}:8080). The interface appears as follows:

Cluster Overview

The Cluster Overview page provides users with a high-level view, displaying core information about the AutoMQ cluster. This includes but is not limited to:

Cluster running status: Displays the current health of the cluster, helping users quickly identify issues.
Storage usage: Shows data storage usage within the cluster, facilitating storage management and planning.
Version information: Displays the current version of the AutoMQ cluster, making it easy to track and upgrade.
Online Broker Count: Displays the number of online Brokers in real-time, a key metric.
Number of Topics and Replicas: Provides information on the number of Topics and Replicas, helping users understand the scale of the cluster and data replication.

Monitoring the cluster status is crucial for ensuring the stability and performance of the message queue system. By monitoring the cluster's runtime status, storage usage, version information, online Broker count, and the number of Topics and Replicas in real-time, operations personnel can quickly identify and resolve potential issues, preventing system failures that could impact business operations. These metrics also aid in capacity planning and resource management, ensuring the system can handle future data growth. Additionally, understanding the cluster's version information can help users perform timely software upgrades to take advantage of the latest features and security fixes, enhancing the overall reliability and efficiency of the system.

Topic Overview

On the Topic list page, users can see a list of all Topics in the current AutoMQ cluster, including key information for each Topic, such as the number of partitions and replica policies. This page allows users to quickly browse and manage Topics.

Topic Details

By clicking on a specific Topic, users will enter the detailed page for that Topic, where they can delve deeper and perform operations:

Message List: Browse and search messages within the Topic, which is very useful for message tracking and debugging.
Consumer Information: Displays information about the current consumers and consumer groups subscribed to the Topic, facilitating monitoring of consumption status.
Partition Status: Shows detailed information for each partition, including key metrics such as Leader and ISR.
Configuration Information: Lists the configuration details of the Topic, supporting modifications to optimize performance or behavior.
ACL (Access Control List): Manages access permissions for the Topic to ensure data security.

Additionally, Redpanda Console allows users to manually create and publish messages, which is highly valuable for testing or injecting messages in specific scenarios.

Monitoring Topic details enables us to gain deep insights into the operation of the message queue. By browsing the message list for tracking and debugging, viewing consumer information to monitor consumption status, understanding partition status to ensure data distribution and high availability, managing configuration to optimize performance, and setting access controls to ensure data security, these features help in timely identification and resolution of issues, improving the overall efficiency and reliability of the system.

Conclusion

This article introduced the process of integrating Redpanda Console with AutoMQ, demonstrating how this powerful tool simplifies and enhances the management of AutoMQ clusters. It is hoped that this article provides a practical reference for users aiming to improve the efficiency and functionality of their message queue management.