Skip to Main Content

Metrics

This article will introduce the observability metrics definitions for AutoMQ, helping you better understand AutoMQ's performance and operational status.

AutoMQ metrics are defined and presented in Prometheus format. If you need other protocol formats, you will need to convert them yourself.

General Metrics​

Kafka_server_connection_count​

The current number of established connections on a node.

  • Type: Gauge

Kafka_network_threads_idle_rate​

Idle rate of Kafka SocketServer network threads, range: [0, 1.0].

  • Type: Gauge

Kafka_io_threads_idle_time_nanoseconds_total​

Idle time of Kafka request handler threads, which is a cumulative value of Apache Kafka's native metric RequestHandlerAvgIdlePercent, measured in nanoseconds. By differentiating this value over time (in nanoseconds), you can obtain the thread idle rate. Note that if the node is a combined node (i.e., serves as both Controller and Broker), the request handler idle rate for both Controller and Broker are summed, making the maximum thread idle rate 2.0.

  • Type: Counter

Controller Metrics​

Kafka_controller_active_count​

Indicates whether the current Controller node is the active Controller, with a value of 1 indicating active and 0 indicating inactive.

  • Type: Gauge

Kafka_broker_active_count​

The number of active Brokers in the current cluster.

  • Type: Gauge

Kafka_broker_fenced_count​

The number of Brokers fenced in the current cluster.

  • Type: Gauge

Kafka_topic_count​

The total number of Topics in the current cluster.

  • Type: Gauge

Kafka_partition_total_count​

The total number of partitions in the current cluster.

  • Type: Gauge

Kafka_partition_offline_count​

The total number of partitions without a leader in the current cluster.

  • Type: Gauge

Kafka_stream_auto_balancer_metrics_time_delay_milliseconds​

The latency of AutoBalancer monitoring metrics reported by each Broker node in the cluster. If the latency exceeds a certain threshold, the Broker node is considered out-of-sync by AutoBalancer and will no longer participate in partition reassignment.

  • Type: Gauge

  • Labels:

    • node_id: The node ID reporting AutoBalancer monitoring metrics.

Kafka_stream_s3_object_count​

The total number of Objects uploaded to Object storage in the current cluster, categorized by Object status.

  • Type: Gauge

  • Labels:

    • state: Object states are categorized into three types:

      • prepared: Objects that have not yet completed writing and have not been committed

      • committed: Objects that have completed writing and have been committed

      • mark_destroyed: Objects marked for deletion, which will be removed from the object storage after a certain delay

Kafka_stream_s3_object_size_bytes​

The total size of objects uploaded to object storage by the current cluster

  • Type: Gauge

Kafka_stream_stream_object_num​

The number of StreamObjects uploaded to object storage by the current cluster

  • Type: Gauge

Kafka_stream_stream_set_object_num​

The number of StreamSetObjects uploaded to object storage by each Broker in the current cluster

  • Type: Gauge

  • Labels:

    • node_id: The corresponding Broker node ID

Broker Metrics​

Kafka_message_count_total​

The total number of messages received by the Broker node; measuring this over time provides the message throughput.

  • Type: Counter

  • Labels:

    • topic

Kafka_network_io_bytes_total​

The total size of messages received and sent by the Broker node; measuring this over time provides the message size throughput.

  • Type: Counter

  • Labels:

    • topic

    • partition

    • direction:

      • in: indicates receiving messages

      • out: indicates sending messages

Kafka_topic_request_count_total​

The total number of requests received for each Topic on the Broker node, including only produce and fetch request types.

  • Type: Counter

  • Labels:

    • topic

    • type: request type

      • produce

      • fetch

Kafka_topic_request_failed_total​

The total number of failed requests for each Topic on the Broker node, including only produce and fetch request types.

  • Type: Counter

  • Labels:

    • topic

    • type: Request Type

      • produce

      • fetch

Kafka_request_count_total​

The total number of requests received by the Broker nodes.

  • Type: Counter

  • Labels:

    • type: Request Type

    • version: The API version of the request type.

Kafka_request_error_count_total​

The total number of failed requests at the Broker nodes. Note that even successful requests are counted in this metric, with the error code for successful requests being NONE.

  • Type: Counter

  • Labels:

    • type: Request Type

    • error: Error code, NONE indicates that the request was successful.

Kafka_request_size_bytes_total​

The total size of requests received by the Broker nodes.

  • Type: Counter

  • Labels:

    • type: Request Type

Kafka_request_size_50p(99p/mean/max)_bytes​

The size of requests received by Broker nodes, represented by different percentiles.

  • Type: Gauge

  • Labels:

    • type: Request Type

Kafka_request_time_milliseconds_total​

The total time consumed by Broker nodes to process requests.

  • Type: Counter

  • Labels:

    • type: Request Type

Kafka_request_time_50p(99p/mean/max)_milliseconds​

The processing time of requests by Broker nodes, represented by different percentiles.

  • Type: Gauge

  • Labels:

    • type: Request Type

Kafka_request_queue_time_milliseconds_total​

The total queuing time of requests at Broker nodes, which increases when Kafka IO threads are busy.

  • Type: Counter

  • Labels:

    • type: Request Type

Kafka_request_queue_time_50p(99p/mean/max)_milliseconds​

Queue time of requests at Broker nodes, presented by different percentiles.

  • Type: Gauge

  • Labels:

    • type: Request Type

Kafka_response_queue_time_milliseconds_total​

Response queue time at Broker nodes. When Kafka Network threads are busy, response queue time increases.

  • Type: Counter

  • Labels:

    • type: Request Type

Kafka_response_queue_time_50p(99p/mean/max)_milliseconds​

Response queue time at Broker nodes, presented by different percentiles.

  • Type: Gauge

  • Labels:

    • type: Request Type

Kafka_request_queue_size​

Request queue size at Broker nodes.

  • Type: Gauge

Kafka_response_queue_size​

The response queue size of the Broker node.

  • Type: Gauge

Kafka_purgatory_size​

The number of requests waiting in the producer or fetch purgatory on the Broker node.

  • Type: Gauge

  • Labels:

    • type:

      • Produce

      • Fetch

Kafka_partition_count​

The number of partitions currently assigned to the Broker node.

  • Type: Gauge

Kafka_logs_flush_time_50p(99p/mean/max)_milliseconds​

The log flush time on the Broker node; in AutoMQ, this represents the flush time of the Delta WAL, shown in different percentiles.

  • Type: Gauge

Kafka_log_end_offset​

The maximum logical offset of each partition on the Broker node.

  • Type: Gauge

  • Labels:

    • topic

    • partition

Kafka_log_size​

The message size of each partition on the Broker node.

  • Type: Gauge

  • Labels:

    • topic

    • partition

Kafka_group_commit_offset​

The consumption offsets for each Consumer Group on the corresponding partitions. Note that this metric is reported by the Broker where the Group Coordinator of each Consumer Group resides.

  • Type: Gauge

  • Labels:

    • consumer_group

    • topic

    • partition

Kafka_group_count​

The number of Consumer Groups managed by the Broker node where each Group Coordinator is located.

  • Type: Gauge

Kafka_group_preparing_rebalance_count​

The number of Consumer Groups preparing for a rebalance.

  • Type: Gauge

Kafka_group_completing_rebalance_count​

The number of Consumer Groups waiting for the Leader to assign state.

  • Type: Gauge

Kafka_group_stable_count​

The number of Consumer Groups in a stable state.

  • Type: Gauge

Kafka_group_empty_count​

The number of Consumer Groups with no members but not yet expired.

  • Type: Gauge

Kafka_group_dead_count​

The number of Consumer Groups with no members and whose metadata has been removed.

  • Type: Gauge

Kafka_stream_upload_size_bytes_total​

The total size of data uploaded by Broker nodes to object storage.

  • Type: Counter

Kafka_stream_download_size_bytes_total​

The total size of data downloaded by Broker nodes from object storage.

  • Type: Counter

Kafka_stream_network_inbound_usage_bytes_total​

The total ingress bandwidth usage of Broker nodes, including received messages and data downloaded from object storage, with the time derivative yielding ingress throughput.

  • Type: Counter

Kafka_stream_network_outbound_usage_bytes_total​

The total outbound bandwidth usage of a Broker node, including message consumption and data upload to object storage, can be differentiated over time to obtain traffic throughput.

  • Type: Counter

Kafka_stream_network_inbound_available_bandwidth_bytes​

The inbound traffic throughput reserved for cold reads and compaction on a Broker node. If this value is less than the required inbound traffic for cold reads and compaction, the corresponding requests will be placed into a throttling queue, and normal message sending and receiving traffic will not be affected by this throttling. Note that this metric value only represents the instantaneous value at the time of sampling. Due to the sampling interval and the specific implementation of the throttling strategy, this metric value is for reference only.

  • Type: Gauge

Kafka_stream_network_outbound_available_bandwidth_bytes​

The outbound traffic throughput reserved for cold reads and compaction on a Broker node. If this value is less than the required outbound traffic for cold reads and compaction, the corresponding requests will be placed into a throttling queue, and normal message sending and receiving traffic will not be affected by this throttling. Note that this metric value only represents the instantaneous value at the time of sampling. Due to the sampling interval and the specific implementation of the throttling strategy, this metric value is for reference only.

  • Type: Gauge

Kafka_stream_network_inbound_limiter_queue_time_50p(99p/mean/max)_nanoseconds​

The queuing time in the throttling queue for inbound traffic requests of cold reads and compaction when they get executed.

  • Type: Gauge

Kafka_stream_network_outbound_limiter_queue_time_50p(99p/mean/max)_nanoseconds​

The queuing time in the throttling queue for outbound traffic requests of cold reads and compaction when they get executed.

  • Type: Gauge

Kafka_stream_operation_latency_50p(99p/mean/max)_nanoseconds​

The operation duration at each stage of the AutoMQ S3Stream module.

  • Type: Gauge

  • Labels:

    • operation_type

    • operation_name