Integrating Metrics with Datadog

AutoMQ supports exporting metrics to Datadog for monitoring and alerting. This guide covers two Gateway-style integration approaches based on where the collector runs:

Kubernetes: Use the Datadog Agent Helm chart’s OTEL Agent Gateway to scrape Prometheus metrics from AutoMQ pods.
Linux: Use a Gateway OTEL Collector or Datadog DDOT Collector on Linux to scrape AutoMQ Prometheus metrics endpoints and forward metrics to Datadog.

Prerequisites

A running AutoMQ Cloud BYOC cluster. BYOC clusters expose the AutoMQ Prometheus metrics endpoint by default.
A valid Datadog API Key
Your Datadog Site URL (e.g., datadoghq.com, us3.datadoghq.com, datadoghq.eu)

Option 1: Kubernetes with Datadog Agent Gateway

In this approach, the Datadog Agent Helm chart runs a single OTEL Agent Gateway Deployment. The Gateway scrapes Prometheus metrics from AutoMQ pods using Kubernetes service discovery, reduces metric cardinality before ingest, then exports the metrics to Datadog.

AutoMQ Pods (Prometheus metrics endpoint on port 9090)
    ↓  Prometheus scrape (Kubernetes service discovery)
Datadog OTEL Agent Gateway
    ↓  Export to Datadog
Datadog

Step 1: Confirm AutoMQ metrics export

AutoMQ Cloud BYOC Kubernetes clusters created by the Console expose Prometheus metrics on the AutoMQ broker and controller pods by default. No AutoMQ Helm values changes are required in the standard BYOC flow.If you manage an AutoMQ Software deployment on Kubernetes and need to manually enable metrics export, see Appendix: Manually enable AutoMQ metrics export for Kubernetes Software.

Step 2: Create the Datadog Agent Gateway values file

Create a datadog-values.yaml file with the OTEL Agent Gateway configuration. The Gateway uses Kubernetes service discovery to find AutoMQ broker and controller pods. Set <your-automq-namespace> to the Kubernetes namespace where your AutoMQ cluster runs. If you need to collect multiple AutoMQ namespaces with the same Datadog Agent, add each namespace to kubernetes_sd_configs.namespaces.names. The same collection profile can be used for AutoMQ Software and AutoMQ Cloud BYOC/CMP clusters. The standard dashboards show cluster_id as the required cluster filter; this dashboard filter reads the Datadog service tag derived from AutoMQ’s Prometheus job label. The dashboards also include env_id and instance_id as optional filters when those labels are exposed by the AutoMQ metrics. The profile does not synthesize env_id or instance_id from Kubernetes metadata. If AutoMQ metrics already expose those labels, honor_labels: true preserves the AutoMQ-provided values.

agents:
  enabled: false
clusterAgent:
  enabled: false
  admissionController:
    enabled: false
datadog:
  kubeStateMetricsCore:
    enabled: false
  orchestratorExplorer:
    enabled: false
  operator:
    enabled: false
  instrumentationCrd:
    enabled: false
  autoscaling:
    workload:
      enabled: false
datadog-crds:
  crds:
    datadogMetrics: false
    datadogPodAutoscalers: false
    datadogPodAutoscalerClusterProfiles: false
    datadogInstrumentations: false
otelAgentGateway:
  enabled: true
  replicas: 1
  rbac:
    create: true
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "list", "watch"]
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "automq"
              scrape_interval: 15s
              honor_labels: true
              kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - <your-automq-namespace>
                  selectors:
                    - role: pod
                      label: 'app.kubernetes.io/name=automq-enterprise,app.kubernetes.io/component in (broker,controller)'
                      field: 'status.phase=Running'
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_container_name]
                  action: keep
                  regex: 'automq-(broker|controller)'
                - source_labels: [__meta_kubernetes_pod_container_port_number]
                  action: keep
                  regex: "9090"
              metric_relabel_configs:
                - source_labels: [__name__]
                  action: keep
                  regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$"
                - action: labeldrop
                  regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$"
    exporters:
      datadog:
        api:
          key: ${env:DD_API_KEY}
          site: ${env:DD_SITE}
    processors:
      cumulativetodelta/automq_counters:
        initial_value: auto
        include:
          match_type: strict
          metrics:
            - kafka_network_io_bytes_total
            - kafka_message_count_total
            - kafka_topic_request_count_total
            - kafka_broker_network_io_bytes_total
            - kafka_request_error_count_total
      transform/automq_rollups:
        error_mode: ignore
        metric_statements:
          - context: metric
            statements:
              - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size"
              - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset"
              - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total"
              - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total"
              - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total"
              - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total"
              - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total"
              - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count"
              - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count"
              - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count"
              - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count"
              - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count"
              - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count"
              - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count"
              - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count"
              - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset"
              - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count"
              - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds"
              - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds"
          - context: datapoint
            statements:
              - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil
              - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil
          - context: metric
            statements:
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset"
              - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag"
      filter/drop_raw_replaced_metrics:
        metrics:
          metric:
            - 'name == "kafka_log_size"'
            - 'name == "kafka_log_end_offset"'
            - 'name == "kafka_network_io_bytes_total"'
            - 'name == "kafka_message_count_total"'
            - 'name == "kafka_broker_network_io_bytes_total"'
            - 'name == "kafka_topic_request_count_total"'
            - 'name == "kafka_request_error_count_total"'
            - 'name == "kafka_stream_topic_partition_count"'
            - 'name == "kafka_group_count"'
            - 'name == "kafka_group_stable_count"'
            - 'name == "kafka_group_empty_count"'
            - 'name == "kafka_group_dead_count"'
            - 'name == "kafka_group_preparing_rebalance_count"'
            - 'name == "kafka_group_completing_rebalance_count"'
            - 'name == "kafka_group_commit_offset"'
            - 'name == "consumer_offset_lag_count"'
            - 'name == "consumer_estimated_time_lag_milliseconds"'
      infraattributes:
        cardinality: 2
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, infraattributes]
          exporters: [datadog]

This example uses the following components:

Component	Purpose
`otelAgentGateway`	Runs one OTEL Agent Gateway Deployment for this collection profile. A single Gateway performs the scrape and rollup before exporting metrics to Datadog.
`prometheus` receiver	Uses Kubernetes service discovery to find running AutoMQ broker and controller pods in the selected namespace and scrape port `9090`. `honor_labels: true` preserves AutoMQ-provided labels such as Prometheus `job`, `env_id`, and `instance_id`. The container-name relabel rule keeps only AutoMQ broker/controller containers and excludes sidecars such as Schema Registry.
`metric_relabel_configs`	Keeps only the exact raw metrics required by the standard-minimal dashboards or by the `automq_*` rollup processor. The profile does not copy `job` to `cluster_id`; Datadog maps the Prometheus `job` label to the `service` tag, and dashboards keep the `cluster_id` filter name while reading values from `service`.
`cumulativetodelta/automq_counters` processor	Converts selected Prometheus cumulative counters to deltas before rollup. This avoids aggregating cumulative counter values from multiple AutoMQ nodes before a rate is calculated.
`transform/automq_rollups` processor	Creates lower-cardinality `automq_*` rollup metrics before Datadog ingest. This profile uses Collector-side OTTL aggregation.
`filter/drop_raw_replaced_metrics` processor	Drops the raw source metrics after their rollup metrics have been created, reducing Datadog custom metric volume while keeping the customer-facing dashboard panels functional.
`datadog` exporter	Sends metrics to Datadog. The API key and site come from the Agent’s environment variables.
`infraattributes` processor	Optional. Adds Datadog infrastructure tags when the required resource attributes are available.

We recommend scheduling the OTEL Agent Gateway on general-purpose nodes or dedicated observability nodes. It does not need to run on the same nodes as AutoMQ brokers or controllers. If your Kubernetes cluster requires custom Gateway scheduling, resource sizing, pod annotations, or pod labels, configure the corresponding Datadog Helm values such as otelAgentGateway.nodeSelector, otelAgentGateway.affinity, otelAgentGateway.tolerations, otelAgentGateway.podAnnotations, otelAgentGateway.additionalLabels, and otelAgentGateway.containers.otelAgent.resources. For details, see the Datadog DDOT Collector Gateway Helm installation guide and the Datadog Helm chart values.yaml.

Standard dashboard collection profile

The recommended configuration is designed for the AutoMQ customer-facing Datadog dashboards. It keeps only the minimum dimensions needed by those dashboards:

Metric area	Retained dimensions
Cluster health and capacity	`service`
CPU and broker metadata	`service`, `broker_id`
Requests, errors, latency, and broker traffic	`service`, `topic`
Topic throughput, message count, size, and offsets	`service`, `topic`
Topic partition distribution across racks	`service`, `topic`, `rack`
Consumer group lag, offset, and throughput	`service`, `consumer_group` or `group`, `topic`

The standard-minimal profile does not keep raw partition-level series, topic-by-broker traffic / offset / size series, broker/node-level drilldown for JVM, connection, request, latency, or traffic series, consumer_group * topic * broker series, non-dashboard kafka_stream_* internal metrics, Kafka Linking metrics, or Detailed Metrics-only internal diagnostic dimensions.

This example is the recommended standard-minimal collection profile for AutoMQ customer-facing dashboards. It reduces Datadog custom metric volume and cardinality by collecting only the exact raw metrics required by the standard AutoMQ dashboards or rollup processor, converting request, traffic, topic, partition, and consumer group sources to lower-cardinality automq_* rollups, and dropping the raw sources after the rollups are created. The standard-minimal profile keeps cluster, topic, consumer group, and rack dimensions, plus CPU and broker metadata, but it does not keep partition-level series, topic-by-broker series, broker/node-level drilldown series, or consumer-group/topic/broker series.

This example includes a Prometheus scrape job and a metrics pipeline for AutoMQ. It also uses the optional infraattributes processor to add Datadog infrastructure tags when the required resource attributes are available. The OTLP receiver and other pipelines (traces, logs) are omitted. If you need to collect OTLP data from other applications through the same Collector, refer to the Datadog OTEL Agent Gateway documentation to add the corresponding receivers and pipelines.

Step 3: Deploy the Datadog Agent

Add the Datadog Helm repository and create a Kubernetes secret for your API key:

helm repo add datadog https://helm.datadoghq.com
helm repo update

kubectl create namespace datadog
kubectl -n datadog create secret generic datadog-secret \
  --from-literal api-key=<your-datadog-api-key>

Install the Datadog Agent using the values file from Step 2:

helm install datadog-agent datadog/datadog \
  -f datadog-values.yaml \
  -n datadog \
  --set datadog.apiKeyExistingSecret=datadog-secret \
  --set datadog.site=<your-datadog-site>

Replace <your-datadog-site> with your Datadog Site (e.g., datadoghq.com, us3.datadoghq.com, datadoghq.eu).

Step 4: Verify the integration

Verify that the Datadog Agent service account can discover pods:

kubectl auth can-i list pods \
  --as=system:serviceaccount:datadog:datadog-agent \
  --all-namespaces

Verify that the OTEL Agent Gateway Deployment is running:

kubectl -n datadog rollout status deployment/datadog-agent-datadog-otel-agent-gateway

Check the OTEL Collector logs and confirm that the AutoMQ scrape job has been added:

kubectl -n datadog logs deployment/datadog-agent-datadog-otel-agent-gateway --since=10m | grep -i automq

Then verify metrics in Datadog:

Open Datadog and go to Metrics > Explorer.
Search for AutoMQ metrics by typing a metric name prefix such as kafka_broker_active_count.
Filter by service:<your-cluster-id> to confirm that the metrics belong to the expected AutoMQ cluster. In the imported dashboards, the filter is still named cluster_id, but it reads values from the Datadog service tag. If your AutoMQ environment exposes native env_id or instance_id labels, you can use them as additional filters.
New metrics may take 1–3 minutes to appear.

Step 5: Import AutoMQ Datadog dashboards

After AutoMQ metrics appear in Datadog, download the Datadog dashboard JSON templates and import them into your Datadog organization.

Dashboard	JSON template
Cluster Overview	Download JSON
Topic Metrics	Download JSON
Group Metrics	Download JSON

To import a dashboard, open Datadog, go to Dashboards, create or open a dashboard, choose the dashboard JSON import option, and paste or upload the downloaded JSON content.

These dashboard templates are designed for the metric names, tags, and collection reduction profile shown in Step 2. Cluster, Topic, and Group views use the automq_* rollup metrics created by the OTEL transform processor. The standard dashboards show cluster_id as the required filter, mapped to the Datadog service tag, and expose env_id and instance_id as optional filters when those labels are present. The AutoMQ Version value should be discovered from the node metadata gauges (kafka_node_metadata, kafka_node_info, or kafka_node) and rendered by the dashboard generation/import workflow. Do not derive it from the Helm Chart/App version label because that label can differ from the runtime image version.

AutoMQ metrics use Prometheus naming conventions with underscores (e.g., kafka_broker_active_count). Datadog’s built-in Apache Kafka dashboards rely on the Datadog Kafka integration, which uses dot-separated names (e.g., kafka.request.time). As a result, AutoMQ metrics do not appear in those built-in dashboards. Use Datadog Metrics Explorer to query AutoMQ metrics directly, or build custom dashboards using the underscore-separated names. For available metrics, see Prometheus Metrics.

The screenshots below illustrate this naming difference — Datadog’s built-in dashboard uses dot-separated format, while AutoMQ metrics use underscore-separated format:

Option 2: Linux with Gateway OTEL Collector

In this approach, one OTEL Collector or Datadog DDOT Collector runs as a Gateway on Linux and scrapes all AutoMQ Prometheus metrics endpoints with static targets. Use the same standard-minimal metric reduction profile as the Kubernetes approach. The Linux-specific difference is service discovery: use a static target list instead of Kubernetes service discovery.

AutoMQ Prometheus metrics endpoints
    ↓  Prometheus scrape from /metrics
Gateway OTEL Collector / DDOT Collector on Linux
    ↓  Forward to Datadog
Datadog

Step 1: Confirm AutoMQ metrics export

AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. No server.properties changes are required in the standard BYOC flow.

Step 2: Install the Collector

Choose one Collector distribution and keep the configuration path and lifecycle commands aligned with that choice.

Datadog DDOT Collector
Upstream otelcol-contrib

Install the Datadog Agent with the embedded DDOT Collector. The official Linux install flow manages the Collector through datadog-agent and uses /etc/datadog-agent/otel-config.yaml as the OpenTelemetry Collector configuration file.

DD_API_KEY="<your-api-key>" \
DD_SITE="<your-datadog-site>" \
DD_OTELCOLLECTOR_ENABLED=true \
DD_AGENT_MAJOR_VERSION=7 \
bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

sudo datadog-agent status

If the Datadog Agent is already installed, follow the Datadog DDOT Collector documentation for the matching Agent version.

Install an OpenTelemetry Collector distribution that includes the Prometheus receiver, Datadog exporter, transform processor, filter processor, cumulative-to-delta processor, memory limiter processor, batch processor, and health check extension. The examples below use /etc/otelcol/config.yaml and a custom otelcol systemd service.Confirm the Collector binary is available:

<otel-collector-binary> --version

Step 3: Configure the Collector

Create the Collector configuration file. The following configuration is the Linux version of the standard-minimal profile: it uses static Prometheus scrape targets, keeps only the raw metrics needed by the standard dashboards or rollup processor, creates lower-cardinality automq_* rollup metrics, and drops the raw source metrics after the rollups are created. Run this configuration once per AutoMQ cluster from a Collector host that can reach every AutoMQ metrics endpoint. Do not deploy the same profile on every AutoMQ node with only 127.0.0.1:9090 as the target, because local collectors would roll up counters independently before exporting to Datadog. Use the configuration path for your selected Collector:

Datadog DDOT Collector: /etc/datadog-agent/otel-config.yaml
Upstream otelcol-contrib: /etc/otelcol/config.yaml

COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector
# COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml"          # Upstream otelcol-contrib

sudo mkdir -p "$(dirname "$COLLECTOR_CONFIG_PATH")"

COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector
# COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml"          # Upstream otelcol-contrib

sudo tee "$COLLECTOR_CONFIG_PATH" > /dev/null << 'EOF'
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "<your-cluster-id>"
          scrape_interval: 15s
          honor_labels: true
          static_configs:
            - targets:
                - "<automq-node-1-host>:9090"
                - "<automq-node-2-host>:9090"
                - "<automq-node-3-host>:9090"
              labels:
                env_id: "<your-env-id>"
                instance_id: "<your-instance-id>"
          metric_relabel_configs:
            - source_labels: [__name__]
              action: keep
              regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$"
            - action: labeldrop
              regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$"
exporters:
  datadog:
    api:
      # Upstream otelcol-contrib: keep these environment references and set DD_API_KEY/DD_SITE before starting the service.
      # Datadog DDOT Collector: either set these environment variables for datadog-agent, or remove this api block to use the core Agent configuration.
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE}
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
  batch:
    timeout: 10s
    send_batch_size: 10000
  cumulativetodelta/automq_counters:
    initial_value: auto
    include:
      match_type: strict
      metrics:
        - kafka_network_io_bytes_total
        - kafka_message_count_total
        - kafka_topic_request_count_total
        - kafka_broker_network_io_bytes_total
        - kafka_request_error_count_total
  transform/automq_rollups:
    error_mode: ignore
    metric_statements:
      - context: metric
        statements:
          - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size"
          - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset"
          - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total"
          - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total"
          - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total"
          - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total"
          - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total"
          - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count"
          - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count"
          - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count"
          - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count"
          - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count"
          - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count"
          - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count"
          - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count"
          - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset"
          - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count"
          - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds"
          - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds"
      - context: datapoint
        statements:
          - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil
          - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil
      - context: metric
        statements:
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset"
          - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag"
  filter/drop_raw_replaced_metrics:
    metrics:
      metric:
        - 'name == "kafka_log_size"'
        - 'name == "kafka_log_end_offset"'
        - 'name == "kafka_network_io_bytes_total"'
        - 'name == "kafka_message_count_total"'
        - 'name == "kafka_broker_network_io_bytes_total"'
        - 'name == "kafka_topic_request_count_total"'
        - 'name == "kafka_request_error_count_total"'
        - 'name == "kafka_stream_topic_partition_count"'
        - 'name == "kafka_group_count"'
        - 'name == "kafka_group_stable_count"'
        - 'name == "kafka_group_empty_count"'
        - 'name == "kafka_group_dead_count"'
        - 'name == "kafka_group_preparing_rebalance_count"'
        - 'name == "kafka_group_completing_rebalance_count"'
        - 'name == "kafka_group_commit_offset"'
        - 'name == "consumer_offset_lag_count"'
        - 'name == "consumer_estimated_time_lag_milliseconds"'
service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, batch]
      exporters: [datadog]
EOF

With honor_labels: true, if the scraped AutoMQ endpoint already exposes a Prometheus job label, that target-provided value is preserved and job_name is not used for that label. Standard AutoMQ metrics normally expose job as the cluster identifier, and Datadog maps that label to the service tag used by the dashboard cluster_id filter. If the endpoint does not expose job, job_name becomes the fallback value. The static env_id and instance_id labels are optional; set them only when you want those Datadog dashboard filters to have values for this Linux collection path.

This Gateway profile performs rollups in the Collector to reduce Datadog custom metrics cardinality. It is not equivalent to a Prometheus recording rule and cannot express max by (..., partition) across scrape targets followed by sum by (...). If you need strict deduplication when the same partition gauge is temporarily exposed by multiple AutoMQ nodes during partition migration, add a Prometheus recording-rule / federation layer before Datadog, or keep the partition dimension and handle it explicitly in Datadog queries.

Step 4: Start the Collector

Datadog DDOT Collector
Upstream otelcol-contrib

Restart the Datadog Agent and verify the embedded Collector status:

sudo systemctl restart datadog-agent
sudo datadog-agent status

For a foreground smoke test, run the Collector directly:

DD_API_KEY="<your-api-key>" \
DD_SITE="<your-datadog-site>" \
<otel-collector-binary> --config /etc/otelcol/config.yaml

For a systemd deployment, create a dedicated service user and environment file:

sudo useradd --system --home /var/lib/otelcol --shell /usr/sbin/nologin otelcol || true
sudo install -d -o otelcol -g otelcol /var/lib/otelcol

sudo tee /etc/default/otelcol > /dev/null << 'EOF'
DD_API_KEY=<your-api-key>
DD_SITE=<your-datadog-site>
EOF

Limit access to this file because it contains your Datadog API key. Also avoid pasting real keys into shared shell history or logs.

sudo chmod 600 /etc/default/otelcol

sudo tee /etc/systemd/system/otelcol.service > /dev/null << 'EOF'
[Unit]
Description=OpenTelemetry Collector for AutoMQ Datadog metrics
After=network-online.target
Requires=network-online.target

[Service]
User=otelcol
Group=otelcol
EnvironmentFile=/etc/default/otelcol
ExecStart=<otel-collector-binary-path> --config /etc/otelcol/config.yaml
Restart=always
RestartSec=5
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
StateDirectory=otelcol

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable otelcol
sudo systemctl start otelcol

Step 5: Verify

Open Datadog Metrics > Explorer, search for AutoMQ metrics such as kafka_broker_active_count, and filter by service:<expected-cluster-id>. In the imported dashboards, the filter is still named cluster_id, but it reads values from the Datadog service tag. If the expected service value does not appear, check one AutoMQ /metrics endpoint and confirm the actual job label because honor_labels: true preserves the target-provided job value. If your AutoMQ environment exposes native env_id, instance_id, or broker_id labels, you can use them as additional filters. New metrics may take 1–3 minutes to appear.

Appendix: Manually enable AutoMQ metrics export for Kubernetes Software

AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. Use this appendix only when you manage an AutoMQ Software deployment on Kubernetes and need to enable the endpoint manually. For AutoMQ Software deployments on Kubernetes via Helm, set the metrics exporter to Prometheus mode in your Helm values:

global:
  config: |
    s3.telemetry.metrics.exporter.uri=prometheus://?host=0.0.0.0&port=9090

This exposes Prometheus-format metrics on port 9090 of each AutoMQ pod. The Datadog Agent Gateway profile in this guide uses selector-based Kubernetes discovery, so it does not require Prometheus scrape annotations to discover AutoMQ pods. If you also use an annotation-based Prometheus or Datadog collector outside of this profile, add the following annotations to the AutoMQ Helm values:

controller:
  annotations:
    prometheus.io/automq-scrape: "true"
    prometheus.io/automq-port: "9090"
    prometheus.io/automq-path: "/metrics"

broker:
  annotations:
    prometheus.io/automq-scrape: "true"
    prometheus.io/automq-port: "9090"
    prometheus.io/automq-path: "/metrics"

Overview

Getting Started

Deploy on Kubernetes

Subscriptions and Billings

Manage Environments

Manage Identities and Access

Managing AutoMQ

Migrate to AutoMQ

Manage Security

Kafka Connect

Monitoring & Alert

Table Topic

Best Practice

Eliminate Inter-Zone Traffics

Release Notes

Support

Appendix

Prerequisites

Option 1: Kubernetes with Datadog Agent Gateway

Step 1: Confirm AutoMQ metrics export

Step 2: Create the Datadog Agent Gateway values file

Standard dashboard collection profile

Step 3: Deploy the Datadog Agent

Step 4: Verify the integration

Step 5: Import AutoMQ Datadog dashboards

Option 2: Linux with Gateway OTEL Collector

Step 1: Confirm AutoMQ metrics export

Step 2: Install the Collector

Step 3: Configure the Collector

Step 4: Start the Collector

Step 5: Verify

Appendix: Manually enable AutoMQ metrics export for Kubernetes Software

References

​Prerequisites

​Option 1: Kubernetes with Datadog Agent Gateway

​Step 1: Confirm AutoMQ metrics export

​Step 2: Create the Datadog Agent Gateway values file

​Standard dashboard collection profile

​Step 3: Deploy the Datadog Agent

​Step 4: Verify the integration

​Step 5: Import AutoMQ Datadog dashboards

​Option 2: Linux with Gateway OTEL Collector

​Step 1: Confirm AutoMQ metrics export

​Step 2: Install the Collector

​Step 3: Configure the Collector

​Step 4: Start the Collector

​Step 5: Verify

​Appendix: Manually enable AutoMQ metrics export for Kubernetes Software

​References

Prerequisites

Option 1: Kubernetes with Datadog Agent Gateway

Step 1: Confirm AutoMQ metrics export

Step 2: Create the Datadog Agent Gateway values file

Standard dashboard collection profile

Step 3: Deploy the Datadog Agent

Step 4: Verify the integration

Step 5: Import AutoMQ Datadog dashboards

Option 2: Linux with Gateway OTEL Collector

Step 1: Confirm AutoMQ metrics export

Step 2: Install the Collector

Step 3: Configure the Collector

Step 4: Start the Collector

Step 5: Verify

Appendix: Manually enable AutoMQ metrics export for Kubernetes Software

References