> ## Documentation Index > Fetch the complete documentation index at: https://docs.automq.com/llms.txt > Use this file to discover all available pages before exploring further. # Integrating Metrics with Datadog > Send AutoMQ metrics to Datadog using a Gateway-style Datadog Agent or OTEL Collector. AutoMQ supports exporting metrics to [Datadog](https://www.datadoghq.com/) for monitoring and alerting. This guide covers two Gateway-style integration approaches based on where the collector runs: * **Kubernetes**: Use the Datadog Agent Helm chart's OTEL Agent Gateway to scrape Prometheus metrics from AutoMQ pods. * **Linux**: Use a Gateway OTEL Collector or Datadog DDOT Collector on Linux to scrape AutoMQ Prometheus metrics endpoints and forward metrics to Datadog. ## Prerequisites * A running AutoMQ Cloud BYOC cluster. BYOC clusters expose the AutoMQ Prometheus metrics endpoint by default. * A valid [Datadog API Key](https://docs.datadoghq.com/account_management/api-app-keys/) * Your Datadog Site URL (e.g., `datadoghq.com`, `us3.datadoghq.com`, `datadoghq.eu`) ## Option 1: Kubernetes with Datadog Agent Gateway In this approach, the Datadog Agent Helm chart runs a single OTEL Agent Gateway Deployment. The Gateway scrapes Prometheus metrics from AutoMQ pods using Kubernetes service discovery, reduces metric cardinality before ingest, then exports the metrics to Datadog. ```text theme={null} AutoMQ Pods (Prometheus metrics endpoint on port 9090) ↓ Prometheus scrape (Kubernetes service discovery) Datadog OTEL Agent Gateway ↓ Export to Datadog Datadog ``` ### Step 1: Confirm AutoMQ metrics export AutoMQ Cloud BYOC Kubernetes clusters created by the Console expose Prometheus metrics on the AutoMQ broker and controller pods by default. No AutoMQ Helm values changes are required in the standard BYOC flow. If you manage an AutoMQ Software deployment on Kubernetes and need to manually enable metrics export, see [Appendix: Manually enable AutoMQ metrics export for Kubernetes Software](#appendix-manually-enable-automq-metrics-export-for-kubernetes-software). ### Step 2: Create the Datadog Agent Gateway values file Create a `datadog-values.yaml` file with the OTEL Agent Gateway configuration. The Gateway uses Kubernetes service discovery to find AutoMQ broker and controller pods. Set `` to the Kubernetes namespace where your AutoMQ cluster runs. If you need to collect multiple AutoMQ namespaces with the same Datadog Agent, add each namespace to `kubernetes_sd_configs.namespaces.names`. The same collection profile can be used for AutoMQ Software and AutoMQ Cloud BYOC/CMP clusters. The standard dashboards show `cluster_id` as the required cluster filter; this dashboard filter reads the Datadog `service` tag derived from AutoMQ's Prometheus `job` label. The dashboards also include `env_id` and `instance_id` as optional filters when those labels are exposed by the AutoMQ metrics. The profile does not synthesize `env_id` or `instance_id` from Kubernetes metadata. If AutoMQ metrics already expose those labels, `honor_labels: true` preserves the AutoMQ-provided values. ```yaml theme={null} agents: enabled: false clusterAgent: enabled: false admissionController: enabled: false datadog: kubeStateMetricsCore: enabled: false orchestratorExplorer: enabled: false operator: enabled: false instrumentationCrd: enabled: false autoscaling: workload: enabled: false datadog-crds: crds: datadogMetrics: false datadogPodAutoscalers: false datadogPodAutoscalerClusterProfiles: false datadogInstrumentations: false otelAgentGateway: enabled: true replicas: 1 rbac: create: true rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] config: | receivers: prometheus: config: scrape_configs: - job_name: "automq" scrape_interval: 15s honor_labels: true kubernetes_sd_configs: - role: pod namespaces: names: - selectors: - role: pod label: 'app.kubernetes.io/name=automq-enterprise,app.kubernetes.io/component in (broker,controller)' field: 'status.phase=Running' relabel_configs: - source_labels: [__meta_kubernetes_pod_container_name] action: keep regex: 'automq-(broker|controller)' - source_labels: [__meta_kubernetes_pod_container_port_number] action: keep regex: "9090" metric_relabel_configs: - source_labels: [__name__] action: keep regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$" - action: labeldrop regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$" exporters: datadog: api: key: ${env:DD_API_KEY} site: ${env:DD_SITE} processors: cumulativetodelta/automq_counters: initial_value: auto include: match_type: strict metrics: - kafka_network_io_bytes_total - kafka_message_count_total - kafka_topic_request_count_total - kafka_broker_network_io_bytes_total - kafka_request_error_count_total transform/automq_rollups: error_mode: ignore metric_statements: - context: metric statements: - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size" - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset" - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total" - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total" - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total" - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total" - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total" - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count" - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count" - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count" - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count" - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count" - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count" - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count" - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count" - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset" - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count" - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds" - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds" - context: datapoint statements: - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil - context: metric statements: - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag" filter/drop_raw_replaced_metrics: metrics: metric: - 'name == "kafka_log_size"' - 'name == "kafka_log_end_offset"' - 'name == "kafka_network_io_bytes_total"' - 'name == "kafka_message_count_total"' - 'name == "kafka_broker_network_io_bytes_total"' - 'name == "kafka_topic_request_count_total"' - 'name == "kafka_request_error_count_total"' - 'name == "kafka_stream_topic_partition_count"' - 'name == "kafka_group_count"' - 'name == "kafka_group_stable_count"' - 'name == "kafka_group_empty_count"' - 'name == "kafka_group_dead_count"' - 'name == "kafka_group_preparing_rebalance_count"' - 'name == "kafka_group_completing_rebalance_count"' - 'name == "kafka_group_commit_offset"' - 'name == "consumer_offset_lag_count"' - 'name == "consumer_estimated_time_lag_milliseconds"' infraattributes: cardinality: 2 service: pipelines: metrics: receivers: [prometheus] processors: [cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, infraattributes] exporters: [datadog] ``` This example uses the following components: | Component | Purpose | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `otelAgentGateway` | Runs one OTEL Agent Gateway Deployment for this collection profile. A single Gateway performs the scrape and rollup before exporting metrics to Datadog. | | `prometheus` receiver | Uses Kubernetes service discovery to find running AutoMQ broker and controller pods in the selected namespace and scrape port `9090`. `honor_labels: true` preserves AutoMQ-provided labels such as Prometheus `job`, `env_id`, and `instance_id`. The container-name relabel rule keeps only AutoMQ broker/controller containers and excludes sidecars such as Schema Registry. | | `metric_relabel_configs` | Keeps only the exact raw metrics required by the standard-minimal dashboards or by the `automq_*` rollup processor. The profile does not copy `job` to `cluster_id`; Datadog maps the Prometheus `job` label to the `service` tag, and dashboards keep the `cluster_id` filter name while reading values from `service`. | | `cumulativetodelta/automq_counters` processor | Converts selected Prometheus cumulative counters to deltas before rollup. This avoids aggregating cumulative counter values from multiple AutoMQ nodes before a rate is calculated. | | `transform/automq_rollups` processor | Creates lower-cardinality `automq_*` rollup metrics before Datadog ingest. This profile uses Collector-side OTTL aggregation. | | `filter/drop_raw_replaced_metrics` processor | Drops the raw source metrics after their rollup metrics have been created, reducing Datadog custom metric volume while keeping the customer-facing dashboard panels functional. | | `datadog` exporter | Sends metrics to Datadog. The API key and site come from the Agent's environment variables. | | `infraattributes` processor | Optional. Adds Datadog infrastructure tags when the required resource attributes are available. | We recommend scheduling the OTEL Agent Gateway on general-purpose nodes or dedicated observability nodes. It does not need to run on the same nodes as AutoMQ brokers or controllers. If your Kubernetes cluster requires custom Gateway scheduling, resource sizing, pod annotations, or pod labels, configure the corresponding Datadog Helm values such as `otelAgentGateway.nodeSelector`, `otelAgentGateway.affinity`, `otelAgentGateway.tolerations`, `otelAgentGateway.podAnnotations`, `otelAgentGateway.additionalLabels`, and `otelAgentGateway.containers.otelAgent.resources`. For details, see the [Datadog DDOT Collector Gateway Helm installation guide](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes_gateway/) and the [Datadog Helm chart values.yaml](https://github.com/DataDog/helm-charts/blob/main/charts/datadog/values.yaml). #### Standard dashboard collection profile The recommended configuration is designed for the AutoMQ customer-facing Datadog dashboards. It keeps only the minimum dimensions needed by those dashboards: | Metric area | Retained dimensions | | -------------------------------------------------- | ----------------------------------------------- | | Cluster health and capacity | `service` | | CPU and broker metadata | `service`, `broker_id` | | Requests, errors, latency, and broker traffic | `service`, `topic` | | Topic throughput, message count, size, and offsets | `service`, `topic` | | Topic partition distribution across racks | `service`, `topic`, `rack` | | Consumer group lag, offset, and throughput | `service`, `consumer_group` or `group`, `topic` | The standard-minimal profile does not keep raw partition-level series, topic-by-broker traffic / offset / size series, broker/node-level drilldown for JVM, connection, request, latency, or traffic series, `consumer_group * topic * broker` series, non-dashboard `kafka_stream_*` internal metrics, Kafka Linking metrics, or Detailed Metrics-only internal diagnostic dimensions. This example is the recommended standard-minimal collection profile for AutoMQ customer-facing dashboards. It reduces Datadog custom metric volume and cardinality by collecting only the exact raw metrics required by the standard AutoMQ dashboards or rollup processor, converting request, traffic, topic, partition, and consumer group sources to lower-cardinality `automq_*` rollups, and dropping the raw sources after the rollups are created. The standard-minimal profile keeps cluster, topic, consumer group, and rack dimensions, plus CPU and broker metadata, but it does not keep partition-level series, topic-by-broker series, broker/node-level drilldown series, or consumer-group/topic/broker series. This example includes a Prometheus scrape job and a metrics pipeline for AutoMQ. It also uses the optional `infraattributes` processor to add Datadog infrastructure tags when the required resource attributes are available. The OTLP receiver and other pipelines (traces, logs) are omitted. If you need to collect OTLP data from other applications through the same Collector, refer to the [Datadog OTEL Agent Gateway documentation](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes/?tab=helm) to add the corresponding receivers and pipelines. ### Step 3: Deploy the Datadog Agent Add the Datadog Helm repository and create a Kubernetes secret for your API key: ```bash theme={null} helm repo add datadog https://helm.datadoghq.com helm repo update ``` ```bash theme={null} kubectl create namespace datadog kubectl -n datadog create secret generic datadog-secret \ --from-literal api-key= ``` Install the Datadog Agent using the values file from Step 2: ```bash theme={null} helm install datadog-agent datadog/datadog \ -f datadog-values.yaml \ -n datadog \ --set datadog.apiKeyExistingSecret=datadog-secret \ --set datadog.site= ``` Replace `` with your Datadog Site (e.g., `datadoghq.com`, `us3.datadoghq.com`, `datadoghq.eu`). ### Step 4: Verify the integration Verify that the Datadog Agent service account can discover pods: ```bash theme={null} kubectl auth can-i list pods \ --as=system:serviceaccount:datadog:datadog-agent \ --all-namespaces ``` Verify that the OTEL Agent Gateway Deployment is running: ```bash theme={null} kubectl -n datadog rollout status deployment/datadog-agent-datadog-otel-agent-gateway ``` Check the OTEL Collector logs and confirm that the AutoMQ scrape job has been added: ```bash theme={null} kubectl -n datadog logs deployment/datadog-agent-datadog-otel-agent-gateway --since=10m | grep -i automq ``` Then verify metrics in Datadog: 1. Open Datadog and go to **Metrics > Explorer**. 2. Search for AutoMQ metrics by typing a metric name prefix such as `kafka_broker_active_count`. 3. Filter by `service:` to confirm that the metrics belong to the expected AutoMQ cluster. In the imported dashboards, the filter is still named `cluster_id`, but it reads values from the Datadog `service` tag. If your AutoMQ environment exposes native `env_id` or `instance_id` labels, you can use them as additional filters. 4. New metrics may take 1–3 minutes to appear. AutoMQ metrics visible in Datadog Metrics Explorer after Kubernetes integration AutoMQ metrics detail view in Datadog Metrics Explorer ### Step 5: Import AutoMQ Datadog dashboards After AutoMQ metrics appear in Datadog, download the Datadog dashboard JSON templates and import them into your Datadog organization. | Dashboard | JSON template | | ---------------- | --------------------------------------------------------------------------------------------------------------------- | | Cluster Overview | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-cluster-overview.datadog.json) | | Topic Metrics | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-topic-metrics.datadog.json) | | Group Metrics | [Download JSON](/automq-cloud/monitoring-alert/integrating-metrics-with-datadog/automq-group-metrics.datadog.json) | To import a dashboard, open Datadog, go to **Dashboards**, create or open a dashboard, choose the dashboard JSON import option, and paste or upload the downloaded JSON content. These dashboard templates are designed for the metric names, tags, and collection reduction profile shown in Step 2. Cluster, Topic, and Group views use the `automq_*` rollup metrics created by the OTEL transform processor. The standard dashboards show `cluster_id` as the required filter, mapped to the Datadog `service` tag, and expose `env_id` and `instance_id` as optional filters when those labels are present. The AutoMQ Version value should be discovered from the node metadata gauges (`kafka_node_metadata`, `kafka_node_info`, or `kafka_node`) and rendered by the dashboard generation/import workflow. Do not derive it from the Helm Chart/App version label because that label can differ from the runtime image version. AutoMQ metrics use Prometheus naming conventions with underscores (e.g., `kafka_broker_active_count`). Datadog's built-in Apache Kafka dashboards rely on the Datadog Kafka integration, which uses dot-separated names (e.g., `kafka.request.time`). As a result, AutoMQ metrics do not appear in those built-in dashboards. Use Datadog Metrics Explorer to query AutoMQ metrics directly, or build custom dashboards using the underscore-separated names. For available metrics, see [Prometheus Metrics](/automq/observability/prometheus-metrics). The screenshots below illustrate this naming difference — Datadog's built-in dashboard uses dot-separated format, while AutoMQ metrics use underscore-separated format: Datadog built-in dashboard showing dot-separated metric naming convention AutoMQ metrics showing underscore-separated Prometheus naming convention ## Option 2: Linux with Gateway OTEL Collector In this approach, one OTEL Collector or Datadog DDOT Collector runs as a Gateway on Linux and scrapes all AutoMQ Prometheus metrics endpoints with static targets. Use the same standard-minimal metric reduction profile as the Kubernetes approach. The Linux-specific difference is service discovery: use a static target list instead of Kubernetes service discovery. ```text theme={null} AutoMQ Prometheus metrics endpoints ↓ Prometheus scrape from /metrics Gateway OTEL Collector / DDOT Collector on Linux ↓ Forward to Datadog Datadog ``` ### Step 1: Confirm AutoMQ metrics export AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. No `server.properties` changes are required in the standard BYOC flow. ### Step 2: Install the Collector Choose one Collector distribution and keep the configuration path and lifecycle commands aligned with that choice. Install the Datadog Agent with the embedded DDOT Collector. The official Linux install flow manages the Collector through `datadog-agent` and uses `/etc/datadog-agent/otel-config.yaml` as the OpenTelemetry Collector configuration file. ```bash theme={null} DD_API_KEY="" \ DD_SITE="" \ DD_OTELCOLLECTOR_ENABLED=true \ DD_AGENT_MAJOR_VERSION=7 \ bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)" sudo datadog-agent status ``` If the Datadog Agent is already installed, follow the [Datadog DDOT Collector documentation](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/linux/) for the matching Agent version. Install an OpenTelemetry Collector distribution that includes the Prometheus receiver, Datadog exporter, transform processor, filter processor, cumulative-to-delta processor, memory limiter processor, batch processor, and health check extension. The examples below use `/etc/otelcol/config.yaml` and a custom `otelcol` systemd service. Confirm the Collector binary is available: ```bash theme={null} --version ``` ### Step 3: Configure the Collector Create the Collector configuration file. The following configuration is the Linux version of the standard-minimal profile: it uses static Prometheus scrape targets, keeps only the raw metrics needed by the standard dashboards or rollup processor, creates lower-cardinality `automq_*` rollup metrics, and drops the raw source metrics after the rollups are created. Run this configuration once per AutoMQ cluster from a Collector host that can reach every AutoMQ metrics endpoint. Do not deploy the same profile on every AutoMQ node with only `127.0.0.1:9090` as the target, because local collectors would roll up counters independently before exporting to Datadog. Use the configuration path for your selected Collector: * Datadog DDOT Collector: `/etc/datadog-agent/otel-config.yaml` * Upstream otelcol-contrib: `/etc/otelcol/config.yaml` ```bash theme={null} COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector # COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml" # Upstream otelcol-contrib sudo mkdir -p "$(dirname "$COLLECTOR_CONFIG_PATH")" ``` ```bash theme={null} COLLECTOR_CONFIG_PATH="/etc/datadog-agent/otel-config.yaml" # Datadog DDOT Collector # COLLECTOR_CONFIG_PATH="/etc/otelcol/config.yaml" # Upstream otelcol-contrib sudo tee "$COLLECTOR_CONFIG_PATH" > /dev/null << 'EOF' receivers: prometheus: config: scrape_configs: - job_name: "" scrape_interval: 15s honor_labels: true static_configs: - targets: - ":9090" - ":9090" - ":9090" labels: env_id: "" instance_id: "" metric_relabel_configs: - source_labels: [__name__] action: keep regex: "^(consumer_(offset_lag_count|estimated_time_lag_milliseconds)|jvm_cpu_(count|time_seconds_total)|kafka_(node(_metadata|_info)?|broker_(active_count|fenced_count|network_io_bytes_total)|group_(count|stable_count|empty_count|dead_count|preparing_rebalance_count|completing_rebalance_count|commit_offset)|log_(size|end_offset)|message_count_total|network_io_bytes_total|partition_total_count|request_error_count_total|stream_topic_partition_count|tabletopic_(delay_milliseconds|fps_fields_per_second)|topic_(count|request_count_total)))$" - action: labeldrop regex: "^(active_controller|customer|exported_env_id|exported_instance_id|exported_job|instance|link_id|operation_name|operation_type|otel_scope_name|pool_type|request|request_type|size|source_node_id)$" exporters: datadog: api: # Upstream otelcol-contrib: keep these environment references and set DD_API_KEY/DD_SITE before starting the service. # Datadog DDOT Collector: either set these environment variables for datadog-agent, or remove this api block to use the core Agent configuration. key: ${env:DD_API_KEY} site: ${env:DD_SITE} extensions: health_check: endpoint: 0.0.0.0:13133 processors: memory_limiter: check_interval: 1s limit_mib: 1024 spike_limit_mib: 256 batch: timeout: 10s send_batch_size: 10000 cumulativetodelta/automq_counters: initial_value: auto include: match_type: strict metrics: - kafka_network_io_bytes_total - kafka_message_count_total - kafka_topic_request_count_total - kafka_broker_network_io_bytes_total - kafka_request_error_count_total transform/automq_rollups: error_mode: ignore metric_statements: - context: metric statements: - copy_metric(name="automq_topic_log_size") where metric.name == "kafka_log_size" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_size" - copy_metric(name="automq_topic_log_end_offset") where metric.name == "kafka_log_end_offset" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_log_end_offset" - copy_metric(name="automq_topic_network_io_bytes_total") where metric.name == "kafka_network_io_bytes_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_network_io_bytes_total" - copy_metric(name="automq_topic_message_count_total") where metric.name == "kafka_message_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "direction"]) where metric.name == "automq_topic_message_count_total" - copy_metric(name="automq_topic_request_count_total") where metric.name == "kafka_topic_request_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "type"]) where metric.name == "automq_topic_request_count_total" - copy_metric(name="automq_cluster_broker_network_io_bytes_total") where metric.name == "kafka_broker_network_io_bytes_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "direction"]) where metric.name == "automq_cluster_broker_network_io_bytes_total" - copy_metric(name="automq_request_error_count_total") where metric.name == "kafka_request_error_count_total" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "error"]) where metric.name == "automq_request_error_count_total" - copy_metric(name="automq_topic_partition_count") where metric.name == "kafka_stream_topic_partition_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic"]) where metric.name == "automq_topic_partition_count" - copy_metric(name="automq_topic_rack_partition_count") where metric.name == "kafka_stream_topic_partition_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "topic", "rack"]) where metric.name == "automq_topic_rack_partition_count" - copy_metric(name="automq_group_count") where metric.name == "kafka_group_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_count" - copy_metric(name="automq_group_stable_count") where metric.name == "kafka_group_stable_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_stable_count" - copy_metric(name="automq_group_empty_count") where metric.name == "kafka_group_empty_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_empty_count" - copy_metric(name="automq_group_dead_count") where metric.name == "kafka_group_dead_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_dead_count" - copy_metric(name="automq_group_preparing_rebalance_count") where metric.name == "kafka_group_preparing_rebalance_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_preparing_rebalance_count" - copy_metric(name="automq_group_completing_rebalance_count") where metric.name == "kafka_group_completing_rebalance_count" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service"]) where metric.name == "automq_group_completing_rebalance_count" - copy_metric(name="automq_consumer_group_topic_commit_offset") where metric.name == "kafka_group_commit_offset" - copy_metric(name="automq_group_topic_offset_lag") where metric.name == "consumer_offset_lag_count" - copy_metric(name="automq_group_topic_estimated_time_lag_milliseconds") where metric.name == "consumer_estimated_time_lag_milliseconds" - aggregate_on_attributes("max", ["env_id", "instance_id", "service", "group", "topic"]) where metric.name == "automq_group_topic_estimated_time_lag_milliseconds" - context: datapoint statements: - set(datapoint.attributes["group"], datapoint.attributes["consumer_group"]) where metric.name == "automq_consumer_group_topic_commit_offset" and datapoint.attributes["consumer_group"] != nil - set(datapoint.attributes["consumer_group"], datapoint.attributes["group"]) where metric.name == "automq_group_topic_offset_lag" and datapoint.attributes["group"] != nil - context: metric statements: - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_consumer_group_topic_commit_offset" - aggregate_on_attributes("sum", ["env_id", "instance_id", "service", "consumer_group", "group", "topic"]) where metric.name == "automq_group_topic_offset_lag" filter/drop_raw_replaced_metrics: metrics: metric: - 'name == "kafka_log_size"' - 'name == "kafka_log_end_offset"' - 'name == "kafka_network_io_bytes_total"' - 'name == "kafka_message_count_total"' - 'name == "kafka_broker_network_io_bytes_total"' - 'name == "kafka_topic_request_count_total"' - 'name == "kafka_request_error_count_total"' - 'name == "kafka_stream_topic_partition_count"' - 'name == "kafka_group_count"' - 'name == "kafka_group_stable_count"' - 'name == "kafka_group_empty_count"' - 'name == "kafka_group_dead_count"' - 'name == "kafka_group_preparing_rebalance_count"' - 'name == "kafka_group_completing_rebalance_count"' - 'name == "kafka_group_commit_offset"' - 'name == "consumer_offset_lag_count"' - 'name == "consumer_estimated_time_lag_milliseconds"' service: extensions: [health_check] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, cumulativetodelta/automq_counters, transform/automq_rollups, filter/drop_raw_replaced_metrics, batch] exporters: [datadog] EOF ``` With `honor_labels: true`, if the scraped AutoMQ endpoint already exposes a Prometheus `job` label, that target-provided value is preserved and `job_name` is not used for that label. Standard AutoMQ metrics normally expose `job` as the cluster identifier, and Datadog maps that label to the `service` tag used by the dashboard `cluster_id` filter. If the endpoint does not expose `job`, `job_name` becomes the fallback value. The static `env_id` and `instance_id` labels are optional; set them only when you want those Datadog dashboard filters to have values for this Linux collection path. This Gateway profile performs rollups in the Collector to reduce Datadog custom metrics cardinality. It is not equivalent to a Prometheus recording rule and cannot express `max by (..., partition)` across scrape targets followed by `sum by (...)`. If you need strict deduplication when the same partition gauge is temporarily exposed by multiple AutoMQ nodes during partition migration, add a Prometheus recording-rule / federation layer before Datadog, or keep the partition dimension and handle it explicitly in Datadog queries. ### Step 4: Start the Collector Restart the Datadog Agent and verify the embedded Collector status: ```bash theme={null} sudo systemctl restart datadog-agent sudo datadog-agent status ``` For a foreground smoke test, run the Collector directly: ```bash theme={null} DD_API_KEY="" \ DD_SITE="" \ --config /etc/otelcol/config.yaml ``` For a systemd deployment, create a dedicated service user and environment file: ```bash theme={null} sudo useradd --system --home /var/lib/otelcol --shell /usr/sbin/nologin otelcol || true sudo install -d -o otelcol -g otelcol /var/lib/otelcol sudo tee /etc/default/otelcol > /dev/null << 'EOF' DD_API_KEY= DD_SITE= EOF ``` Limit access to this file because it contains your Datadog API key. Also avoid pasting real keys into shared shell history or logs. ```bash theme={null} sudo chmod 600 /etc/default/otelcol ``` ```bash theme={null} sudo tee /etc/systemd/system/otelcol.service > /dev/null << 'EOF' [Unit] Description=OpenTelemetry Collector for AutoMQ Datadog metrics After=network-online.target Requires=network-online.target [Service] User=otelcol Group=otelcol EnvironmentFile=/etc/default/otelcol ExecStart= --config /etc/otelcol/config.yaml Restart=always RestartSec=5 NoNewPrivileges=true ProtectSystem=full ProtectHome=true PrivateTmp=true StateDirectory=otelcol [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl enable otelcol sudo systemctl start otelcol ``` ### Step 5: Verify Open Datadog **Metrics > Explorer**, search for AutoMQ metrics such as `kafka_broker_active_count`, and filter by `service:`. In the imported dashboards, the filter is still named `cluster_id`, but it reads values from the Datadog `service` tag. If the expected service value does not appear, check one AutoMQ `/metrics` endpoint and confirm the actual `job` label because `honor_labels: true` preserves the target-provided `job` value. If your AutoMQ environment exposes native `env_id`, `instance_id`, or `broker_id` labels, you can use them as additional filters. New metrics may take 1–3 minutes to appear. AutoMQ metrics visible in Datadog Metrics Explorer after Linux Gateway OTEL Collector integration ## Appendix: Manually enable AutoMQ metrics export for Kubernetes Software AutoMQ Cloud BYOC clusters expose the Prometheus metrics endpoint by default. Use this appendix only when you manage an AutoMQ Software deployment on Kubernetes and need to enable the endpoint manually. For AutoMQ Software deployments on Kubernetes via Helm, set the metrics exporter to Prometheus mode in your Helm values: ```yaml theme={null} global: config: | s3.telemetry.metrics.exporter.uri=prometheus://?host=0.0.0.0&port=9090 ``` This exposes Prometheus-format metrics on port 9090 of each AutoMQ pod. The Datadog Agent Gateway profile in this guide uses selector-based Kubernetes discovery, so it does not require Prometheus scrape annotations to discover AutoMQ pods. If you also use an annotation-based Prometheus or Datadog collector outside of this profile, add the following annotations to the AutoMQ Helm values: ```yaml theme={null} controller: annotations: prometheus.io/automq-scrape: "true" prometheus.io/automq-port: "9090" prometheus.io/automq-path: "/metrics" broker: annotations: prometheus.io/automq-scrape: "true" prometheus.io/automq-port: "9090" prometheus.io/automq-path: "/metrics" ``` ## References * [Datadog Agent Helm Chart](https://github.com/DataDog/helm-charts/tree/main/charts/datadog) * [Datadog OTEL Collector on Kubernetes](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/kubernetes/?tab=helm) * [Datadog DDOT Collector on Linux](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/install/linux/) * [OpenTelemetry Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver)