> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Broker and Controller Configuration

> Optimize AutoMQ deployment with advanced broker and controller configurations, leveraging cloud-native scalability and Kafka compatibility to reduce costs.

This document aims to explain the configuration parameters involved in the deployment of AutoMQ. It includes configuration definitions, descriptions, setting ranges, and specifications to aid developers in making necessary custom adjustments in production environments.

AutoMQ implements a storage-compute separation based on object storage and is fully compatible with Apache Kafka. Therefore, Kafka's functional configurations (e.g., ACL, network, etc.) can be referred to from the [official configuration documentation](https://kafka.apache.org/documentation/#brokerconfigs). This document only lists the configuration parameters related to the new storage module added by AutoMQ.

## Public Configuration

### elasticstream.enable

| Item<br />                      | Description<br />                                                   |
| ------------------------------- | ------------------------------------------------------------------- |
| Configuration Description<br /> | Whether to enable AutoMQ, this parameter must be set to true.<br /> |
| Value Type<br />                | boolean<br />                                                       |
| Default Value<br />             | false<br />                                                         |
| Valid Input Range<br />         | N/A<br />                                                           |
| Importance Level<br />          | High, requires careful configuration<br />                          |

### automq.zonerouter.channels

By enabling cross-AZ (Availability Zone) request routing, you can reduce cross-AZ data transmission and decrease traffic costs. For more details, refer to [Overview▸](/automq/eliminate-inter-zone-traffics/overview).

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                 |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | Configuration for Inter-Zone channel. By configuring Inter-Zone routing components, the cost caused by cross-zone traffic can be significantly reduced. Currently, only object storage is supported. The format is: `0@s3://$bucket?region=$region[&batchInterval=250][&maxBytesInBatch=8388608]` |
| Value Type<br />          | string<br />                                                                                                                                                                                                                                                                                      |
| Default Value<br />       | null<br />                                                                                                                                                                                                                                                                                        |
| Valid Input Range<br />   | N/A<br />                                                                                                                                                                                                                                                                                         |
| Importance Level<br />    | High, requires careful configuration<br />                                                                                                                                                                                                                                                        |

## S3Stream Related Configuration

### s3.data.buckets

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The URI for data plane object storage. The format is: `0@s3://$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey][&checksumAlgorithm=$checksumAlgorithm]`.<br />For configurations from different vendors, see: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                                                                       |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                                                                         |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                                                                          |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                         |

### s3.ops.buckets

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The URI for control plane object storage. The format is: `1@s3://$bucket?region=$region[&endpoint=$endpoint][&pathStyle=$enablePathStyle][&authType=$authType][&accessKey=$accessKey][&secretKey=$secretKey][&checksumAlgorithm=$checksumAlgorithm]`.<br />For configurations from different vendors, see: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration).<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                                                                          |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                                                                            |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                                                                             |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                            |

### s3.wal.path

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                    |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The mount path for block storage devices used for storing local WAL, with the format: <br />`0@s3://$bucket?region=$region[&batchInterval=250][&maxBytesInBatch=8388608]`<br />For details on different vendors, refer to: [Object Storage Configuration▸](/automq/configuration/object-storage-configuration)<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                         |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                           |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                            |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                           |

### s3.wal.cache.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | WAL (Write-Ahead Logging) cache is a FIFO (first-in-first-out) queue that contains data which has not been uploaded to object storage, as well as data that has been uploaded but not yet evicted from the cache. When the cache data that has not been uploaded fills up the entire capacity, storage will exert backpressure on subsequent requests until the data upload is completed. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                                                                                                                                                                |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                                                         |

### s3.wal.upload.threshold

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The threshold that triggers WAL uploads to object storage. The configuration value needs to be less than `s3.wal.cache.size`. The larger the configuration value, the higher the data aggregation degree, leading to lower metadata storage costs. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                    |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                         |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                  |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                  |

### s3.block.cache.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | `s3.block.cache.size` is the size of the block cache. The block cache is used to cache cold data read from object storage. It is recommended to set this configuration item to more than 4MB \* the number of concurrent cold reads per partition to achieve better cold read performance. By default, it sets a reasonable value based on memory.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                            |
| Default Value<br />             | -1, automatically set by the program to a suitable parameter value<br />                                                                                                                                                                                                                                                                                 |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                          |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                          |

### s3.stream.object.compaction.interval.minutes

| Item<br />                      | Description<br />                                                                                                                                                    |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The interval period for compaction in Stream objects. The larger the interval, the lower the cost of API calls, but it increases the size of metadata storage.<br /> |
| Value Type<br />                | int, measured in minutes<br />                                                                                                                                       |
| Default Value<br />             | 30<br />                                                                                                                                                             |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                      |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                      |

### s3.stream.object.compaction.max.size.bytes

| Item<br />                      | Description<br />                                                                                                                                                                        |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Stream object compaction allows for the maximum size of synthetic objects. The larger this value, the higher the cost of API calls, but the smaller the scale of metadata storage.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                            |
| Default Value<br />             | 10737418240<br />                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                          |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                          |

### s3.stream.set.object.compaction.interval.minutes

| Item<br />                      | Description<br />                                                                                                                                                                                                                              |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Sets the interval for stream object compaction. The smaller this value, the smaller the scale of the metadata storage, and the data becomes compacted sooner. However, the final stream objects will undergo compaction more frequently.<br /> |
| Value Type<br />                | int, measured in minutes<br />                                                                                                                                                                                                                 |
| Default Value<br />             | 5<br />                                                                                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                |

### s3.stream.set.object.compaction.cache.size

| Item<br />                | Description<br />                                                                                                                 |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | The size of memory available during the stream object compaction process. The larger this value, the lower the cost of API calls. |
| Value Type                | long, in bytes                                                                                                                    |
| Default Value             | 209715200                                                                                                                         |
| Legal Input Range<br />   | \[1048576, ...]<br />                                                                                                             |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                   |

### s3.stream.set.object.compaction.stream.split.size

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | During the Stream object compaction process, if the data volume within a single Stream exceeds this threshold, the Stream's data will be directly split and written into individual Stream objects. The smaller this value, the earlier the data is split from the Stream set object, resulting in lower subsequent API call costs for Stream object compaction, but leading to higher API call costs for the split.<br /> |
| Value Type<br />                | long, measured in bytes<br />                                                                                                                                                                                                                                                                                                                                                                                              |
| Default Value<br />             | 8388608<br />                                                                                                                                                                                                                                                                                                                                                                                                              |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                            |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                            |

### s3.network.baseline.bandwidth

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | The total available bandwidth for object storage requests. This is used to prevent stream set object compaction and catch-up reads from occupying normal read-write traffic. Production and consumption will also individually consume inbound and outbound traffic.<br /> For instance, if this value is set to 100MB/s and normal read-write traffic is 80MB/s, then the available traffic for stream set object compaction would be 20MB/s.<br /> |
| Value Type<br />                | long, measured in byte/s<br />                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Default Value<br />             | 1073741824<br />                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                                                                                                                      |

### s3.stream.allocator.policy

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                  |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | S3Stream memory allocator policy. Note that when configured to use DIRECT memory, the heap size in the virtual machine options (e.g., -Xmx) and the direct memory size (e.g., -XX:MaxDirectMemorySize) need to be adjusted. You can set them via the environment variable KAFKA\_HEAP\_OPTS.<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                       |
| Default<br />                   | POOLED\_HEAP<br />                                                                                                                                                                                                                                                                                 |
| Valid Values<br />              | POOLED\_HEAP, POOLED\_DIRECT<br />                                                                                                                                                                                                                                                                 |
| Importance Level<br />          | Low, set relatively broad<br />                                                                                                                                                                                                                                                                    |

### s3.telemetry.metrics.level

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                                            |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | Sets the level of Metrics logging. The "INFO" level includes metrics that most users should be concerned with, such as throughput and latency of common stream operations. The "DEBUG" level includes detailed metrics useful for diagnostics, such as latencies at various stages when writing to underlying block devices. |
| Value Type<br />          | string<br />                                                                                                                                                                                                                                                                                                                 |
| Default Value             | INFO                                                                                                                                                                                                                                                                                                                         |
| Valid Input Range         | INFO, DEBUG                                                                                                                                                                                                                                                                                                                  |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                              |

### s3.telemetry.exporter.report.interval.ms

| Item<br />                      | Description<br />                           |
| ------------------------------- | ------------------------------------------- |
| Configuration Description<br /> | Sets the interval for Metrics export.<br /> |
| Value Type<br />                | int, in milliseconds<br />                  |
| Default Value<br />             | 30000<br />                                 |
| Valid Input Range<br />         | N/A<br />                                   |
| Importance Level<br />          | Low, set relatively broad<br />             |

### s3.telemetry.metrics.base.labels

| Item<br />                      | Description<br />                                                                                                                                                                                                                         |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description<br /> | Multi-dimensional labels for metrics, used to attach static multi-dimensional labels to all monitoring metrics, enabling categorization, aggregation, and fine-grained analysis of metrics. The format is: key1=value1,key2=value2.<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                              |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                 |
| Importance level<br />          | Medium, requires careful configuration<br />                                                                                                                                                                                              |

### s3.telemetry.metrics.exporter.uri

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                           |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration description<br /> | The export URI for Metrics. The format is: `$type://?$param1=$value1&$param2=$value2`. Currently the supported types for 'type' are prometheus and otlp.<br />The format for prometheus: `prometheus://?host=$hostname&port=$port`<br />The format for otlp: `otlp://?endpoint=$endpoint&protocol=$protocol&compression=$compression`<br /> |
| Value Type<br />                | string<br />                                                                                                                                                                                                                                                                                                                                |
| Default Value<br />             | null<br />                                                                                                                                                                                                                                                                                                                                  |
| Valid Input Range<br />         | N/A<br />                                                                                                                                                                                                                                                                                                                                   |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                  |

## Persistent Data Rebalancing Configuration

### metric.reporters

| Item<br />                | Description<br />                                                                                                                                                                                                                                                                                                                                       |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration Description | A list of classes for metrics reporters. By implementing the `org.apache.kafka.common.metrics.MetricsReporter` interface, you can dynamically load new metrics. `JmxReporter` is always included to register JMX statistics. To enable AutoBalancing, `metric.reporters` must include `kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter`. |
| Value Type<br />          | list<br />                                                                                                                                                                                                                                                                                                                                              |
| Default Value<br />       | ""<br />                                                                                                                                                                                                                                                                                                                                                |
| Valid Input Range<br />   | N/A<br />                                                                                                                                                                                                                                                                                                                                               |
| Importance Level<br />    | Low, set relatively broad<br />                                                                                                                                                                                                                                                                                                                         |

### autobalancer.reporter.metrics.reporting.interval.ms

| Item<br />                      | Description<br />                                      |
| ------------------------------- | ------------------------------------------------------ |
| Configuration Description<br /> | Interval for reporting data by Metrics Reporter.<br /> |
| Value Type<br />                | long, unit is milliseconds<br />                       |
| Default Value<br />             | 10000<br />                                            |
| Legal Input Range<br />         | \[1000, ...]<br />                                     |
| Importance Level<br />          | High, requires careful configuration<br />             |

### autobalancer.controller.enable

| Item<br />                      | Description<br />                              |
| ------------------------------- | ---------------------------------------------- |
| Configuration Description<br /> | Whether to enable automatic rebalancing.<br /> |
| Value Type<br />                | boolean<br />                                  |
| Default Value<br />             | true<br />                                     |
| Valid Input Range<br />         | N/A<br />                                      |
| Importance Level<br />          | High, requires careful configuration<br />     |

### autobalancer.controller.anomaly.detect.interval.ms

| Item<br />                      | Description<br />                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration description<br /> | The minimum interval for the Controller to check if a data rebalance is needed. The actual time for the next rebalance also depends on the number of partitions that have been reassigned. Reducing the minimum check interval can increase the sensitivity of data rebalancing. This value should be greater than the broker metrics reporting interval to prevent the controller from missing recent reassignment results.<br /> |
| Value Type<br />                | long, unit is milliseconds<br />                                                                                                                                                                                                                                                                                                                                                                                                   |
| Default Value<br />             | 60000<br />                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Legal Input Range<br />         | \[1, ...]<br />                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Importance Level<br />          | High, requires careful configuration<br />                                                                                                                                                                                                                                                                                                                                                                                         |

### autobalancer.controller.exclude.topics

| Item<br />                      | Description<br />                                          |
| ------------------------------- | ---------------------------------------------------------- |
| Configuration Description<br /> | List of Topics to be excluded from data rebalancing.<br /> |
| Value Type<br />                | list<br />                                                 |
| Default Value<br />             | ""<br />                                                   |
| Valid Input Range<br />         | N/A<br />                                                  |
| Importance Level<br />          | High, requires careful configuration<br />                 |

### autobalancer.controller.exclude.broker.ids

| Item<br />                      | Description<br />                                                       |
| ------------------------------- | ----------------------------------------------------------------------- |
| Configuration Description<br /> | List of Broker Ids that should be excluded from data rebalancing.<br /> |
| Value Type<br />                | list<br />                                                              |
| Default Value<br />             | ""<br />                                                                |
| Valid Input Range<br />         | N/A<br />                                                               |
| Importance Level<br />          | High, requires careful configuration<br />                              |

## Table Topic

<Tip>
  Table Topic is a core feature designed by AutoMQ for modern data lake architectures. It integrates streaming data with data lake storage through architectural innovation, addressing traditional challenges such as the separation of streaming and batch processing, complex ETL processes, and high costs. For more details, see [Overview▸](/automq/table-topic/overview).
</Tip>

For the current Table Topic configuration model, including topic-level settings, record conversion, transformations, error handling, Schema Registry client options, Catalog authentication, and Catalog-specific options, see [Table Topic Configuration▸](/automq/table-topic/table-topic-configuration).