This article outlines the configuration parameters involved in deploying AutoMQ, detailing their definitions, descriptions, applicable settings, and standards to aid developers in making necessary adjustments in a production setting.
AutoMQ functions as a compute-storage separated variant of Apache Kafka®, supporting all of Apache Kafka's parameters except for multi-replica storage. These parameters are not detailed in this guide but can be found in the official configuration documentation.
Common Configurations
Elasticstream.enable
List Item
| Description
|
|---|
Configuration Description
| Determines whether to launch AutoMQ, this parameter must be set to true.
|
Value Type
| boolean
|
Default Value
| false
|
Legal Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
S3.endpoint
List Item
| Description
|
|---|
Configuration Description
| Endpoint address for Object storage. For example: https://s3.{region}.amazonaws.com.
|
Value Type
| string
|
Default Value
| null
|
Valid Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
S3.region
List Item
| Description
|
|---|
| Configuration Description | The regional identifier for Object storage services, refer to the documentation provided by Cloud providers, such as us-east-1. |
| Value Type | string |
| Default Value | null |
| Valid Input Range | N/A |
| Importance Level | High, requires careful configuration |
S3.bucket
| List Item | Description |
|---|
| Configuration Description | Object storage bucket, used for storing messages. |
| Value Type | string |
Default value
| null
|
Valid input range
| N/A
|
Importance level
| High, requires careful configuration
|
S3.path.style
List item
| Description
|
|---|
Configuration description
| Whether to enable object storage path format. Must be set to true when using MinIO as the storage service.
|
Value type
| boolean
|
Default value
| false
|
Valid input range
| N/A
|
| Importance Level | High, requires careful configuration |
S3.wal.path
| List Item | Description |
|---|
| Configuration Description | Mount path for block storage device used to store local WAL, such as /dev/xxx or other paths. |
| Value Type | string |
| Default Value | null |
| Valid Input Range | N/A |
| Importance Level | High, requires careful configuration |
S3.wal.capacity
| List item | Description |
|---|
| Configuration description | The size of the local WAL (Write-Ahead Logging) for AutoMQ. This setting determines the maximum amount of data that can be written to the buffer before being uploaded to object storage. A larger capacity can tolerate more write jitter in the object storage. |
| Value type | long, in bytes |
| Default value | 2147483648 |
| Valid input range | [10485760, ...] |
| Importance level | Low, relatively flexible setting |
S3.wal.cache.size
| List item | Description |
|---|
| Configuration description | WAL (Write-Ahead Logging) cache is a FIFO (First In, First Out) queue, which contains data that has not yet been uploaded to object storage, as well as data that has been uploaded but not yet evicted from the cache. When the unuploaded cache data fills the entire capacity, the storage will backpressure subsequent requests until the data upload is completed. By default, it sets a reasonable value based on memory settings. |
| Value Type | long, unit: byte |
| Default Value | -1, automatically set to an appropriate parameter by the program |
| Valid Input Range | [1, ...] |
| Importance Level | Low, settings are relatively flexible |
S3.wal.iops
| List Item | Description |
|---|
| Configuration Description | Data is batched when written to the WAL, and is persistently written to the WAL disk at a certain frequency. This configuration item determines the frequency of data writes to the WAL. The higher the configuration value, the more frequent the writes per second, reducing latency. However, as the IOPS performance varies among different block devices, setting a value beyond the device's IOPS limit may lead to increased latency due to queuing. |
| Value Type | int, unit: iops |
| Default Value | 3000 |
| Valid Input Range | [1, ...] |
| Importance Level | High, recommended to be set to the actual available IOPS of the WAL disk |
S3.wal.upload.threshold
| List Item | Description |
|---|
| Configuration Description | Triggers the threshold for WAL to upload to object storage. The configuration value needs to be less than s3.wal.cache.size. The larger the configuration value, the higher the data aggregation, and the lower the cost of metadata storage. By default, it sets a reasonable value based on memory settings. |
| Value Type | long, in bytes |
| Default Value | -1, automatically set by the program to an appropriate parameter value |
| Valid Input Range | [1, ...] |
| Importance Level | Low, settings are relatively broad |
S3.block.cache.size
| Item | Description |
|---|
| Configuration Description | s3.block.cache.size is the size of the block cache, which is used to cache cold data read from object storage. It is recommended to set this configuration to greater than 4MB * the number of concurrent cold reads per partition to achieve better cold read performance. By default, it sets a reasonable value based on memory. |
| Value Type | long, in bytes |
| Default Value | -1, automatically set by the program to a suitable parameter value |
| Valid Input Range | [1, ...] |
| Importance Level | Low, setting is relatively broad |
S3.stream.object.compaction.interval.minutes
| Item | Description |
|---|
| Configuration Description | The interval period for compaction of Stream objects. The larger the interval, the lower the cost of API calls, but it increases the scale of metadata storage. |
Value Type
| int, in minutes
|
Default Value
| 30
|
Valid Input Range
| [1, ...]
|
Importance Level
| Low, setting is relatively flexible
|
S3.stream.object.compaction.max.size.bytes
List Item
| Description
|
|---|
Configuration Description
| Stream object compaction allows the maximum size of compounded objects. The larger this value, the higher the cost of API calls, but the smaller the scale of metadata storage.
|
Value Type
| long, in bytes
|
Default Value
| 1073741824
|
| Legal Input Range | [1, ...] |
| Importance Level | Low, relatively flexible settings |
S3.stream.set.object.compaction.interval.minutes
| List Item | Description |
|---|
| Configuration Description | Sets the interval for stream object compaction. The smaller this value, the smaller the scale of metadata storage and the sooner the data becomes compact. However, the resulting stream objects will undergo more frequent compactions. |
| Value Type | int, in minutes |
| Default Value | 20 |
| Legal Input Range | [1, ...] |
| Importance Level | Low, relatively flexible settings |
S3.stream.set.object.compaction.cache.size
| List Item | Description |
|---|
| Configuration Description | Memory size available during the stream object compaction process. The larger this value, the lower the cost of API calls. |
| Value Type | long, in bytes |
| Default Value | 209715200 |
| Valid Input Range | [1048576, ...] |
| Importance Level | Low, settings are relatively flexible |
S3.stream.set.object.compaction.stream.split.size
| List Item | Description |
|---|
| Configuration Description | During the Stream object compaction process, if the data volume in a single Stream exceeds this threshold, the data from this Stream will be directly split and written into individual Stream objects. The smaller this value, the sooner the data is split from the Stream set object, reducing the subsequent API call costs for Stream object compaction, but increasing the API call costs for splitting. |
Value Type
| long, in bytes
|
Default Value
| 8388608
|
Valid Input Range
| [1, ...]
|
Importance Level
| Low, relatively flexible settings
|
S3.stream.set.object.compaction.max.num
List Item
| Description
|
|---|
Configuration Description
| The maximum number of stream set objects that can be compressed at one time.
|
Value Type
| int
|
Default Value
| 500
|
| Legal Input Range | [1, ...] |
| Importance Level | Low, set relatively broadly |
S3.network.baseline.bandwidth
| List Item | Description |
|---|
| Configuration Description | Total available bandwidth for Object storage requests. This is used to prevent stream set object compaction and catch-up reads from occupying the normal read/write traffic. Production and consumption will also separately consume inbound and outbound traffic. For example, if this value is set to 100MB/s, and the normal read/write traffic is 80MB/s, then the available bandwidth for stream set object compaction is 20MB/s. |
| Value Type | long, in bytes/s |
| Default Value | 104857600 |
| Legal Input Range | [1, ...] |
| Importance Level | Low, set relatively broadly |
S3.stream.allocator.policy
| Item | Description |
|---|
| Configuration Description | S3Stream memory allocation strategy. Note that when configured to use DIRECT memory, it is necessary to modify the heap size (e.g., -Xmx) and direct memory size (e.g., -XX:MaxDirectMemorySize) in the JVM options. These can be set through the environment variable KAFKA_HEAP_OPTS. |
| Value Type | string |
| Default Value | POOLED_HEAP |
| Valid Input Range | POOLED_HEAP, POOLED_DIRECT |
| Importance Level | Low, relatively flexible settings |
S3.telemetry.metrics.enable
| Item | Description |
|---|
| Configuration Description | Whether to enable the OTel metrics exporter. |
| Value Type | boolean |
| Default Value | true |
| Allowed Input Range | true, false |
| Importance Level | Low, with relatively flexible settings |
S3.telemetry.metrics.level
| List Item | Description |
|---|
| Configuration Description | Sets the level for recording Metrics. The "INFO" level includes most metrics that users should care about, such as throughput and latency for common stream operations. The "DEBUG" level includes detailed metrics helpful for diagnostics, such as latencies at different stages of writing to underlying block storage. |
| Value Type | string |
| Default Value | INFO |
Legal Input Range
| INFO, DEBUG
|
Importance Level
| Low, relatively broad settings
|
S3.telemetry.metrics.exporter.type
List Item
| Description
|
|---|
Configuration Description
| Metrics exporter type list:
- The "otlp" type will use the OTLP protocol to export metrics to backend services.
- The "prometheus" type will start an internal HTTP server, allowing Prometheus backends to fetch metrics from it.
|
Value Type
| list
|
Default Value
| null
|
Legal Input Range
| otlp, prometheus
|
Importance Level
| Low, relatively broad settings
|
S3.telemetry.exporter.report.interval.ms
| List Item | Description |
|---|
| Configuration Description | Sets the interval for exporting Metrics. |
| Value Type | int, in milliseconds |
| Default Value | 30000 |
| Valid Input Range | N/A |
| Importance Level | Low, setting is relatively flexible |
S3.telemetry.exporter.otlp.protocol
| List Item | Description |
|---|
| Configuration Description | Transport protocol used by the OTLP exporter. |
| Value Type | string |
| Default | grpc |
| Valid Inputs | grpc, http |
| Importance | Low, settings are relatively flexible |
S3.telemetry.exporter.otlp.endpoint
| List Item | Description |
|---|
| Configuration Description | The exposed address of the backend service when using the OTLP Exporter. |
| Value Type | string |
| Default | null |
Valid Input Range
| N/A
|
Importance Level
| Low, settings are relatively broad
|
S3.telemetry.exporter.otlp.compression.enable
List Item
| Description
|
|---|
Configuration Description
| Determines whether OTLP exporter enables data compression; if enabled, OTLP uses the gzip compression algorithm to compress Metrics.
|
Value Type
| boolean
|
Default Value
| false
|
Valid Input Range
| true, false
|
Importance Level
| Low, settings are relatively broad
|
S3.metrics.exporter.prom.host
| List Item | Description |
|---|
| Configuration Description | Address of the built-in Prometheus HTTP server for exposing OTel Metrics. |
| Value Type | string |
| Default Value | null |
| Valid Input Range | N/A |
| Importance Level | Low, settings are relatively flexible |
S3.metrics.exporter.prom.port
| List Item | Description |
|---|
| Configuration Description | Port of the built-in Prometheus HTTP server for exposing OTel Metrics. |
Value Type
| int
|
Default Value
| 0
|
Valid Input Range
| N/A
|
Importance Level
| Low, setting is relatively lenient
|
Self-balancing Configuration
Metric.reporters
List Item
| Description
|
|---|
Configuration Description
| List of class names for metrics reporters. Implementing the org.apache.kafka.common.metrics.MetricsReporter interface allows for the dynamic loading of new Metrics. JmxReporter is always included to register JMX statistics. To enable AutoBalancing, metric.reporters must include kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter.
|
Value Type
| list
|
Default Value
| ""
|
Valid Input Range
| N/A
|
Importance Level
| Low, settings are relatively flexible
|
Autobalancer.reporter.metrics.reporting.interval.ms
List Item
| Description
|
|---|
Configuration Description
| The interval for reporting data on the Metrics Reporter.
|
Value Type
| long, in milliseconds
|
Default Value
| 10000
|
Valid Input Range
| [1000, ...]
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.enable
List Item
| Description
|
|---|
Configuration Description
| Enables self-balancing.
|
Value Type
| boolean
|
Default Value
| false
|
Valid Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.anomaly.detect.interval.ms
List Item
| Description
|
|---|
| Configuration Description | The minimum interval at which the Controller checks for the need to self-balance data. The actual timing of the next self-balancing also depends on the number of partitions that have been reassigned. Lowering this minimum check interval can increase the sensitivity of data self-balancing. This value should be greater than the reporting interval of broker metrics to prevent the controller from missing the results of recent reassignments. |
| Value Type | long, in milliseconds |
| Default Value | 60000 |
| Valid Input Range | [1, ...] |
| Importance Level | High, requires careful configuration |
Autobalancer.controller.metrics.delay.ms
| List Item | Description |
|---|
| Configuration Description | When the Controller performs load balancing, if the latest Metrics delay of a broker exceeds the configured value, the broker will be excluded due to its lag in sync status. This configuration should not be less than the interval of broker metrics reporting. |
| Value Type | long, in milliseconds |
| Default Value | 60000 |
| Valid Input Range | [1, ...] |
| Importance Level | High, requires cautious configuration |
Autobalancer.controller.goals
| List Item | Description |
|---|
| Configuration Description | Target settings for self-balancing optimization. |
| Value Type | list |
| Default Value | kafka.autobalancer.goals.NetworkInUsageDistributionGoal,kafka.autobalancer.goals.NetworkOutUsageDistributionGoal |
| Valid Input Range | N/A |
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.network.in.usage.distribution.detect.threshold
List Item
| Description
|
|---|
Configuration Description
| Detection threshold for NetworkInUsageDistributionGoal. If a broker's write bandwidth falls below this configuration value, the broker will not be actively reassigned for self-balancing.
|
Value Type
| long, unit is byte/s
|
Default Value
| 1048576
|
Valid Input Range
| [1, ...]
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.network.out.usage.distribution.detect.threshold
List Item
| Description
|
|---|
| Configuration Description | Detection threshold for NetworkOutUsageDistributionGoal. If a broker's read bandwidth falls below this configured value, the broker will not be actively reassigned for self-balancing. |
| Value Type | long, unit is byte/s |
| Default Value | 1048576 |
| Valid Input Range | [1, ...] |
| Importance Level | High, requires careful configuration |
Autobalancer.controller.network.in.distribution.detect.avg.deviation
| List Item | Description |
|---|
| Configuration Description | This configuration item defines the acceptable deviation range for average write bandwidth. The default value is 0.2, which means the expected network traffic range will be [0.8 loadAvg, 1.2 loadAvg]. |
| Value Type | double |
Default Value
| 0.2
|
Valid Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.network.out.distribution.detect.avg.deviation
List Item
| Description
|
|---|
Configuration Description
| This configuration parameter defines the acceptable deviation range for the average read bandwidth. The default value is 0.2, which means the expected network traffic range will be [0.8 * loadAvg, 1.2 * loadAvg].
|
Value Type
| double
|
Default Value
| 0.2
|
Valid Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.exclude.topics
List Item
| Description
|
|---|
Configuration Description
| List of Topics to be excluded from self-balancing.
|
Value Type
| list
|
Default Value
| ""
|
Valid Input Range
| N/A
|
Importance Level
| High, requires careful configuration
|
Autobalancer.controller.exclude.broker.ids
List Item
| Description
|
|---|
| Configuration Description | List of Broker Ids that should be excluded from self-balancing. |
| Value Type | list |
| Default Value | "" |
| Valid Input Range | N/A |
| Importance Level | High, requires careful configuration |