> ## Documentation Index
> Fetch the complete documentation index at: https://docs.automq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Splunk Sink Connector

> Configure and operate the Splunk Sink Connector in AutoMQ Connect, including prerequisites, configuration, monitoring, and troubleshooting.

## Overview

The Splunk Sink Connector consumes records from Kafka topics and writes them to Splunk through the Splunk HTTP Event Collector (HEC). It sits between Kafka and Splunk and can continuously deliver application logs, audit events, metric data, and business events for search, analysis, and alerting.

By default, the connector uses the HEC `/event` endpoint, wraps record values as events, and can assign different indexes, sources, and sourcetypes to different topics. It can also use the `/raw` endpoint for raw content or override Splunk metadata dynamically through Kafka headers. Tasks send records in batches. With HEC acknowledgement disabled by default, a successful HTTP response marks a batch as complete; when acknowledgement is enabled, the connector waits for the corresponding Splunk indexer acknowledgement before advancing Kafka offsets. Recovery can still replay records whose offsets have not been committed, so downstream processing should tolerate duplicate events.

## Prerequisites

* HEC is enabled in Splunk, a usable HEC URI and Token are available, and the Token has permission to write to the default or explicitly configured index.
* When HTTPS certificate validation is enabled, every Worker that may run a Task can read a trust store file containing the required CA chain.
* When HEC acknowledgement is enabled, indexer acknowledgement is enabled for the corresponding Token in Splunk.

## License

Uses Apache License 2.0.

## Quick Start

Prepare a Connect Cluster, Kafka input Topic, and Splunk HEC in advance. Confirm network connectivity, Token permissions, and Worker access to the trust store. For creation and management operations, see [Manage Connectors](../manage-connectors). The following configuration uses the HTTPS `/event` endpoint and relies on the HEC Token's defaults for the index, source, and sourcetype.

```properties theme={null}
connector.class=com.splunk.kafka.connect.SplunkSinkConnector
topics=<input-topic>
splunk.hec.uri=https://<splunk-host>:8088
splunk.hec.token=<hec-token>
splunk.hec.ssl.trust.store.path=<trust-store-path>
splunk.hec.ssl.trust.store.password=<trust-store-password>
```

Replace the input Topic, HEC address, Token, and Worker-local trust store information. The default trust store type is `JKS`; when using another JVM-supported type, add `splunk.hec.ssl.trust.store.type`. Supply the Token and trust store password through a controlled configuration management mechanism. This example inherits the Worker's Converters. Explicitly configure `key.converter` and `value.converter` when the actual record encoding does not match them.

## Configuration

### HEC Connection and Authentication

#### `splunk.hec.token`

The Token used to access Splunk HEC.

* **Type**: `password`
* **Default**: None, required
* **Importance**: High
* **Valid Values / Notes**: Use an enabled Token that is authorized to write to the target index. Manage it as a sensitive configuration value and do not include it in logs or shared examples.

#### `splunk.hec.uri`

The base URI for Splunk HEC.

* **Type**: `string`
* **Default**: None, required
* **Importance**: High
* **Valid Values / Notes**: Multiple HTTP or HTTPS URIs may be separated by commas. HTTPS is enforced by default. Avoid whitespace before or after each URI entry.

#### `splunk.validation.disable`

Whether to skip live requests to Splunk during Connector configuration validation.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: `false` sends a validation event to the configured index through the first HEC URI. `true` skips only this remote validation; it does not skip required-field, URI, Kerberos, or Task configuration checks.

#### `kerberos.user.principal`

The Kerberos Principal used for HEC HTTP requests.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: When Kerberos is enabled, set this together with `kerberos.keytab.path`. Kerberos does not replace the HEC Token.

#### `kerberos.keytab.path`

The Worker-local path to the Kerberos Keytab.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: When Kerberos is enabled, set this together with `kerberos.user.principal`, and ensure that every Worker hosting a Task can read the file.

### TLS Security

#### `splunk.hec.ssl.enforced`

Whether HEC URIs must use HTTPS.

* **Type**: `boolean`
* **Default**: `true`
* **Importance**: High
* **Valid Values / Notes**: `true` rejects HTTP. When set to `false`, only HTTP or HTTPS is still accepted. HTTP transmits events and the Token in plaintext.

#### `splunk.hec.ssl.validate.certs`

Whether HTTPS connections validate certificates and hostnames.

* **Type**: `boolean`
* **Default**: `true`
* **Importance**: Medium
* **Valid Values / Notes**: When kept at `true`, a nonempty `splunk.hec.ssl.trust.store.path` must be configured. Disable validation only for controlled diagnostics or when the risk is explicitly accepted.

#### `splunk.hec.ssl.trust.store.path`

The path to the trust store file used to validate Splunk certificates.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: High
* **Valid Values / Notes**: Required when HTTPS and certificate validation are enabled. The path must be visible and readable to every Task process.

#### `splunk.hec.ssl.trust.store.type`

The trust store type.

* **Type**: `string`
* **Default**: `JKS`
* **Importance**: Low
* **Valid Values / Notes**: Common values include `JKS` and `PKCS12`. Actual support depends on the JVM security providers.

#### `splunk.hec.ssl.trust.store.password`

The password required to load the trust store.

* **Type**: `password`
* **Default**: Empty string
* **Importance**: High
* **Valid Values / Notes**: Enter the password configured for the trust store and manage it as a sensitive configuration value. Empty string is the ConfigDef default.

### Endpoint and Event Format

#### `splunk.hec.raw`

Selects the HEC `/raw` or `/event` endpoint.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: `false` uses the JSON `/event` endpoint. `true` uses `/raw`, where metadata applies to the entire request batch and event-field enrichment and custom Header fields do not apply.

#### `splunk.hec.raw.line.breaker`

Appends a delimiter to each record in `/raw` mode.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Used only when `splunk.hec.raw=true`. It should match the line-breaking and event-segmentation rules of the Splunk sourcetype.

#### `splunk.hec.json.event.formatted`

Attempts to parse the record value as a complete HEC JSON Event object.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Low
* **Valid Values / Notes**: Used only with `/event`. If parsing fails, the record value falls back to ordinary event content rather than retaining preformatted-object semantics.

### Splunk Metadata Routing

#### `splunk.indexes`

Specifies the Splunk index for explicitly subscribed Topics.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: An empty value uses the HEC Token's default index. Supply one value shared by all Topics, or the same number of comma-separated values in `topics` order. `topics.regex` does not support this positional mapping.

#### `splunk.sources`

Specifies the Splunk source for explicitly subscribed Topics.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Supply one shared value, or the same number of comma-separated values in `topics` order. An empty value uses the HEC default, and `topics.regex` does not support positional mapping.

#### `splunk.sourcetypes`

Specifies the Splunk sourcetype for explicitly subscribed Topics.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Supply one shared value, or the same number of comma-separated values in `topics` order. An empty value uses the HEC default, and `topics.regex` does not support positional mapping.

### Kafka Header Mapping

#### `splunk.header.support`

Enables mapping from Kafka Headers to Splunk metadata and fields.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: When enabled, configured Headers can override the index, host, source, and sourcetype. `/raw` groups records by these four metadata categories.

#### `splunk.header.custom`

Specifies custom Kafka Header names to write as `/event` fields.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Separate Header names with commas and also enable `splunk.header.support`. These Headers are not used as indexed fields with `/raw`.

#### `splunk.header.index`

Specifies the Kafka Header name that overrides the Splunk index.

* **Type**: `string`
* **Default**: `splunk.header.index`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when `splunk.header.support=true`. If multiple Headers have the same name, the last value is used.

#### `splunk.header.source`

Specifies the Kafka Header name that overrides the Splunk source.

* **Type**: `string`
* **Default**: `splunk.header.source`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when `splunk.header.support=true`. The Header value is converted to a string.

#### `splunk.header.sourcetype`

Specifies the Kafka Header name that overrides the Splunk sourcetype.

* **Type**: `string`
* **Default**: `splunk.header.sourcetype`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when `splunk.header.support=true`. The Header value is converted to a string.

#### `splunk.header.host`

Specifies the Kafka Header name that overrides the Splunk host.

* **Type**: `string`
* **Default**: `splunk.header.host`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when `splunk.header.support=true`. The Header value is converted to a string.

### Event Fields and Time

#### `splunk.hec.json.event.enrichment`

Adds fixed fields to `/event` records.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Low
* **Valid Values / Notes**: Use comma-separated `key=value` entries. Each entry must produce exactly one nonempty key and value; for duplicate keys, the last value wins. Does not apply to `/raw`.

#### `splunk.hec.track.data`

Adds Kafka position and record-tracking fields to `/event`.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Low
* **Valid Values / Notes**: May include the Topic, partition, Offset, timestamp, record key, and Connect host information. Assess exposure of sensitive data before enabling it.

#### `splunk.hec.use.record.timestamp`

Uses the Kafka record timestamp as the event time for ordinary `/event` events.

* **Type**: `boolean`
* **Default**: `true`
* **Importance**: Medium
* **Valid Values / Notes**: Successful regular-expression time extraction can override this time. It is not used for `/raw` or successfully parsed preformatted events.

#### `splunk.hec.auto.extract.timestamp`

Whether to append the `auto_extract_timestamp` parameter to HEC requests.

* **Type**: `boolean`
* **Default**: `null`
* **Importance**: Low
* **Valid Values / Notes**: `null` means the parameter is not sent. Explicitly setting `true` or `false` passes that value to HEC. This is independent of the Connector's regular-expression time extraction.

#### `enable.timestamp.extraction`

Enables regular-expression event-time extraction in the Connector.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: Used only for ordinary `/event` events. When enabled, `timestamp.regex` must contain a named group called `time`, and an applicable time format must be configured.

#### `timestamp.regex`

The Java regular expression used to extract a time value from event text.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: When extraction is enabled, the expression must contain a `(?<time>...)` named group. The expression runs against the string representation of the event.

#### `timestamp.regex.timeout.ms`

The timeout for running the time regular expression on one record.

* **Type**: `int`
* **Default**: `500`
* **Importance**: Medium
* **Valid Values / Notes**: In milliseconds, with a minimum of `1`. On timeout, the event retains any time value it already had.

#### `timestamp.format`

The time format used to parse the value extracted by the regular expression.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Use `epoch` when extracting Unix Epoch time. Other values are parsed as Java `SimpleDateFormat` patterns. Set this explicitly when extraction is enabled.

#### `timestamp.timezone`

The time zone used to parse non-Epoch time.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: Use a valid Java time zone ID. It applies only with regular-expression extraction and a non-Epoch format.

### HEC Acknowledgement and Retries

#### `splunk.hec.ack.enabled`

Enables polling for Splunk HEC indexer acknowledgement.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: Enable indexer acknowledgement for the HEC Token before enabling this setting. When disabled, a successful HTTP response marks a batch as complete but does not prove that its events are durable and searchable. The default is the ConfigDef value `false`.

#### `splunk.hec.event.timeout`

The timeout for waiting for HEC acknowledgement.

* **Type**: `int`
* **Default**: `300`
* **Importance**: Medium
* **Valid Values / Notes**: In seconds and used only when `splunk.hec.ack.enabled=true`. Set a positive value appropriate for HEC processing latency.

#### `splunk.hec.ack.poll.interval`

The interval for polling HEC acknowledgement status.

* **Type**: `int`
* **Default**: `10`
* **Importance**: Medium
* **Valid Values / Notes**: In seconds and used only when HEC acknowledgement is enabled. Thread scheduling requires a usable positive value.

#### `splunk.hec.ack.poll.threads`

The number of threads used to poll HEC acknowledgement status.

* **Type**: `int`
* **Default**: `2`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when HEC acknowledgement is enabled and should be a positive integer. The default is the ConfigDef value `2`.

#### `splunk.hec.max.outstanding.events`

Limits the number of in-flight events that have not completed.

* **Type**: `int`
* **Default**: `1000000`
* **Importance**: Medium
* **Valid Values / Notes**: Reaching the threshold triggers backpressure and a retriable exception. Set a positive value based on individual record size, batching, acknowledgement latency, and Worker memory.

#### `splunk.hec.max.retries`

Controls the Connector's internal retry count for failed batches.

* **Type**: `int`
* **Default**: `5`
* **Importance**: Medium
* **Valid Values / Notes**: A positive value limits the retry count for a failed batch. In this implementation, `0` or a negative value does not enter the count limit and behaves as unlimited retries. This differs from the Kafka Connect framework's `errors.retry.timeout`.

#### `splunk.hec.backoff.threshhold.seconds`

Sets the backpressure duration after an HEC channel failure.

* **Type**: `int`
* **Default**: `60`
* **Importance**: Medium
* **Valid Values / Notes**: In seconds. The spelling `threshhold` in the configuration name is the existing public key and must be used verbatim.

### Batching, Concurrency, and Transport

#### `splunk.hec.max.batch.size`

Sets the maximum number of records in one HEC batch.

* **Type**: `int`
* **Default**: `500`
* **Importance**: Medium
* **Valid Values / Notes**: Measured in records rather than bytes. The ConfigDef default is `500`. Use a positive value and adjust it based on individual event size and HEC request limits.

#### `splunk.flush.window`

Sets the time window for batch sending.

* **Type**: `int`
* **Default**: `30`
* **Importance**: Low
* **Valid Values / Notes**: In seconds. Sending is triggered when the batch record count is reached or when a subsequent record arrives after the window has elapsed. Only a positive value replaces the internal window.

#### `splunk.hec.threads`

Sets the number of threads in each Task that send HEC batches.

* **Type**: `int`
* **Default**: `1`
* **Importance**: Low
* **Valid Values / Notes**: A value greater than `1` enables concurrent sending and a bounded queue. This differs from `tasks.max`; increasing it may change batch completion order.

#### `splunk.hec.concurrent.queue.capacity`

Sets the concurrent HEC sending queue capacity in batches.

* **Type**: `int`
* **Default**: `100`
* **Importance**: Low
* **Valid Values / Notes**: Must be greater than `0`. It primarily holds pending batches when `splunk.hec.threads>1`.

#### `splunk.hec.total.channels`

Sets the total number of HEC channels used by each Task.

* **Type**: `int`
* **Default**: `2`
* **Importance**: High
* **Valid Values / Notes**: Use a positive value. Together with multiple URIs and HEC threads, it determines channel allocation; it is not the number of Tasks.

#### `splunk.hec.max.http.connection.per.channel`

Sets the upper limit for the HTTP connection pool per target.

* **Type**: `int`
* **Default**: `2`
* **Importance**: Medium
* **Valid Values / Notes**: Use a positive value. The total connection pool is also affected by the number of HEC URIs.

#### `splunk.hec.http.keepalive`

Whether HEC HTTP connections use Keep-Alive.

* **Type**: `boolean`
* **Default**: `true`
* **Importance**: Medium
* **Valid Values / Notes**: `true` or `false`. Normally keep it enabled to reuse connections.

#### `splunk.hec.socket.timeout`

Sets the Socket timeout declared by the HEC client.

* **Type**: `int`
* **Default**: `60`
* **Importance**: Low
* **Valid Values / Notes**: In seconds. Use a positive value based on network conditions and HEC response time.

#### `splunk.hec.lb.poll.interval`

Sets the health-check interval for HEC URIs.

* **Type**: `int`
* **Default**: `120`
* **Importance**: Low
* **Valid Values / Notes**: In seconds. A value less than or equal to `0` disables out-of-band health checks.

#### `splunk.hec.enable.compression`

Whether HEC request entities use Gzip compression.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: Applies to both `/event` and `/raw`. It can reduce network transfer volume but increases compression processing overhead on the Worker and Splunk.

### Kafka Connect Identity, Subscription, and Conversion

#### `connector.class`

Specifies the Connector implementation class.

* **Type**: `string`
* **Default**: None, required
* **Importance**: High
* **Valid Values / Notes**: Use `com.splunk.kafka.connect.SplunkSinkConnector`.

#### `tasks.max`

Sets the maximum number of Tasks the Connector may create.

* **Type**: `int`
* **Default**: `1`
* **Importance**: High
* **Valid Values / Notes**: The minimum is `1`. Effective parallelism is also limited by the number of input Topic partitions. Each Task has an independent HEC client and internal threads.

#### `tasks.max.enforce`

Whether Kafka Connect enforces the `tasks.max` limit for the Connector.

* **Type**: `boolean`
* **Default**: `true`
* **Importance**: Low
* **Valid Values / Notes**: This setting is deprecated and planned for removal in a future major version. Keep it at `true` and manage Task count through `tasks.max`; there is no same-named replacement property.

#### `topics`

Specifies the Kafka Topics consumed by the Connector.

* **Type**: `list`
* **Default**: Empty list
* **Importance**: High
* **Valid Values / Notes**: Use comma-separated values and choose exactly one of this setting and `topics.regex`. This setting is required when using positional `splunk.indexes`, `splunk.sources`, or `splunk.sourcetypes` values.

#### `topics.regex`

Subscribes to Kafka Topics with a regular expression.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: High
* **Valid Values / Notes**: Must be a valid Java regular expression, and exactly one of this setting and `topics` must be used. Positional Splunk metadata lists cannot be used; use HEC defaults or Kafka Headers instead.

#### `key.converter`

Specifies the Kafka Connect Converter for record keys.

* **Type**: `class`
* **Default**: `null`
* **Importance**: Low
* **Valid Values / Notes**: `null` inherits the Worker configuration. An explicit class must be an instantiable `Converter`. By default, the record key is not sent as event content.

#### `value.converter`

Specifies the Kafka Connect Converter for record values.

* **Type**: `class`
* **Default**: `null`
* **Importance**: Low
* **Valid Values / Notes**: `null` inherits the Worker configuration. The Connector serializes the converted result as text, JSON, or HEC Event content.

#### `header.converter`

Specifies the Converter for Kafka Headers.

* **Type**: `class`
* **Default**: `null`
* **Importance**: Low
* **Valid Values / Notes**: `null` inherits the Worker configuration. When `splunk.header.support` is enabled, ensure that Header conversion produces the expected values.

#### `transforms`

Specifies the aliases of single-message transforms (SMTs) applied in order before SinkTask processing.

* **Type**: `list`
* **Default**: Empty list
* **Importance**: Low
* **Valid Values / Notes**: Aliases must be unique. Each transform also requires `transforms.<alias>.type` and its transform-specific properties.

#### `config.action.reload`

Controls how changes or expiration of Config Provider values are handled.

* **Type**: `string`
* **Default**: `restart`
* **Importance**: Low
* **Valid Values / Notes**: Allowed values are `restart` and `none`. Config Provider registration is part of Worker configuration.

### Kafka Connect Error Handling

#### `errors.retry.timeout`

Sets the total retry duration for retriable errors in Kafka Connect framework stages.

* **Type**: `long`
* **Default**: `0`
* **Importance**: Medium
* **Valid Values / Notes**: In milliseconds. `0` disables retries, and `-1` means unlimited retries. This does not replace the Connector's own HEC batch retries.

#### `errors.retry.delay.max.ms`

Sets the maximum wait between Kafka Connect framework retries.

* **Type**: `long`
* **Default**: `60000`
* **Importance**: Medium
* **Valid Values / Notes**: In milliseconds and used with `errors.retry.timeout`. It does not affect HEC channel backpressure duration.

#### `errors.tolerance`

Controls whether the Kafka Connect framework tolerates reportable record-processing errors.

* **Type**: `string`
* **Default**: `none`
* **Importance**: Medium
* **Valid Values / Notes**: Allowed values are `none` and `all`. This primarily applies to framework stages such as conversion and SMTs and does not guarantee coverage of asynchronous HEC failures inside the Connector.

#### `errors.log.enable`

Whether Kafka Connect framework processing errors are logged.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: This logging is independent of the Connector's own logs. When enabled, use it with appropriate log access controls and retention policies.

#### `errors.log.include.messages`

Whether framework error logs include record context.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: When enabled, logs may contain the Topic, partition, Offset, timestamp, and other information. Assess the risk of data exposure in logs and use it with `errors.log.enable`.

#### `errors.deadletterqueue.topic.name`

Specifies the Kafka Connect framework dead-letter Topic.

* **Type**: `string`
* **Default**: Empty string
* **Importance**: Medium
* **Valid Values / Notes**: An empty value disables the DLQ. It is commonly used with `errors.tolerance=all`. The DLQ Topic must not be subscribed by `topics` or `topics.regex`, and this mechanism does not cover every internal HEC failure in the Connector.

#### `errors.deadletterqueue.topic.replication.factor`

Sets the replication factor when Kafka Connect automatically creates the DLQ Topic.

* **Type**: `short`
* **Default**: `3`
* **Importance**: Medium
* **Valid Values / Notes**: Used only when Connect creates the DLQ Topic. The value must not exceed the number of available Brokers in the Kafka cluster.

#### `errors.deadletterqueue.context.headers.enable`

Whether Kafka Connect error-context Headers are added to DLQ records.

* **Type**: `boolean`
* **Default**: `false`
* **Importance**: Medium
* **Valid Values / Notes**: When enabled, adds `__connect.errors.*` Headers. This is meaningful only when a DLQ is configured.

## Best Practices

### Enable HEC Acknowledgement for Production Use

**Applicable Scenario**: The Quick Start has established basic delivery, and the production environment needs to advance the corresponding Kafka offset only after Splunk returns indexer acknowledgement. The environment can tolerate duplicate events that may occur during failure recovery.

**Configuration Example**: Keep the Quick Start configuration, enable indexer acknowledgement in Splunk for the same HEC Token, and add the following settings.

```properties theme={null}
splunk.hec.ack.enabled=true
splunk.hec.event.timeout=300
splunk.hec.ack.poll.interval=10
splunk.hec.ack.poll.threads=2
```

**Key Notes**: When indexer acknowledgement is enabled for the HEC Token and acknowledgement polling operates normally, this mode provides at-least-once delivery, not end-to-end transactions or exactly-once delivery. A batch may be replayed if HEC receives it but the acknowledgement response is lost, or after a timeout, restart, or rebalance. Adjust the timeout and polling threads to actual indexing latency and concurrency. Give events stable identifiers or implement deduplication in Splunk when strict deduplication is required.

### Expand to Fixed Metadata Routing for Multiple Topics

**Applicable Scenario**: The initial single-Topic integration is stable, and application logs and audit logs now need to be written to different indexes with an explicit source and sourcetype for each data category.

**Configuration Example**: Replace `topics` in the Quick Start configuration and add metadata lists whose positions correspond one-to-one with the Topic order.

```properties theme={null}
topics=<application-topic>,<audit-topic>
splunk.indexes=<application-index>,<audit-index>
splunk.sources=<application-source>,<audit-source>
splunk.sourcetypes=<application-sourcetype>,<audit-sourcetype>
```

**Key Notes**: Each metadata list may contain one value shared by all Topics or the same number of values as `topics`. With multiple values, positions must correspond exactly. This approach does not apply to `topics.regex`. For dynamic routing, enable Kafka Header mapping or use the HEC Token's default metadata, and validate producer Header value ranges and access permissions.

### Increase Sending Parallelism Gradually as Throughput Grows

**Applicable Scenario**: One Task and one HEC sending thread have been stable, but the Kafka backlog continues to grow while Splunk HEC and the network still have spare capacity. More Tasks, connections, and compression are needed to increase throughput.

**Configuration Example**: Assuming at least four input partitions and sufficient HEC concurrency capacity, add the following starting values to the Quick Start configuration, then adjust them individually through load testing.

```properties theme={null}
tasks.max=2
splunk.hec.threads=2
splunk.hec.total.channels=4
splunk.hec.max.http.connection.per.channel=2
splunk.hec.concurrent.queue.capacity=100
splunk.hec.enable.compression=true
```

**Key Notes**: Effective Task count is limited by the number of input partitions, while HEC threads and channels apply to each Task. More parallelism increases Worker memory usage, connections, and Splunk processing pressure, and it does not guarantee global ordering across partitions, Tasks, channels, or batches. Adjust only one group of capacity settings at a time, observe Kafka lag, HEC responses, retries, Worker heap memory, and CPU, and then decide whether to scale further.

## Monitoring

### What to Monitor

Monitor Kafka Connect cluster health, Connector and Task status, consumption and HEC write throughput, Kafka lag and end-to-end latency, Offset commit progress, errors and retries, and Worker JVM heap, GC, and thread signals. Also monitor HEC health, acknowledgement timeouts, backpressure, and duplicate events. Monitor DLQ activity only when the corresponding framework error handling is enabled in the deployment; do not treat the DLQ as a fallback for all HEC delivery failures.

### Import the Grafana Dashboard

Download the [AutoMQ Connect Cluster Grafana Dashboard](https://automq-download-center.oss-cn-hangzhou.aliyuncs.com/connect-dashboard/automq-connect-cluster-dashboard.json), prepare a Prometheus-compatible data source that can query Kafka Connect metrics, and ensure that collected labels match the dashboard filters. On Grafana's import page, upload the JSON, select the corresponding data source, and save the dashboard.

## Limitations

* With HEC acknowledgement enabled and its conditions satisfied, delivery is at least once; retries, crashes, rebalances, or missing acknowledgement responses can still produce duplicate events. Without acknowledgement, the connector completes a batch after a successful HTTP response and provides no indexer-acknowledgement protection. Neither mode provides exactly-once delivery or destination-side deduplication.
* When Splunk returns `Invalid data format`, the Connector treats the response as completed and may advance the Kafka Offset even if the corresponding event is ignored.
* `topics.regex` cannot use positional mapping from `splunk.indexes`, `splunk.sources`, and `splunk.sourcetypes`; use Kafka Headers or HEC Token defaults instead.
* When HTTPS certificate validation is enabled, a custom trust store path readable by the Worker must be configured.
* In `/raw` mode, Splunk metadata applies to the entire request batch, and custom Kafka Headers are not added to raw events as indexed fields.

## FAQ

### The Connector Is Running Normally, but No Events Appear in Splunk

Check Task logs and status, HEC health, whether the Token is enabled and allowed to write to the target index, and whether the URI, endpoint mode, and `/raw` line-breaking rules are correct. If HEC acknowledgement is enabled, verify the Splunk Token's indexer acknowledgement setting. If Splunk returns `Invalid data format`, the Connector may already have advanced the Offset. After correcting HEC, permissions, or event formatting, validate again with one small, valid event.

### Duplicate Events Appear After Recovery or Restart

HEC may have received a batch before the Kafka Offset was committed, or the acknowledgement response may have been lost during a timeout, connection switch, or rebalance. Check HEC acknowledgement and retry logs, Task restart records, and Offset progress. Enable indexer acknowledgement correctly and set the timeout according to actual latency. For data that cannot tolerate duplicates, retain a stable identifier in each event and deduplicate during queries or downstream processing.

### Kafka Lag Keeps Growing or the Same Batch Is Replayed Repeatedly

Check the HEC response, acknowledgement polling, failure retries, and in-flight event count for the earliest incomplete batch, and confirm that at least one HEC channel is healthy. Restore the Token, index, network, or HEC capacity first, then evaluate parallelism. Do not reset Offsets before confirming the data boundary, because doing so may cause data loss or more duplicates.

### Trust Store Errors Occur During HTTPS Validation or Task Startup

Confirm that `splunk.hec.ssl.trust.store.path` exists and is readable on every Worker, that its type and password are correct, that the trust store contains the issuing chain for the Splunk certificate, and that the URI hostname matches the certificate. Restart the Task after correcting the trust store. Do not use disabled certificate validation as a long-term solution.

### Events Are Written to the Wrong Index, Source, or Sourcetype

Check the order of `topics`, the number and order of the three metadata lists, and whether Kafka Header mapping overrides the static values. With `topics.regex`, positional mapping does not take effect; use Headers or the HEC Token defaults instead. `/raw` also sends a batch with one set of request metadata, so ensure that records in the same batch can share that metadata.

### Event Times in Splunk Do Not Match Expectations

Check the Kafka record timestamp, `splunk.hec.use.record.timestamp`, the regular-expression named group, timeout logs, `timestamp.format`, `timestamp.timezone`, and HEC automatic time extraction settings in that order. Choose one primary time source and validate it with representative events. Avoid enabling extraction paths that override one another.