Skip to main content

​
Overview

The ArangoDB Sink Connector consumes records from a Kafka topic and writes the records to a specified ArangoDB database. Each record is routed to an ArangoDB collection based on the Kafka topic name: if the topic name does not contain a period, the full name is used; if it contains a period, the suffix after the last period is used. For example, dbserver1.inventory.customers is written to the customers collection. The connector uses the first field of the Kafka record key as the ArangoDB document _key. A non-null object value inserts a new document or fully replaces the existing document with the same _key; in normal mode, a tombstone record with a null value deletes the document by _key. It is suitable for continuously synchronizing business objects, cache views, or change events to ArangoDB, but it does not provide field-level partial updates, relationship edge mapping, or transactions across Kafka and ArangoDB.

​
Prerequisites

  • Use ArangoDB 3.4 or later, pre-create the target database, and pre-create a collection for each subscribed topic using the suffix after the topic’s last period as the collection name. The connector does not create these resources automatically, and the account used must have document write and delete permissions for the target database and collections.

​
License

Uses the MIT License.

​
Quick Start

Prepare a Connect cluster, Kafka, the target ArangoDB database, and the customers collection in advance, and confirm network connectivity and access permissions. For specific preparation and management operations, see Manage Connectors.
Replace the ArangoDB address, port, username, password, and database placeholders. Write an object key such as {"id":"customer-1001"} and an object value such as {"id":"customer-1001","name":"Alice","tier":"gold"} to the customers topic. The connector uses the value of the first field in the key as _key and removes the id field with the same name from the written document’s value. Sending a new object value with the same key later fully replaces the original document instead of updating only the fields that changed. Before deploying to production, also assess the risk that this version may expose the password when recording the original configuration.

​
Configuration

​
Connector Identity and Tasks

​
connector.class

The implementation class of the ArangoDB Sink Connector to load.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / considerations: Use io.github.jaredpetersen.kafkaconnectarangodb.sink.ArangoDbSinkConnector. Do not use the old class name in the installation package’s quick start file.
  • Required: Yes

​
tasks.max

The maximum number of tasks that the connector requests to create.
  • Type: int
  • Default: 1
  • Importance: High
  • Valid values / considerations: Must be at least 1. Actual parallelism is limited by topic partition assignment; tasks do not have global ordering or conflict coordination for the same document key.

​
Topic Subscription

​
topics

The list of Kafka topics to consume.
  • Type: list
  • Default: Empty list
  • Importance: High
  • Valid values / considerations: Use comma-separated literal topic names. Exactly one of topics and topics.regex must be configured with a non-empty value; the suffix after the last period of each topic must correspond to an existing ArangoDB collection.
  • Required: conditional (required when topics.regex is not configured)

​
topics.regex

The regular expression used to select Kafka topics.
  • Type: string
  • Default: ""
  • Importance: High
  • Valid values / considerations: Use Java regular expression syntax. Exactly one of topics and topics.regex must be configured with a non-empty value; do not allow the expression to match the DLQ topic used for error handling.
  • Required: conditional (required when topics is not configured)

​
ArangoDB Connection and Target

​
arangodb.host

The hostname or address of the ArangoDB server.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / considerations: Enter an ArangoDB hostname or address accessible to the Connect worker. This configuration has no non-empty or reachability validation.
  • Required: Yes

​
arangodb.port

The port of the ArangoDB server.
  • Type: int
  • Default: None
  • Importance: High
  • Valid values / considerations: Enter the actual listening port. The connector only checks whether the value can be parsed as an integer; it does not check for a positive value or a valid TCP port range.
  • Required: Yes

​
arangodb.database.name

The name of the ArangoDB database to which all records are written.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / considerations: The database must already exist, and all subscribed topics use this one database. Collection names are determined by topic suffixes and cannot be mapped in this configuration.
  • Required: Yes

​
ArangoDB Authentication

​
arangodb.user

The username used to connect to ArangoDB.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / considerations: The account must have document write and delete permissions for the target database and collections. This configuration has no non-empty or authentication precheck.
  • Required: Yes

​
arangodb.password

The password used to connect to ArangoDB.
  • Type: password
  • Default: ""
  • Importance: High
  • Valid values / considerations: An omitted value uses an empty string. Version 1.0.4 records the original configuration Map in INFO logs, which may expose the password provided here; restrict access to and retention of Worker logs, and use a fixed build or another assessed mitigation before deploying to production.

​
Record Conversion

​
key.converter

The connector-level Kafka record key converter class.
  • Type: class
  • Default: null
  • Importance: Low
  • Valid values / considerations: When omitted, inherits the Worker’s key converter. The converter must produce an object key with at least one field whose first value is not null; the connector uses only the value of the object’s first field as the ArangoDB _key and does not support raw strings, numbers, byte arrays, or stable composite-key mappings.

​
value.converter

The connector-level Kafka record value converter class.
  • Type: class
  • Default: null
  • Importance: Low
  • Valid values / considerations: When omitted, inherits the Worker’s value converter. After conversion, a non-null value must be an object; arrays and scalars cannot be written as ArangoDB documents. In normal mode, a converted null value indicates deletion.

​
Best Practices

​
Write to Multiple Collections by Topic Rules When Expanding an Integration

Applicable business scenario: You have already created multiple collections for business objects in one ArangoDB database and want the same connector to subscribe to subsequent topics that follow a common naming rule. The following configuration matches topics with the inventory. prefix; each topic is still written to the collection corresponding to the name after its last period.
Key considerations: Remove topics from the quick start configuration because it is mutually exclusive with topics.regex. Before enabling the connector, create a same-named collection for the suffix of every topic that might match, such as customers and orders for inventory.customers and inventory.orders, respectively; also confirm that the expression does not accidentally subscribe to internal topics or DLQ topics. All matching topics are written to the same arangodb.database.name; different databases cannot be selected by topic.

​
Handle Updates and Deletes Consistently When Ingesting CDC Events

Applicable business scenario: An upstream CDC system emits objects with before and after, and you want to write after as the current complete document to ArangoDB and delete the corresponding document when after=null. The following configuration adds the connector’s built-in CDC SMT to the quick start configuration.
Key considerations: A non-null after must be a complete object, and the connector fully replaces the document with the same _key; after=null follows the key-based deletion path. CDC mode drops a tombstone whose top-level value is null, so a delete event must be expressed as after=null inside a non-null envelope. For schemaless objects, a missing after cannot be distinguished from an explicit after=null, and is also treated as a delete; validate the upstream event structure before enabling this mode, and ensure that the key is a supported single-field object.

​
Monitoring

​
What to Monitor

Monitor Kafka Connect health, Connector and Task status, throughput, latency, offset commits, errors, retries, and Worker JVM signals; monitor DLQ activity only when the corresponding error handling is enabled. Because the connector has no plugin-specific health checks or metrics, also use the state of target documents in ArangoDB and Worker logs to determine whether writes match expectations.

​
Import the Grafana Dashboard

Confirm that Connect metrics are connected to a Grafana data source and that collection labels meet the dashboard’s filtering requirements; download the Kafka Connect Dashboard, import the JSON into Grafana, and select the corresponding data source.

​
Limitations

  • The connector does not automatically create an ArangoDB database or collection and does not provide an explicit topic-to-collection mapping; each record can only be written to the pre-created collection corresponding to the suffix after the current topic’s last period.
  • A non-null value is inserted or replaced as a complete document. The connector does not support field-level partial updates, creating a dedicated schema, soft deletes, foreign-key conversion, or edge relationship mapping; old fields omitted from the new value are not merged or retained.
  • A Kafka key must be an object with at least one field whose first value is not null, and the connector uses only the first field’s value; raw-type keys, empty keys, and stable composite-key mappings are not supported. A non-null value must convert to an object; arrays and scalars are not supported.
  • Version 1.0.4 does not provide connector-level TLS, truststore, connection timeout, retry-count, or backoff configurations.
  • Record conversion exceptions and ArangoDB write exceptions are thrown directly from task.put and cause the task to fail; errors.tolerance and the DLQ cannot handle invalid keys, invalid values, or database write errors on this path.
  • There is no cross-system transaction between the connector and Kafka offsets. Earlier writes in a polling batch may already have succeeded when a later write causes the task to fail; after a manual restart, records after the last committed offset may be replayed. Full replacement or deletion by a stable _key can reduce the likelihood of duplicate document creation, but does not provide exactly-once, no-replay, or global-order guarantees.
  • Version 1.0.4 may log the original configuration Map containing arangodb.password at INFO level; restricting log access alone does not provide encryption for the transport path, so the risk of credential exposure must be assessed separately before production use.

​
FAQ

​
Why did some fields disappear from the original document after an update?

The connector performs a complete document replacement for a non-null value and does not merge fields. Check whether the new message value contains every field that the final document must retain, and confirm that the first field of the key still points to the same _key. If the upstream system produces only field differences, first reconstruct the complete object in Kafka Streams or another processing layer, and then pass it to the connector for writing.

​
Why was a record written to the wrong collection, or did it fail because the collection does not exist?

The connector uses the string after the last period in the topic name as the collection name. Check the actual topic name and the matching range of topics or topics.regex, and pre-create the corresponding collection; for example, dbserver1.inventory.customers requires a customers collection. The connector cannot use configuration to remap this topic to another collection.

​
Why did a tombstone record not delete the document?

First confirm that the key is an object with at least one field whose first value is not null. In normal mode, a tombstone with a null value deletes the document by that key. When the CDC SMT is enabled, a top-level tombstone is dropped, and deletion must use a non-null CDC envelope with after=null. Do not use an unschematized envelope without after to represent another event, because it also enters the deletion path.

​
How should I recover after a task fails, and will records be written again?

First confirm the cause from the Worker logs and ArangoDB state, fix the invalid records, permissions, target resources, network, or server-side issue, and then manually restart the failed task. Ordinary connector or driver exceptions are not automatically categorized and retried by the plugin, and errors.tolerance or the DLQ cannot be relied on to skip failures inside task.put. Because successful writes and Kafka offset commits are not part of one transaction, a restart may replay records after the last committed offset, including earlier batches that were already written before the failure. After recovery, check the target documents and consumer-group offsets; do not treat returning the task to RUNNING as proof that no replay or data overwrite occurred.