Manage Connectors

AutoMQ Connect uses two resource types. A Connect Cluster provides the Kafka Connect worker runtime and compute capacity. A Connector defines a source or sink data integration workload that runs on a Connect Cluster.

Prerequisites

Before you begin, prepare the following resources:

A running AutoMQ instance.
Kafka credentials with permissions for the topics and consumer groups that the Connector will use. For more information, see Manage Kafka ACLs.
The source or sink system and its access credentials.
A Kubernetes cluster, namespace, and ServiceAccount for the Connect Cluster workers. On AWS, bind the ServiceAccount to an IAM role that grants access to the Ops Bucket and any external AWS resources required by the plugin. For EKS preparation instructions, see Deploy to Amazon EKS.

Grant only the permissions required by the workload. A Sink Connector requires read permission for its Kafka topics and consumer group. A Source Connector requires write permission for its destination topics.

Create a Connect Cluster

Sign in to the AutoMQ Console.
In the navigation pane, choose Kafka Connect, choose the Connect Clusters tab, and then choose Create Connect Cluster.
Configure the Connect Cluster parameters.

Parameter	Description
Name	A name that is unique within the current environment.
Description	An optional description of the worker cluster.
Version	The Kafka Connect runtime version.
Plugins	The Managed or custom plugin versions and connector classes to install on the worker cluster.
AutoMQ Instance	The AutoMQ instance used by the Connect Cluster.
Number of Workers	The number of Kafka Connect workers. Valid values are 1 through 200.
Worker Tier	The CPU and memory allocated to each worker: 0.5 vCPU and 1 GiB, 1 vCPU and 2 GiB, 2 vCPU and 4 GiB, or 4 vCPU and 8 GiB.
Kubernetes Cluster	The Kubernetes cluster where the workers run.
Namespace	The Kubernetes namespace for the workers.
ServiceAccount	The Kubernetes ServiceAccount used by the worker Pods.
Cloud Identity	The cloud identity associated with the ServiceAccount, such as an AWS IAM role. It must grant access to the cloud resources required by the plugins.
SchedulingSpec	Optional Kubernetes scheduling settings, including node selectors, affinity and anti-affinity rules, and tolerations.
Worker Configuration	Optional Kafka Connect worker properties.
Prometheus Remote Write	Optional endpoint and authentication settings for pushing Connect Cluster metrics to Prometheus-compatible storage.

Review the settings and choose Create. Wait until the Connect Cluster status is Running before creating a Connector.

Ensure that the selected Kubernetes node pool has enough allocatable CPU and memory for the worker count and Worker Tier that you select.

Create a Connector

In Kafka Connect, choose the Connectors tab and then choose Create Connector.
Configure the Connector parameters.

Parameter	Description
Connect Cluster	The running Connect Cluster on which the Connector runs.
Name	A name that is unique within the selected Connect Cluster.
Description	An optional description of the data integration workload.
Plugin	A plugin installed on the selected Connect Cluster.
Connector Type	Source or Sink. The available type depends on the selected plugin.
Connector Class	The connector implementation class provided by the plugin.
Task Count	The maximum number of Connector tasks that can run in parallel.
Kafka Authentication	The Kafka security protocol and credentials used to connect to the AutoMQ instance.
Sink Topics	For a Sink Connector, the topics to consume and, when applicable, the consumer group.
Source Offsets	For a Source Connector, the offset topic and initial offset settings when supported by the plugin.
Plugin Configuration	Connection information and connector-specific properties for the external system.

Enter the plugin configuration. Use the structured form when it is available, or enter key-value properties in custom mode.
Review all parameters and choose Create.

Manage a Connector

Open a Connector details page to view its status, tasks, metrics, and logs. You can update supported plugin properties and the task count, and you can perform the following lifecycle operations:

Pause: Stops data processing without deleting the Connector.
Resume: Resumes a paused Connector.
Restart: Restarts the Connector and its tasks to recover from an error or apply supported runtime changes.

Worker count, Worker Tier, Kubernetes placement, installed plugins, and Prometheus Remote Write settings belong to the Connect Cluster. Update those settings on the Connect Cluster rather than on an individual Connector.

Delete a Connector

On the Connectors tab, locate the Connector, choose Delete, and confirm the operation.

Deleting a Connector cannot be undone. Verify the Connector name before you confirm the operation.

Overview

Getting Started

Deploy on Kubernetes

Subscriptions and Billings

Manage Environments

Manage Identities and Access

Managing AutoMQ

Migrate to AutoMQ

Manage Security

Kafka Connect

Monitoring & Alert

Table Topic

Best Practice

Eliminate Inter-Zone Traffics

Release Notes

Support

Appendix

Prerequisites

Create a Connect Cluster

Create a Connector

Manage a Connector

Delete a Connector

​Prerequisites

​Create a Connect Cluster

​Create a Connector

​Manage a Connector

​Delete a Connector

Prerequisites

Create a Connect Cluster

Create a Connector

Manage a Connector

Delete a Connector