Skip to main content

​
Overview

The Azure IoT Hub Source Connector reads device telemetry from the Event Hubs-compatible endpoint of Azure IoT Hub and writes the messages to a Kafka topic. The connector assigns IoT Hub partitions to tasks and converts each message into a fixed structured record containing the device identifier, Event Hubs offset, enqueue time, sequence number, original content, and system and application properties. Device payloads remain in the content field, so downstream consumers can process telemetry from different devices with consistent metadata.

​
Prerequisites

  • Azure IoT Hub must expose a readable Event Hubs-compatible endpoint, and you must have the endpoint, compatible name, partition count, consumer group, and shared access policy ready.
  • The shared access policy must have permission to read the IoT Hub event stream. Treat the primary key as sensitive information and keep it out of logs and source control.

​
License

Licensed under the MIT License.

​
Quick Start

Prepare a Connect Cluster, Kafka, and an accessible Azure IoT Hub. Confirm network connectivity and access permissions, then submit the configuration by following Manage Connectors.
Replace the angle-bracketed values with your resources. If IotHub.ConsumerGroup is omitted, $Default is used. If neither an initial time nor initial offsets are specified, the first startup begins at the beginning of the stream. After startup, records are written to Kafka.Topic.

​
Configuration

​
Azure IoT Hub Connection

​
IotHub.EventHubCompatibleName

The Event Hubs-compatible name of the IoT Hub events endpoint.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / notes: Required. Find it in Azure Portal under IoT Hub, Endpoints, Events.

​
IotHub.EventHubCompatibleEndpoint

The Event Hubs-compatible address of the IoT Hub events endpoint.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / notes: Required and must be a parseable endpoint URI.

​
IotHub.AccessKeyName

The shared access policy name used to read the event stream.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / notes: Required, for example the name of a configured service policy.

​
IotHub.AccessKeyValue

The primary key for the selected shared access policy.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / notes: Required. Provide it through a secure configuration mechanism.

​
Azure IoT Hub Consumption

​
IotHub.ConsumerGroup

The consumer group used to read the IoT Hub event stream.
  • Type: string
  • Default: $Default
  • Importance: Medium
  • Valid values / notes: Use a consumer group that exists on the IoT Hub events endpoint. Use a dedicated group for this application to isolate its read position from other consumers.

​
IotHub.Partitions

The number of partitions in the IoT Hub.
  • Type: int
  • Default: None
  • Importance: High
  • Valid values / notes: Required and must match the actual IoT Hub partition count. It is used to build task partition assignments.

​
Initial Position

​
IotHub.StartTime

Reads messages starting at the specified UTC time.
  • Type: string
  • Default: Empty string
  • Importance: Medium
  • Valid values / notes: Optional. Use an ISO-8601 timestamp such as 2026-09-18T00:00:00Z. It takes precedence over IotHub.Offsets for initial positioning; a saved Kafka Connect source offset still takes precedence.

​
IotHub.Offsets

Specifies the initial Event Hubs offset for each IoT Hub partition as a comma-separated string.
  • Type: string
  • Default: Empty string
  • Importance: Medium
  • Valid values / notes: Optional. Supply offsets in partition order. This setting is ignored when IotHub.StartTime is set. An empty entry starts the corresponding partition at the beginning of the stream.

​
Polling

​
BatchSize

The maximum number of messages requested from IoT Hub per poll.
  • Type: int
  • Default: 100
  • Importance: Medium
  • Valid values / notes: Use a positive integer. A larger value can reduce polling frequency but increases the amount of data returned by one poll.

​
ReceiveTimeout

The maximum time to wait for messages, in seconds.
  • Type: int
  • Default: 60
  • Importance: Medium
  • Valid values / notes: Passed to the Event Hubs receiver. Adjust it according to message arrival rate and latency requirements.

​
Kafka Output

​
Kafka.Topic

The Kafka topic that receives IoT Hub messages.
  • Type: string
  • Default: None
  • Importance: High
  • Valid values / notes: Required. Records read from all assigned partitions are written to this topic.

​
Best Practices

​
Set an explicit starting point for the first ingestion

Applicable business scenario: You are connecting an existing IoT Hub to Kafka for the first time and need to establish a baseline from a selected time while continuing to read subsequent messages. Configuration example:
Key points: IotHub.StartTime controls the initial position only when no saved source offset exists. Use a dedicated consumer group to isolate this connector’s read progress from other applications. Use UTC and choose the baseline time for the business requirement; an earlier time may produce more data during the first ingestion.

​
Scale tasks with IoT Hub partitions

Applicable business scenario: The IoT Hub has multiple partitions and one task cannot meet the required throughput or latency, so you need more parallel readers. Configuration example:
Key points: IotHub.Partitions should match the actual partition count, and tasks.max can be set to a parallelism level no greater than that count. The connector assigns partitions to tasks in round-robin order, so one task may own multiple partitions. More tasks than partitions do not add read parallelism.

​
Monitoring

​
What to Monitor

Monitor Kafka Connect Worker, Connector, and Task health; task restarts; throughput; end-to-end latency; source offset commits; errors and retries; and Worker JVM heap, thread, and GC signals. If error handling and a dead-letter queue are enabled, also monitor dead-letter writes and backlog.

​
Import the Grafana Dashboard

Download the dashboard from the download URL, use a Prometheus data source that contains Kafka Connect metrics, ensure the metric labels match the dashboard variables, and import the JSON file in Grafana.

​
Limitations

  • One Kafka topic receives records from all assigned IoT Hub partitions; the connector does not automatically route records to different topics by device or message schema.
  • IotHub.StartTime and IotHub.Offsets are initial-position settings used only when no saved source offset exists; a saved Kafka Connect offset has higher precedence.
  • The connector depends on the Azure IoT Hub Event Hubs-compatible interface and valid shared access credentials and cannot produce source records without those external resources.

​
FAQ

​
Why does the connector start without reading messages?

Check that the Event Hubs-compatible name, endpoint, shared access policy, and consumer group belong to the same IoT Hub, and confirm that IotHub.Partitions matches the actual partition count. If IotHub.StartTime is configured, confirm that it is UTC and within the retained readable data range. If Kafka Connect has a saved source offset, the connector resumes from that position.

​
Why did IotHub.Offsets not determine the starting position?

If IotHub.StartTime is also set, the offsets are ignored. An existing Kafka Connect source offset also takes precedence over both initial-position settings. Check the consumer group and offset storage used by the connector, then recreate the connector’s read state if a new initial position is required.

​
How should I choose tasks.max?

Start with the IoT Hub partition count and set tasks.max to a value no greater than that count. Tasks work on assigned partitions, so values above the partition count do not increase read parallelism. After changing it, monitor Task health, throughput, and latency.