Skip to main content

​
Overview

The IRC Source Connector connects to an IRC server, joins specified channels, and writes received PRIVMSG messages to a Kafka topic. It can feed chat messages from public channels or controlled internal channels into stream-processing, search, and archival pipelines. The Kafka message key is the message target, usually the channel name, while the value contains the receive time, message target, sender information, and message text. When multiple channels are configured, collection can be distributed across one or more Source Tasks.

​
Prerequisites

  • The IRC network must provide a plaintext TCP endpoint accessible to the Connect Worker and allow the selected bot nickname to complete registration. This Connector does not support TLS and should therefore be used only on trusted or isolated networks.
  • The target channels must allow the configured nickname to join directly. This Connector cannot provide a JOIN key for channels that require a channel key.

​
License

Uses Apache License 2.0.

​
Quick Start

Prepare a Connect Cluster, Kafka, the target Kafka topic, and an accessible IRC server and channel, and confirm network connectivity and access permissions. For preparation and management, see Manage connectors.
Replace <irc-server> with the IRC server hostname or address, <irc-channel> with the channel name including its channel prefix, such as #events, and <kafka-topic> with the Kafka topic that receives the messages. The example uses the default port 6667 for plaintext IRC; changing the port does not enable TLS. When the bot name is omitted, the Connector dynamically generates a nickname.

​
Configuration

​
IRC Connection

​
irc.server

The IRC server hostname or address.
  • Type: string
  • Default: None
  • Importance: High
  • Valid Values / Notes: Must be a non-empty address accessible to the Connect Worker and, together with irc.server.port, point to a plaintext IRC service.
  • Required: Yes

​
irc.server.port

The TCP port used to connect to the IRC server.
  • Type: int
  • Default: 6667
  • Importance: Low
  • Valid Values / Notes: Use a valid port on which the target IRC service is listening. Changing the port does not enable TLS and cannot be used to connect securely to a TLS-only port.

​
IRC Identity and Authentication

​
irc.bot.name

The nickname, username, and real name that the Source Task uses when registering with the IRC server.
  • Type: string
  • Default: Dynamically generated in the format KafkaConnectBot_ followed by 6 random letters or digits
  • Importance: Low
  • Valid Values / Notes: Must comply with the target IRC network’s nickname rules. When omitted, each Task generates its own nickname. Do not explicitly reuse the same fixed nickname when running multiple Tasks.

​
irc.password

The optional server password sent to the IRC server during connection registration.
  • Type: password
  • Default: Empty string
  • Importance: Low
  • Valid Values / Notes: An empty string means that no PASS command is sent. A non-empty password is sent over the plaintext IRC connection and should not be used on untrusted networks or written to logs or version control.

​
Channel Subscriptions and Kafka Destination

​
irc.channels

The list of IRC channels that the Source Task joins and collects messages from.
  • Type: list
  • Default: None
  • Importance: High
  • Valid Values / Notes: Provide at least one non-empty channel. Separate multiple channels with commas and use a channel prefix accepted by the target IRC network, usually #. The Connector does not reject empty elements, duplicate channels, or invalid channel syntax in advance.
  • Required: Yes

​
kafka.topic

The single Kafka topic that receives all IRC messages.
  • Type: string
  • Default: None
  • Importance: High
  • Valid Values / Notes: Specify a valid Kafka topic name. This value is not a list and does not support generating a separate topic for each channel.
  • Required: Yes

​
Kafka Connect Runtime

​
connector.class

The IRC Source Connector implementation class to load.
  • Type: string
  • Default: None
  • Importance: High
  • Valid Values / Notes: Use com.github.cjmatta.kafka.connect.irc.IrcSourceConnector.
  • Required: Yes

​
tasks.max

The maximum number of Source Tasks used to distribute the configured IRC channels.
  • Type: int
  • Default: 1
  • Importance: High
  • Valid Values / Notes: The minimum value is 1. The actual number of Tasks is the smaller of the number of elements in the channel list and tasks.max. Each Task establishes an independent IRC connection.

​
key.converter

The Connector-level Converter override used to serialize string message keys.
  • Type: class
  • Default: null (inherits the Worker configuration)
  • Importance: Low
  • Valid Values / Notes: Specify an instantiable Kafka Connect Converter implementation class. The message key is the PRIVMSG target, usually the channel name.

​
value.converter

The Connector-level Converter override used to serialize the schema-bearing IRC message Struct.
  • Type: class
  • Default: null (inherits the Worker configuration)
  • Importance: Low
  • Valid Values / Notes: Specify an instantiable Kafka Connect Converter implementation class that can handle a schema-bearing Struct. The message value contains the createdat, channel, sender, and message fields.

​
Best Practices

​
Scale to Multiple Channels with Independent Connections

Applicable scenario: After ingesting a single channel, you need to aggregate multiple channels from the same IRC server into one Kafka topic and distribute their connections across different Tasks. This is suitable for routine expansion as the number of channels grows, but it does not provide parallel collection within a single channel. Configuration example: Starting with the Quick Start configuration, expand the channel list to three channels and set the maximum number of Tasks to 3. Omit irc.bot.name so that each Task generates its own nickname, avoiding explicit reuse of the same fixed nickname across multiple connections.
Key considerations: The actual number of Tasks does not exceed the number of channels. In this example, each channel is assigned to at most one independent Task and IRC connection. Messages from all channels are still written to the same kafka.topic, and the message key identifies the PRIVMSG target. Changing tasks.max rebuilds the channel groupings and connections. Because the change provides neither historical replay nor message handoff between the old and new Tasks, apply it during a maintenance window in which a brief collection gap is acceptable.

​
Monitoring

​
What to Monitor

Monitor Kafka Connect health, Connector and Task status, throughput, latency, offset commits, errors, retries, and Worker JVM signals; monitor DLQ activity only when the corresponding error handling is enabled. Also use message throughput and IRC server connection information to determine whether collection is functioning, because a runtime disconnection does not necessarily cause the Task status to change to failed.

​
Import the Grafana Dashboard

Ensure that Connect metrics are available in a Grafana data source and that the collected labels match the dashboard filters; download the Kafka Connect Dashboard, import the JSON into Grafana, and select the corresponding data source.

​
Limitations

  • The Connector collects only IRC PRIVMSG messages. Events such as JOIN, PART, NOTICE, TOPIC, KICK, and QUIT are not written to Kafka. Private messages sent directly to the bot nickname may also be collected.
  • Only plaintext IRC TCP connections are supported. TLS, STARTTLS, and certificate configuration are not supported. Changing the port does not provide transport encryption, and a non-empty server password is also sent in plaintext.
  • The Connector cannot provide a JOIN key for channels protected by a channel key.
  • Records do not contain a recoverable IRC source offset. After a restart, the Connector can collect only new messages received after it rejoins the channels. Messages sent while it was stopped or messages still in memory but not yet delivered cannot be replayed from the source.
  • The Connector does not provide exactly-once, at-least-once, duplicate-free, or loss-free guarantees.
  • Runtime disconnections, nickname conflicts, and some IRC server errors do not automatically trigger reconnection, channel rejoin, or Task failure. A Task may still appear to be running while no longer producing messages.
  • Incoming messages use an unbounded in-memory queue, and each poll drains the entire backlog available at that time. An incoming rate that remains higher than Kafka’s write capacity may increase Worker memory pressure.
  • Explicitly using the same fixed irc.bot.name for multiple Tasks may cause nickname conflicts. The Connector does not derive a unique nickname for each Task or handle conflicts automatically.
  • When a Task stops or is reconfigured, it waits for the IRC reader thread to exit without a bounded timeout. This may delay the stop process and clears in-memory messages that have not yet been delivered.

​
FAQ

​
The Task Is Running, but the Kafka Topic No Longer Receives New Messages

Runtime IRC disconnections, nickname conflicts, incorrect passwords, or channel rejections do not necessarily cause the Kafka Connect Task to enter a failed state, and the Connector does not reconnect automatically. Check the Worker logs, IRC server connection records, nickname usage, and channel access rules. After confirming that the configuration is valid, restart the Connector or Task and check message throughput again. IRC messages sent during the restart cannot be replayed.

​
Why Are Messages Sent During Downtime Not Collected After a Restart?

This Connector does not record a recoverable IRC message position or request historical messages from the IRC server. After a restart, it reconnects and rejoins the channels, and receives only PRIVMSG messages that arrive afterward. If the business requires complete history, provide a separate historical source on the IRC side or explicitly accept and monitor collection gaps in the downstream design.