> ## Documentation Index > Fetch the complete documentation index at: https://docs.automq.com/llms.txt > Use this file to discover all available pages before exploring further. # RSS Source Connector > Configure and operate the RSS Source Connector in AutoMQ Connect, including prerequisites, configuration, monitoring, and troubleshooting. ## Overview The RSS Source Connector periodically reads one or more RSS or Atom feeds and writes their entries to a single Kafka topic. Each processable entry becomes one Kafka record. The record value contains the entry title, identifier, link, `description` content, author, and date, together with the feed URL and title; the record key is null. On each poll, the Connector reads the current feed content and uses entry fingerprints saved for each URL to avoid resending old entries that it can still identify. A fingerprint is derived from the entry title, link, identifier, `description` content, and author; it does not include the date or feed title. The Connector is suitable for ingesting updates from news sites, blogs, announcements, or other feeds into Kafka for downstream search, notification, analytics, or stream-processing applications. ## License This Connector is licensed under the MIT License. ## Quick Start Prepare a Connect Cluster, Kafka, and an accessible RSS or Atom feed, and confirm network connectivity and access permissions. For preparation and management procedures, see [Manage Connectors](../manage-connectors). ```properties theme={null} connector.class=org.kaliy.kafka.connect.rss.RssSourceConnector rss.urls= topic= value.converter=org.apache.kafka.connect.json.JsonConverter value.converter.schemas.enable=true ``` Replace `` with an RSS or Atom URL accessible to the Worker, and replace `` with the Kafka topic that receives the records. Spaces in the URL must be percent-encoded. This example uses JSON output with a schema to preserve the field structure defined by the Connector. After you submit the configuration, the first poll reads the entries currently present in the feed. Subsequent polls output only entries that have not been recorded or whose fingerprints have changed. A change to only the date or feed title does not produce a new fingerprint. ## Configuration ### Feed Input #### `rss.urls` The RSS or Atom feed URLs to poll. * **Type**: `string` * **Default**: None * **Importance**: High * **Valid Values / Notes**: Provide at least one valid URL. Separate multiple URLs with a single space. Spaces within a URL must be percent-encoded; do not use repeated spaces, tabs, or line breaks as separators. Keep URLs unique because a Task fails with an initialization conflict if it receives the same URL more than once. URLs must not contain credentials, access tokens, or signed query parameters. * **Required**: Yes ### Kafka Output #### `topic` The Kafka topic that receives entries from all feeds. * **Type**: `string` * **Default**: None * **Importance**: High * **Valid Values / Notes**: Use a valid, writable Kafka topic name. All feeds in one Connector instance write to this topic; separate topics cannot be configured for individual feeds. * **Required**: Yes ### Polling and Tasks #### `sleep.seconds` The number of seconds each Task waits before starting the next poll after the initial poll. * **Type**: `int` * **Default**: `60` * **Importance**: Medium * **Valid Values / Notes**: Must be greater than or equal to `0`. Setting this property to `0` removes the delay between polls and may create a high-frequency loop. The actual polling interval also includes the time the Task takes to read and parse all feeds. #### `tasks.max` The maximum number of Tasks that the Connector may create. * **Type**: `int` * **Default**: `1` * **Importance**: High * **Valid Values / Notes**: Must be greater than or equal to `1`. The actual number of Tasks does not exceed the number of URLs in `rss.urls`. Multiple URLs are assigned to Tasks in round-robin order based on their input order, and URLs within the same Task are still polled sequentially. ### Connector and Serialization #### `connector.class` The RSS Source Connector implementation class to load. * **Type**: `string` * **Default**: None * **Importance**: High * **Valid Values / Notes**: Use `org.kaliy.kafka.connect.rss.RssSourceConnector`. * **Required**: Yes #### `value.converter` Specifies the Converter for record values at the Connector level. * **Type**: `class` * **Default**: `null` * **Importance**: Low * **Valid Values / Notes**: When omitted or set to `null`, this property inherits the Worker's value Converter. When explicitly configured, it must be an instantiable Kafka Connect `Converter` implementation, and its schema or serialization options must be configured for the selected Converter. The Connector outputs a fixed Connect Schema and Struct value. ## Best Practices ### Scale Polling Tasks by Feed Count **Use Case**: One Connector must continuously read multiple independent feeds, and having one Task fetch all URLs sequentially has increased update latency. Gradually increase the number of Tasks based on the number of URLs so that different feeds can be assigned to different Tasks and polled in parallel. **Configuration Example**: Keep the Connector, topic, and Converter settings from the Quick Start, expand `rss.urls` into a list of unique URLs, and add or override the following properties: ```properties theme={null} rss.urls= tasks.max=3 ``` **Key Considerations**: The actual number of Tasks is the smaller of `tasks.max` and the number of URLs. The five URLs in this example are assigned in round-robin order to no more than three Tasks. A URL is not split across multiple Tasks, and feeds within the same Task are still fetched sequentially, so a slow feed delays subsequent feeds in that Task. Start with a low degree of parallelism, then adjust it gradually based on Task latency, source-site capacity, and Worker resources. Keep the URL list free of duplicates to prevent an initialization failure if duplicate URLs are assigned to the same Task. ### Schedule Polling by Update Frequency and Retain Enough Content **Use Case**: A feed is continuously updated, and you need to balance the source request rate, message freshness, and the recovery window after downtime. Set the polling interval according to acceptable business latency and ensure that the upstream feed retains entries for a sufficiently long window. **Configuration Example**: Add or override the following property in the Quick Start configuration: ```properties theme={null} sleep.seconds=300 ``` **Key Considerations**: `300` seconds is only an example. Set this value according to the acceptable discovery delay and account for the time needed to fetch and parse each polling cycle. A shorter interval discovers new entries sooner but increases load on the source site and Worker; a longer interval increases the risk that an entry appears and then rolls out of the feed between polls. The Connector offset is a window of entry fingerprints from the current feed, not a historical cursor on the source. The upstream feed should therefore retain enough entries, and after maintenance or prolonged downtime, you should check for content that may have already rolled out of the feed. ## Monitoring ### What to Monitor Monitor Kafka Connect health, Connector and Task status, throughput, latency, offset commits, errors, retries, and Worker JVM signals. Monitor DLQ activity only when the corresponding error handling is enabled. For RSS fetching, also use Task logs to distinguish between a feed with no new entries and a failed URL read or XML parse. ### Import the Grafana Dashboard Confirm that Connect metrics are available in a Grafana data source and that collected labels meet the dashboard's filtering requirements. Download the [Kafka Connect Dashboard](https://automq-download-center.oss-cn-hangzhou.aliyuncs.com/connect-dashboard/automq-connect-cluster-dashboard.json), import the JSON into Grafana, and select the corresponding data source. ## Limitations * The Connector reads only the current entries still visible in the feed at each poll and has no replayable upstream log. Entries that roll out of the feed between polls or during downtime cannot be retrieved later. * A duplicate window exists between successful Kafka record delivery and Source Offset persistence, while rolling feeds also create an unrecoverable window. Therefore, the Connector cannot guarantee exactly-once delivery, no duplicates, no omissions, or unconditional at-least-once delivery. * Entries must contain usable `title`, `id`, and `link` fields. An entry missing any of these required fields cannot be constructed with the fixed Schema, is filtered after the exception is logged, and is not sent to a Connector-specific DLQ. * Version 0.1.1 does not provide Connector-level configuration for HTTP authentication, custom request headers, conditional requests, proxies, connection timeouts, or read timeouts. Do not put credentials or tokens in `rss.urls`, because Task startup logs may contain the complete configuration. * A single poll returns all new entries from every feed assigned to that Task, with no Connector-level limit on record count, byte size, or memory use. Large feeds or accumulated content may create a large batch in one poll. * The Connector does not specify Kafka record keys or target partitions. The feed URL is used only for the Source Offset partition; it does not provide fixed Kafka partition routing or guarantee global ordering across feeds, Tasks, or Kafka partitions. * `content` maps only to the feed entry description. Categories, enclosures, comments, raw XML, and other extension fields are not written to the record value. ## FAQ ### Why Does a Task Fail During Startup? Check that `rss.urls` contains at least one valid URL, that multiple URLs are separated by exactly one space, that spaces within URLs are percent-encoded, and that the list contains no duplicate URLs. A Task encounters an initialization conflict if it receives the same URL more than once. Then confirm that `topic` is a valid, writable Kafka topic and inspect the Task logs for configuration validation or initialization exceptions. Correct the configuration and restart the failed Task. ### Why Is the Connector Running but Producing No New Records to the Topic? The feed may have no new entries, or the URL read or XML parse may have failed during the current poll. Confirm that the Worker can access the feed, verify that the response is still valid RSS or Atom XML, and inspect the Task logs for fetch warnings. If the feed contains entries, confirm that each entry includes `title`, `id`, and `link`; entries missing these fields are filtered. After you fix the source content or network issue, the Connector reads the URL again on a subsequent poll. ### Why Are Records Duplicated or Entries from the Downtime Period Missing After a Restart? If a failure occurs after Kafka acknowledges a record but before the corresponding Source Offset is persisted, an entry that remains in the feed may be output again after restart. Conversely, the Connector can view only the current feed snapshot, so entries that rolled out of the feed during downtime cannot be recovered. Downstream systems should use stable business fields for idempotent processing, and the upstream feed's entry retention window should account for the longest expected maintenance period. ### Why Does the Task Count Stop Increasing After I Increase `tasks.max`? Each feed URL can be assigned to at most one Task, so the actual number of Tasks cannot exceed the number of URLs. Confirm that `rss.urls` contains enough unique URLs and check the actual Task count in the Connector status. If multiple URLs are still assigned to one Task, you can increase `tasks.max` further, but increasing it beyond the URL count does not provide additional parallelism. ### Why Do the Output Fields Not Exactly Match the Original Feed XML? The Connector uses a fixed field model instead of forwarding raw XML. `content` contains only the entry description. `date` uses the update time when available, otherwise the publication time, and is written to the record value as an ISO-8601 string rather than as the Kafka record timestamp. If categories, enclosures, raw XML, or other extension fields are required, transform the feed upstream or use another downstream ingestion method that preserves those fields.