Kafka Sink Connector Consuming in Bursts Despite Tuning – Potential Backpressure or Polling Issue

[captainpro-eng]

Hi team,

I'm using the ClickHouseSinkConnector from the Kafka Connect framework, and I'm experiencing persistent burst patterns in Kafka consumption, even after trying extensive tuning. The sink does not consume continuously, which leads to lag buildup and inefficient processing.

Configuration:-
class: com.clickhouse.kafka.connect.ClickHouseSinkConnector
tasksMax: 15
config:
topics.regex: '^progate.progate.*'
hostname: sd-clickhouse
port: 8123
protocol: http
database: progate
table: progate_mdr_tbl_dist
username: mcarbon
password: "mcarbon"
ignorePartitionsWhenBatching: true
topic2TableMap: progate.progate.debezium_signal=debezium_signal,...
transforms: unwrap
transforms.unwrap.type: io.debezium.transforms.ExtractNewRecordState
errors.log.enable: true
value.converter.schemas.enable: false
consumer.override.max.poll.records: 20000
consumer.override.fetch.max.bytes: 30000000
consumer.override.fetch.max.wait.ms: 1000
consumer.override.max.partition.fetch.bytes: 2000000
consumer.override.fetch.min.bytes: 5000000
key.converter: io.apicurio.registry.utils.converter.AvroConverter
value.converter: io.apicurio.registry.utils.converter.AvroConverter

Behavior Observed:
Kafka consumer lags increase during idle periods, then drop rapidly (burst consumption).

No errors reported in connector logs.

Verified ClickHouse is reachable and not overloaded.

Tried reducing fetch.min.bytes, max.poll.records, increasing/decreasing parallelism, and switching schema handling.

Please help confirm:

Whether this behavior is expected with the current sink connector.

Any recommended tuning for low-latency streaming workloads?

If this is an area that could be improved for smoother streaming.

Let me know if logs or profiling data are needed. I’d be happy to provide more.

Thanks in advance!