Behaviour worth knowing

Nine things about the Kafka connector that are easy to be surprised by, and two that are decided rather than surprising. Read this before it reaches production.

1. auto.offset.reset is required and has no default. Every candidate default loses something real. latest silently skips whatever history the topic still retains. earliest replays that whole history and starts one run per record. The major Kafka clients do not even agree on a default. The one irreversible choice is yours to make explicitly.

2. A topic pattern is matched against the whole topic name. ^orders does not reach orders-eu or orders-dlq. Write ^orders.* if you meant every topic under that prefix. This is the Java consumer's subscribe(Pattern) semantics, and the connector holds that line deliberately: a pattern that reaches further than it reads starts durable runs on topics nobody chose. A pattern without a trailing .* or $ gets a startup warning saying so:

{"time":"2026-07-29T10:00:00Z","level":"WARN","msg":"duraton.topicPattern \"^orders\" is held to the whole topic name, so it does not reach a topic whose name merely starts that way: add \".*\" if you meant every topic under that prefix, or \"$\" to say the whole name is meant and silence this"}

3. A tombstone is routine traffic on a compacted topic, and it is skipped by default. A record with a null value is Kafka's marker that a key was deleted. duraton.onNullValue decides what happens to it: skip (default) sends no event, commits the offset, and counts the skip; deliver sends an envelope whose value is JSON null; poison hands it to the poison policy. Skipping is the only default that is correct by inaction - delivering would start durable runs on null input, and poisoning would dead-letter ordinary traffic the first time you point the connector at a compacted topic. A null value and a zero-length value are different things and are not conflated.

onNullValue: deliver together with envelope: value-only is accepted but warned about: the payload is bare null, and the key and topic that identify which deletion it was are carried only by the wrapped envelope.

That distinction bites the moment you try to test it, because the usual way of producing "nothing" produces an empty value rather than a null one, and the connector will correctly treat that as a record whose value the decoder rejects. Producing a real tombstone takes an explicit flag:

# rpk: -Z, without it a valueless record is produced with an empty-string value
printf '\n' | rpk topic produce orders.events -k ord-1004 -Z -f '%v\n'

# the Apache console producer: declare a marker, then send it as the value
kafka-console-producer.sh --topic orders.events \
  --property parse.key=true --property key.separator=: \
  --property null.marker=NULL
> ord-1004:NULL

A consumer printing the record back shows the difference: a tombstone has no value field at all, while an empty record has "value": "". Both report a value length of zero, so length is not the thing to check.

4. The dedupe window is fixed at 24 hours. The connector configures the dedupe id, not the window. Duraton's event endpoint always deduplicates over 24 hours.

5. tags default to none. It would be tempting to derive them from record headers, and it would be wrong: tag names are constrained and tags are capped at 20 per event, while Kafka header keys are unconstrained strings. A header-derived default would turn any record with 21 headers, or one header named with a /, into a rejected event - which is to say into a poison record. So tags are an explicit map of tag name to path, and a bad tag name fails once at startup rather than on every record.

6. Commit is on accept, and a suspended workspace stalls rather than drops. See Delivery.

7. The record's own timestamp survives only in the wrapped envelope. The event contract has no occurred-at field - event time is assigned when Duraton receives it - so value-only loses the record timestamp entirely.

8. The max.poll.records check is a startup sanity gate, not a guarantee. The connector refuses a batch size that cannot be worked through inside max.poll.interval.ms even on the fast path:

  - max.poll.records 5000 with request.timeout.ms 10000 and retry.backoff.max.ms 1000 needs up to
    34999 ms to work through one poll, which does not fit inside max.poll.interval.ms 20000: lower
    max.poll.records, lower request.timeout.ms, or raise max.poll.interval.ms

It charges one record the slow path (a full request timeout plus the retry ladder) and every other record the fast path. Sustained backpressure retries one record indefinitely by design, so no static estimate can bound the real window. The check refuses a combination that cannot hold; it does not promise that one which passes always will.

9. duraton.assign commits nothing. A directly assigned consumer joins no group, so there is nowhere for an offset to be stored. Every start reads from the offsets written in the file. That is the point of the mode - a fixed, repeatable window over a partition - but it means the delivery contract reads slightly differently: "commit and continue" becomes "continue", and a restart re-reads from the configured offset rather than from where the last run got to. Use a group id for anything that should make progress across restarts.

A rebalance waiting on this member is reported, not obeyed

A rebalance cannot interrupt a batch the connector has already polled. A cooperative rebalance leaves the fetch position of a partition this member keeps exactly where it is, so records that were polled and then dropped would be neither sent nor polled again, and committing across the rebalance is the wrong-ownership commit that blocking it exists to prevent. The member therefore finishes its batch, commits, and only then releases the group.

While the group is waiting, the connector says so - a warning line per notice and duraton_kafka_rebalances_blocked_total climbing. If the stall outlasts max.poll.interval.ms the group evicts the member, its partitions move to another member, and that member resumes from the last committed offset.

Nothing is lost and nothing is skipped. The records between the last commit and the eviction are redelivered to their new owner, and the default {topic}:{partition}:{offset} dedupe id is identical on the redelivery, so Duraton collapses it into the run that already exists. The cost of an eviction is a rebalance and some repeated work, not a gap.

Next

• Delivery - the commit rule these all sit on.
• Observability - what a stall looks like from outside.
• duraton: reference - onNullValue, assign, mapping.tags.