Delivery

This is the durability contract of the Kafka connector: when an offset moves, and what happens to a record Duraton will not take. It is the page to read before deciding that a record was lost.

Commit on accept, not on completion

The connector commits a record's offset once Duraton has accepted the event, not when the workflow it started finishes.

  poll record --> POST /events --> Duraton records the run --> 202 accepted --> COMMIT
                                                                                  |
                        the workflow runs on from here, for a second or an hour ---+
                                     (the offset has already moved)

That is deliberate and it is not a weaker guarantee. Acceptance is the durability boundary: the run exists and is recorded before the response is returned, so committing there never advances past non-durable work. Waiting for completion instead would mean a workflow containing a one-hour sleep or a human approval holds its partition for as long as that takes, and a partition held is lag for every other record behind it.

Delivery is at-least-once, and the default dedupe id - the topic:partition:offset triple - is stable across any redelivery, which is what collapses at-least-once into effectively-once through Duraton's own deduplication.

A record the destination will not take

  any other 4xx --> poison --> onPoison?
                                  |
                                  +-- commit-and-log (default) --> log, commit, continue
                                  +-- commit-and-produce --------> write to poisonTopic, commit, continue
                                  +-- halt ----------------------> stop consuming, exit non-zero

Under commit-and-produce the record is written with its original key, value and headers, plus duraton.source.topic, duraton.source.partition, duraton.source.offset, duraton.source.timestamp, duraton.error, and duraton.status when the destination answered at all. The timestamp is there because the written record is a new record carrying its own: without it a drain consumer cannot recover when the original was produced.

The write is acknowledged before the offset is committed, so a crash between the two redelivers the record rather than losing it. If the write cannot land the connector retries it on the same backoff as a refused delivery, indefinitely, and commits nothing: giving up would discard the record you explicitly asked to keep. Lag climbing on a poison topic that has stopped accepting writes is the intended signal.

Startup refuses a poison topic the connector is also consuming, whether it is named in duraton.topics, keyed in duraton.assign, or matched by duraton.topicPattern. Listing it in duraton.excludeTopics is the escape if a pattern is broad enough to catch it.

Backoff

A retried record is worked again on an exponential backoff. The window doubles from duraton.retry.min and is capped at duraton.retry.max, and each wait is picked at random inside that window, so a fleet that stalled on the same destination does not come back to it in step. A wait is never shorter than duraton.retry.min: a destination that has just refused is the last thing worth retrying immediately. That same window also governs the wait after a failed fetch from the cluster, so it is one knob for how hard this process hammers something that is not answering, whichever side is not answering.

Setting neither key keeps those two defaults. A min longer than max is refused at start-up with a message naming both keys and both values, and so is a value of zero or less for either. That is a start-up failure rather than a silent reordering, because an operator who inverted them meant something by it.

There is no attempt limit, and deliberately no key for one. Giving up on a record would mean skipping work the destination never actually refused, which loses data nobody asked to lose - so a stall ends when the destination changes its mind, or when the connector is shut down, and never because a counter ran out. What tells you it is happening is the group's lag at the cluster, plus the stall warning in the log.

Next

• Observability - which series actually shows a stall.
• Filtering - dropping a record before it reaches any of this.
• Behaviour worth knowing - including why duraton.assign commits nothing.
• duraton: reference - onPoison, retry.min, retry.max.