Observability
The health and readiness endpoints, every metric the connector exports, and the one thing to get right - a stall freezes the lag gauge, so alert on the cluster's view of the group instead.
Health, readiness and metrics for the Kafka connector all need --listen. Without it
there is no port, no probe and no metric - the connector runs inside someone else's network, so it
opens a port only when asked.
The short version: alert on the group's lag at the cluster, not on liveness and not on this process's own lag gauge. Why is below.
| Path | Answers | What it actually checks |
|---|---|---|
/healthz | 200 ok | That the process is running. Nothing else, deliberately: see below. |
/readyz | 200 ready, or 503 with one line per failing dependency | Two dependencies, named in the response: kafka (a broker is answering) and duraton (the event API is answering). Checked per request, never cached, and bounded at 5 s. |
/metrics | Prometheus text | Served only when the listener is open. |
A failing readiness response names the side that is down and says nothing about the side that is up:
$ curl -sS -w '%{http_code}\n' http://localhost:9464/readyz
duraton: Get "https://api.duraton.dev/healthz": dial tcp 10.0.0.5:443: connect: connection refused
503The broker was answering, so kafka is not on that list at all.
Metrics
Exposed names are the OpenTelemetry instrument names translated for Prometheus: dots become
underscores and a counter gains _total.
| Series | Type | Unit | Attributes | Read it for |
|---|---|---|---|---|
duraton_kafka_records_polled_total | counter | records | - | Throughput off the broker. Flat while the group's lag at the cluster climbs is a stalled loop, not an idle topic. |
duraton_kafka_records_settled_total | counter | records | duraton_disposition | What became of each record that will not be worked again: forwarded, deduped, filtered, tombstone, poisoned. poisoned is the series to alert on, and it counts a record whatever the poison policy did with it: which policy is configuration, not a property of the record, so a record kept under commit-and-produce settles as poisoned too. |
duraton_kafka_delivery_attempts_total | counter | attempts | duraton_outcome, duraton_status | Attempts by what came back: accepted, suspended, poison, fatal, retry. Counted per attempt, so attempts far above settled records is a retry storm. |
duraton_kafka_delivery_duration_seconds | histogram | seconds | duraton_outcome, duraton_status | Latency of one attempt. Bucket boundaries run 0.005 to 30 seconds, matching the transport timeout, so a stalled attempt lands in a bucket rather than in overflow. |
duraton_kafka_commit_failures_total | counter | commits | - | Offset batches the cluster would not store. Non-zero means a restart would redeliver. |
duraton_kafka_rebalances_blocked_total | counter | rebalances | - | Times the group waited on this member. Climbing alongside a stall means eviction is close. |
duraton_kafka_consumer_lag | gauge | records | messaging_destination_name, messaging_destination_partition_id | Per-partition lag as of the last fetch. Fresh while records are moving, and frozen while the loop is stalled, so this is not the series to alert a stall on: see below. |
duraton_status is omitted entirely when nothing answered, so a transport fault never renders as
status 0.
Lag is read off the fetch responses the loop is already making, so it costs no request to the cluster and is exactly as fresh as the last fetch. That has one consequence worth reading before you write an alert on it.
A stall freezes the gauge. The loop holds one buffered fetch and issues the next only once that
one is taken. A stalled loop - a suspended workspace, a destination refusing every event, a poison
record under onPoison: halt - stops polling and retries the same record instead, so no new fetch
goes out, no new end of log comes back, and the figure sits at its last value while records pile up
behind it. A lag alert on this series will not fire during the stall it was meant to catch.
The cluster's own view of the group is the authority during a stall.
kafka-consumer-groups --describe against your group id compares the committed offset against an end
of log that keeps moving, so it climbs correctly exactly when this gauge cannot. Alert on that for
"the connector has stopped making progress". This gauge remains the right one for "how far behind is
it while running", which is the common case, and duraton_kafka_records_polled_total going flat is
the other in-process sign that the loop has stopped.
A quiet topic is not the same thing, and reports honestly: a fetch answers for every partition it asked about whether or not any records came back, so a partition nobody is producing to keeps reporting its true lag. The staleness comes from the consumer stopping, never from the topic going quiet.
The lag gauge's attributes are OpenTelemetry messaging semantic
conventions rather than
names this connector invented, so a dashboard built for Kafka consumers reads the series without
being taught a new vocabulary. Only what those conventions do not name - a disposition, an outcome, a
status - carries a duraton. prefix.
Liveness does not fail when the loop is stalled
A suspended workspace, or a destination refusing every event, stalls the loop on purpose: the offset is held rather than skipped, so the records are still there and the group's lag at the cluster is the signal. A liveness probe that failed on that would have the orchestrator restarting a connector that is working exactly as designed, and the restart would neither fix the cause nor stop it happening again - it would simply lose the in-flight record and rejoin the group, provoking a rebalance for nothing.
So /healthz answers while the process runs, and a dependency that is down is reported on /readyz,
where it removes the pod from service without killing it. The three signals that tell you a stall is
happening are the group's lag at the cluster climbing, duraton_kafka_records_polled_total flat, and
a warning line every 30 seconds naming the reason:
{"time":"...","level":"WARN","msg":"the workspace is suspended, so nothing is being started: the offset is held rather than skipped, and the group's lag at the cluster is what climbs","topic":"payments.events","partition":3,"offset":91422,"attempts":118,"stalledFor":"1h2m0s"}Note which lag that is. duraton_kafka_consumer_lag is frozen for the duration of a
stall, so the series to alert on here is the group's lag as the cluster reports it, not the
connector's own gauge.
Alert on the group's lag at the cluster, not on liveness.
Next
- Delivery - what causes a stall in the first place.
- Deploying - the probes wired up.
- Behaviour worth knowing - including blocked rebalances.
What the connector refuses
Three kinds of refusal and the three different fixes they imply, the one documented escape for pasting a working client.properties, and why two names for one property is an error.
Behaviour worth knowing
Nine things about the Kafka connector that are easy to be surprised by in production - a required offset reset, whole-name pattern matching, tombstones, the fixed dedupe window, and why a blocked rebalance loses nothing.