Deploying

Run the connector in production: the binary and the container image, how it drains on SIGTERM and the grace period that implies, and a Kubernetes Deployment.

The Kafka connector is one stateless process: the offset in Kafka is the whole of its durable state, so a plain Deployment is the right shape and there is nothing to back up.

For a broker on your laptop instead, see Local stack.

The artifacts

The connector is a single static binary, duraton-kafka, with no runtime library dependencies, and a container image built on a distroless nonroot base whose entrypoint is that binary. Both take the same arguments.

Artifact	Name	Notes
Image	`ghcr.io/duraton/duraton-kafka:<version>`	`linux/amd64` and `linux/arm64` in one index. Published by tagging `kafka/v<version>`, on its own cadence rather than riding an SDK release.
Moving tag	`ghcr.io/duraton/duraton-kafka:latest`	Moved only by a non-prerelease tag, so tracking it never hands you a preview build.

The current release is the prerelease 0.1.0-beta.2. Because the moving tag advances only on a non-prerelease version, :latest does not exist yet - pin the explicit tag until the first 1.0.0.

docker run --rm \
  -v "$PWD/kafka.yaml:/etc/duraton/kafka.yaml:ro" \
  -e DURATON_URL -e DURATON_APP -e DURATON_API_KEY \
  -p 9464:9464 \
  ghcr.io/duraton/duraton-kafka:0.1.0-beta.2 \
  consume --config /etc/duraton/kafka.yaml --listen :9464

Swap consume for validate and the same command checks the file without dialling anything.

Stopping

SIGTERM and SIGINT both mean drain. Polling stops; the record already in flight is finished and its offset stored, so the partition is handed over at a known position rather than at a guess:

  SIGTERM
     |
     +--> stop polling
     +--> finish the record in flight        (bounded at 45s)
     +--> store the offsets not yet stored   (bounded at 45s)
     +--> drain in-flight metric scrapes     (bounded at 5s)
     +--> exit 0

The two 45-second bounds run on a clock that shutting down does not cancel, which is the point: a request already sent finishes and commits rather than being redelivered after a restart. Worst case is therefore 95 seconds, which is where the grace period below comes from.

Kubernetes

One consumer group member per replica. Nothing is stateful, so a plain Deployment is the right shape; scale it to at most the partition count, because a member beyond that is assigned nothing.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: duraton-kafka
spec:
  replicas: 3
  selector:
    matchLabels: { app: duraton-kafka }
  template:
    metadata:
      labels: { app: duraton-kafka }
    spec:
      # 45s for the delivery in flight, 45s for the final commit, 5s to drain scrapes.
      terminationGracePeriodSeconds: 120
      containers:
        - name: connector
          image: ghcr.io/duraton/duraton-kafka:0.1.0-beta.2
          args: ["consume", "--config", "/etc/duraton/kafka.yaml", "--listen", ":9464"]
          ports:
            - { name: ops, containerPort: 9464 }
          env:
            - name: DURATON_URL
              value: https://api.duraton.dev
            - name: DURATON_APP
              value: billing-ingress
            - name: DURATON_API_KEY
              valueFrom:
                secretKeyRef: { name: duraton-kafka, key: apiKey }
            - name: KAFKA_SASL_PASSWORD
              valueFrom:
                secretKeyRef: { name: duraton-kafka, key: saslPassword }
          # Liveness is not lag: a stalled loop must not be restarted.
          livenessProbe:
            httpGet: { path: /healthz, port: ops }
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /readyz, port: ops }
            periodSeconds: 10
            timeoutSeconds: 6
          volumeMounts:
            - { name: config, mountPath: /etc/duraton, readOnly: true }
      volumes:
        - name: config
          configMap: { name: duraton-kafka }

timeoutSeconds: 6 clears the readiness check's own 5-second ceiling, so a slow dependency is reported as unready with its reason rather than as a probe that timed out with none.

Why liveness must not fail on a stall is Observability.

Static membership

Every restart of a dynamic member costs the group a rebalance. group.instance.id makes a member static: the group holds its partitions for session.timeout.ms while it restarts, so a rolling update moves no partitions at all.

The value has to be distinct per replica, and the configuration file is read literally - there is no environment interpolation in it. Kubernetes expands $(VAR) in args (docs), so a StatefulSet can select one file per ordinal out of a single ConfigMap:

# Only what differs from the Deployment above; the env, probes, volume and grace period
# are unchanged.
apiVersion: apps/v1
kind: StatefulSet
spec:
  replicas: 3
  serviceName: duraton-kafka
  template:
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: connector
          image: ghcr.io/duraton/duraton-kafka:0.1.0-beta.2
          # POD_NAME is duraton-kafka-0, -1, -2; the ConfigMap holds one key per name.
          args: ["consume", "--config", "/etc/duraton/$(POD_NAME).yaml", "--listen", ":9464"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef: { fieldPath: metadata.name }

apiVersion: v1
kind: ConfigMap
metadata:
  name: duraton-kafka
data:
  duraton-kafka-0.yaml: |
    duraton: { topics: [payments.events] }
    kafka:
      bootstrap.servers: kafka-1:9092,kafka-2:9092
      group.id: duraton-billing
      group.instance.id: duraton-billing-0
      auto.offset.reset: latest
  # duraton-kafka-1.yaml and -2.yaml differ in group.instance.id and nothing else.

If that is more machinery than the churn is worth, leave group.instance.id out. The default cooperative-sticky assignment already keeps a rebalance from stopping the partitions that are not moving.

Observability - what --listen opens, and what to alert on.
Security - where the SASL password in that Secret comes from.
Behaviour worth knowing - before this reaches production.

The artifacts

Stopping

Kubernetes

Static membership

Next

On this page