Quickstart

You have a broker and a workspace and have never run the Kafka connector. This page takes you from an empty file to a run, and the first two commands contact nothing.

You need a broker address, a topic, and a write-scope API key. You do not need the key until step 3.

A minimal configuration

# kafka.yaml
duraton:
  app: billing-ingress
  topics: [payments.events]

kafka:
  bootstrap.servers: kafka-1:9092,kafka-2:9092
  group.id: duraton-billing
  auto.offset.reset: latest

Everything else has a default. Three things are required and are not in the file:

Why the file has two blocks, and what else can go in them, is Configuration.

Trying it, in four steps

1. Check the file. No broker, no Duraton, nothing to undo. Every fault in the file comes back in one pass, and what it prints is every default you did not write:

duraton-kafka validate --config kafka.yaml

2. See what your own records become. This one reads the topic and sends nothing, joins no group and stores no offset, so it is safe against a topic that matters:

duraton-kafka dry-run --config kafka.yaml --records 3

Each line is the request body Duraton would receive. A decision of skip or poison is the mapping or the filter telling you now rather than after the fact.

3. Consume. Same file, plus the three environment variables above and a write-scope API key:

DURATON_URL=https://api.duraton.dev \
DURATON_APP=billing-ingress \
DURATON_API_KEY=dtn_live_... \
duraton-kafka consume --config kafka.yaml --listen :9464

4. Watch a run appear. Produce a record, and each one that passes the filter becomes a run under billing-ingress - visible in your workspace with its step history, and named by whatever mapping.eventName resolved to in step 2:

kafka-console-producer.sh --topic payments.events --bootstrap-server kafka-1:9092
> {"type":"payment.captured","amount":4200}

If nothing arrives, --log-level debug puts one line per record in the connector's log, run id included, which says whether the record was accepted, filtered or never read at all.

Next

• Diagnostics - what steps 1 and 2 print, in full.
• Configuration - naming events from the payload, and the envelope.
• Local stack - a broker and a UI on your laptop, if you have no cluster.
• Deploying - the container image and a Kubernetes shape.