Local stack
A broker, a web UI to watch it, and the connector - in one compose file, with nothing else to install.
A broker, a web UI to watch it, and the connector, with nothing else to install. Redpanda in dev-container mode is ready in about a second and speaks the Kafka protocol, so the configuration below is the same one a production cluster takes.
You need Docker and a Duraton API key. For a real deployment see Deploying.
# compose.yaml
services:
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v26.2.1
# A broker hands every client the address to dial back on, so one listener cannot serve
# both sides: containers must be told redpanda:9092 and your shell localhost:19092.
command:
- redpanda
- start
- --mode dev-container
- --smp 1
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
- --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
- --rpc-addr redpanda:33145
- --advertise-rpc-addr redpanda:33145
ports: ["19092:19092"]
healthcheck:
test: ["CMD", "rpk", "cluster", "health"]
interval: 3s
timeout: 5s
retries: 30
# One-shot: creates the topic so the UI has something to show before anything produces.
topics:
image: docker.redpanda.com/redpandadata/redpanda:v26.2.1
entrypoint: ["/bin/sh", "-c"]
command:
- >-
rpk topic create payments.events -p 3 -X brokers=redpanda:9092 ||
rpk topic describe payments.events -X brokers=redpanda:9092
restart: "no"
depends_on:
redpanda: { condition: service_healthy }
console:
image: docker.redpanda.com/redpandadata/console:v3.9.0
environment:
KAFKA_BROKERS: "redpanda:9092"
ports: ["8080:8080"]
depends_on:
redpanda: { condition: service_healthy }
connector:
image: ghcr.io/duraton/duraton-kafka:0.1.0-beta.2
command:
["consume", "--config", "/etc/duraton/kafka.yaml", "--listen", ":9464", "--log-level", "debug"]
environment:
DURATON_URL: "https://api.duraton.dev"
DURATON_APP: "billing-ingress"
DURATON_API_KEY: "${DURATON_API_KEY:?set DURATON_API_KEY in .env}"
ports: ["9464:9464"]
volumes:
- ./kafka.yaml:/etc/duraton/kafka.yaml:ro
depends_on:
redpanda: { condition: service_healthy }# kafka.yaml, next to it
duraton:
topics: [payments.events]
mapping:
eventName: value.type
kafka:
bootstrap.servers: redpanda:9092
group.id: duraton-billing
# The topic has no history and the connector has no committed offset, so start at the
# beginning rather than skip what you are about to produce.
auto.offset.reset: earliestdocker compose up -d
# produce something to consume
docker compose exec redpanda rpk topic produce payments.events -X brokers=redpanda:9092
{"type":"payment.captured","amount":4200}--log-level debug is what puts the per-record decision in the log, run id included. The Console at
http://localhost:8080 shows the same records and the group's lag beside them. Note that
dev-container mode turns auto topic creation on at the broker while the connector keeps
allow.auto.create.topics: false: a misspelled topic still fails on the connector side, which is the
side you are testing.
Next
- Diagnostics -
dry-runagainst the topic you just produced to. - Configuration - naming events from the payload.
- Deploying - the same image, in Kubernetes.
Diagnostics
validate checks a file and prints every default you did not write; dry-run reads your real records and prints the events they would become. Neither sends anything to Duraton.
Deploying
Run the connector in production: the binary and the container image, how it drains on SIGTERM and the grace period that implies, and a Kubernetes Deployment.