Kafka connector

Local stack

A broker, a web UI to watch it, and the connector - in one compose file, with nothing else to install.

A broker, a web UI to watch it, and the connector, with nothing else to install. Redpanda in dev-container mode is ready in about a second and speaks the Kafka protocol, so the configuration below is the same one a production cluster takes.

You need Docker and a Duraton API key. For a real deployment see Deploying.

# compose.yaml
services:
  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:v26.2.1
    # A broker hands every client the address to dial back on, so one listener cannot serve
    # both sides: containers must be told redpanda:9092 and your shell localhost:19092.
    command:
      - redpanda
      - start
      - --mode dev-container
      - --smp 1
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
      - --rpc-addr redpanda:33145
      - --advertise-rpc-addr redpanda:33145
    ports: ["19092:19092"]
    healthcheck:
      test: ["CMD", "rpk", "cluster", "health"]
      interval: 3s
      timeout: 5s
      retries: 30

  # One-shot: creates the topic so the UI has something to show before anything produces.
  topics:
    image: docker.redpanda.com/redpandadata/redpanda:v26.2.1
    entrypoint: ["/bin/sh", "-c"]
    command:
      - >-
        rpk topic create payments.events -p 3 -X brokers=redpanda:9092 ||
        rpk topic describe payments.events -X brokers=redpanda:9092
    restart: "no"
    depends_on:
      redpanda: { condition: service_healthy }

  console:
    image: docker.redpanda.com/redpandadata/console:v3.9.0
    environment:
      KAFKA_BROKERS: "redpanda:9092"
    ports: ["8080:8080"]
    depends_on:
      redpanda: { condition: service_healthy }

  connector:
    image: ghcr.io/duraton/duraton-kafka:0.1.0-beta.2
    command:
      ["consume", "--config", "/etc/duraton/kafka.yaml", "--listen", ":9464", "--log-level", "debug"]
    environment:
      DURATON_URL: "https://api.duraton.dev"
      DURATON_APP: "billing-ingress"
      DURATON_API_KEY: "${DURATON_API_KEY:?set DURATON_API_KEY in .env}"
    ports: ["9464:9464"]
    volumes:
      - ./kafka.yaml:/etc/duraton/kafka.yaml:ro
    depends_on:
      redpanda: { condition: service_healthy }
# kafka.yaml, next to it
duraton:
  topics: [payments.events]
  mapping:
    eventName: value.type

kafka:
  bootstrap.servers: redpanda:9092
  group.id: duraton-billing
  # The topic has no history and the connector has no committed offset, so start at the
  # beginning rather than skip what you are about to produce.
  auto.offset.reset: earliest
docker compose up -d
# produce something to consume
docker compose exec redpanda rpk topic produce payments.events -X brokers=redpanda:9092
{"type":"payment.captured","amount":4200}

--log-level debug is what puts the per-record decision in the log, run id included. The Console at http://localhost:8080 shows the same records and the group's lag beside them. Note that dev-container mode turns auto topic creation on at the broker while the connector keeps allow.auto.create.topics: false: a misspelled topic still fails on the connector side, which is the side you are testing.

Next

On this page