Native sources

Everything on the connector pages describes a process you run. A native source is the same consumer with the process taken away: you save the configuration in your workspace, and Duraton joins the consumer group itself.

Nothing else changes. Same topics, same mapping, same filter, same poison policy, same offsets. The boundary is unchanged too - Duraton is still only ever a consumer, still does not own your partitions, your retention or your rewind. A native source is better ergonomics for the same position, not a different one.

Which one to run

The deciding question is usually the last row of the middle: your broker has to be reachable from Duraton for a native source. A cluster inside a private network with no route out is what the connector is for.

Config parity

Every option the connector accepts is settable on a native source. That is a guarantee with a test behind it, not an intention: the two paths run the same validation over the same property table, and a parity suite walks every forwarded property through both and compares the verdicts. A property the connector accepts and a source refuses is a failing test.

Two narrowings are declared, and they are the only two:

• Direct partition assignment is refused. A stored source runs on every Duraton replica at once, and with no consumer group nothing decides which replica owns which partition - every replica would consume every record. Subscribe with a group.id instead, which is what every stored source does.
• Filesystem paths are refused. A source is a row, and a row has no filesystem, so ssl.ca.location, ssl.certificate.location, ssl.key.location and every .file companion have nowhere to point. Their inline equivalents carry the same material - see credentials below.

Everything else - bootstrap.servers, auto.offset.reset, the SASL settings, the fetch and poll tuning, the decoder, the envelope mode, the mapping, the CEL filter, the poison policy - is written the same way and means the same thing.

Creating one

In the console: Workspace -> Sources -> Kafka -> New source. The form has two halves that mirror the two configuration blocks: the fields Duraton owns (topics, decoder, mapping, filter, poison policy), and the forwarded Kafka properties as key/value pairs, written under Apache Kafka's own names exactly as you would write them in a client.properties.

await duraton.kafka.sources.create({
  app: "shop",
  topics: ["orders"],
  mapping: { eventName: "order.placed" },
  properties: {
    "bootstrap.servers": "broker-1:9092,broker-2:9092",
    "group.id": "duraton-orders",
    "auto.offset.reset": "earliest",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "duraton",
  },
  secrets: { "sasl.password": process.env.KAFKA_PASSWORD! },
  enabled: true,
});

curl -X POST "$DURATON_URL/kafka-sources" \
  -H "Authorization: Bearer $DURATON_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "app": "shop",
    "topics": ["orders"],
    "mapping": { "eventName": "order.placed" },
    "properties": {
      "bootstrap.servers": "broker-1:9092,broker-2:9092",
      "group.id": "duraton-orders",
      "auto.offset.reset": "earliest",
      "security.protocol": "SASL_SSL",
      "sasl.mechanism": "SCRAM-SHA-512",
      "sasl.username": "duraton"
    },
    "secrets": { "sasl.password": "'"$KAFKA_PASSWORD"'" },
    "enabled": true
  }'

Credentials are sealed and write-only

Credentials go in secrets, never in properties. The forwarded property block is stored in plaintext, so a secret-bearing property written there is refused rather than stored - the write fails with the name of the property and where to put it instead.

secrets is keyed by the name you would otherwise have written the credential under:

They are sealed at rest with the same encryption the inbound webhook secrets use, and no read path returns them. Reading a source back gives you its configuration with no credential fields at all - not empty ones. To rotate, send the new value; to change a topic without holding the credential, omit secrets entirely.

For mTLS, the certificate and its key are the inline pair - ssl.certificate.pem in properties (a certificate is public) and ssl.key.pem in secrets. The path spellings are refused because a row has no filesystem to read them from.

The delivery log

A source records what it consumed, so you can see why a record did not become a run without reading broker logs. How much it records is per-source, because the cost scales with your throughput:

all writes a row with the payload for every record, on top of the ingest write. It is meant for sources below roughly a thousand records a second, or for a window while you are debugging one.

Each entry carries the record's coordinates (topic, partition, offset), the broker's own timestamp, what became of it, and the event and run it produced - so you can pivot from a record to the run it caused. Opening one shows the record's key, headers and payload, and its attempt log.

Replay is not a rewind

A replay takes the payload Duraton retained and runs it through the source's current mapping again. It is a Duraton operation on Duraton's copy.

It does not touch Kafka. No offset is moved, nothing is sought, the consumer group is not repositioned. If you want to reprocess a topic from an earlier offset, that is kafka-consumer-groups --reset-offsets against your group id - a Kafka operation, exactly as it is for any other consumer, and exactly as the boundary says.

What replay is good for is the case where the record was fine and the configuration was wrong: a mapping that named the wrong field, a filter that was too broad, a decoder that did not match the framing. Fix the source, replay the records it poisoned, and they become runs. A replay goes through the ordinary ingest path including dedupe, so replaying a record that already ingested inside the dedupe window records a deduped attempt rather than starting a second run.

Filtered and tombstone records are not replayable: a filtered record was excluded deliberately and the broker still holds it, and a tombstone has no payload to re-ingest.

Migrating from the connector

The move is invisible from your side - same topic, same mapping, the consumer just changes address. The one operational subtlety is the consumer group handover, and the order matters.

Reuse the same group.id. That is what makes the native source resume from the offsets the connector already committed, rather than replaying the topic from the start or skipping to its end.

1. Create the source disabled. Save it with enabled: false and the same group.id, bootstrap.servers and mapping the connector used. Nothing consumes yet, and you can check the configuration was accepted before anything moves.
2. Stop the connector. A clean shutdown finishes the record in flight and commits its offset, which is the whole reason to stop it rather than kill it.
3. Confirm it committed. kafka-consumer-groups --describe --group <your-group-id> should show no members and a stable current offset. If members are still listed, the connector has not fully left the group yet - wait rather than proceeding.
4. Enable the source. Duraton joins the same group and resumes from those committed offsets.

Rolling back is the same sequence in reverse: disable the source, confirm the group is empty, start the connector on the same group id.

Next

• Inbound sources - what each transport guarantees, and why per-partition ordering is not an ordering your runs keep.
• Configuration - the two blocks, and what a record becomes. It applies unchanged to a native source.
• Security - the authentication properties, and where each secret comes from.