Kafka connector

What the connector refuses

Three kinds of refusal and the three different fixes they imply, the one documented escape for pasting a working client.properties, and why two names for one property is an error.

The Kafka connector refuses at startup rather than starting with a setting that can never take effect. This page is how to read a rejection. The property lists themselves are in the kafka: reference.

Three refusals, and the difference between them is the difference between three fixes. All of them happen at startup, and every fault in the file is reported in one pass rather than one per restart:

the configuration is not valid:
  - unknown property "sesion.timeout.ms" - did you mean "session.timeout.ms"?
  - property "enable.auto.commit" cannot be set: Offsets would advance on a timer whether or not
    Duraton accepted the record, so a crash would silently drop events. The committed offset is this
    connector's only durability record. Automatic commits are off; enable.auto.commit: false is
    accepted and changes nothing.
  - property "queued.min.messages" is a recognised Kafka property but is not supported: This
    connector has no local prefetch queue to bound. The equivalent memory bound is fetch.max.bytes
    multiplied by the number of brokers.
  - property "auto.offset.reset" is required: there is no safe default - "latest" starts at the end
    of every partition and silently skips the history the topic still retains, while "earliest"
    replays that whole history and starts one run per record

An unrecognised name is an error, not a default. A typo that silently means "use the default" is the worst failure mode a config file has, so the connector refuses it and offers the closest known name. This is stricter than most Kafka tooling, and deliberately so: an unrecognised property here reaches nothing at all, so tolerating it would mean tolerating a setting that can never take effect.

A controlled property is refused with its reason, never silently overridden. There are exactly three, all durability-critical, listed in the reference. A short refusal list is a contract; a long one is a config that lies.

A recognised property the connector cannot honour is refused with the alternative. These are real Kafka properties whose meaning depends on a JVM, or that this client genuinely has no equivalent for. The error names the working substitute where one exists - ssl.truststore.location points you at ssl.ca.location, sasl.jaas.config points you at sasl.username and sasl.password.

The escape

Pasting a working client.properties should be a five-second diagnosis, not an edit-restart loop. So there is one documented escape:

duraton-kafka validate --config kafka.yaml --allow-unknown-kafka-properties
{"time":"...","level":"WARN","msg":"property \"ssl.ca.pem.chain\" is not recognised and will have no effect"}
the configuration is valid; it resolves to:
  ...

It downgrades unrecognised names to a warning naming each one. It never applies to a controlled property, to a recognised-but-unsupported property, or to the duraton: block, where an unknown key stays fatal.

One property, two spellings

Several librdkafka spellings are accepted as alternatives to their Apache Kafka names. Writing both with different values is an error rather than a silent pick:

  - "bootstrap.servers" and "metadata.broker.list" are two names for the same property but are set to
    different values ("b:9092" and "c:9092") - set only one

The full mapping is in the kafka: reference.

Next

On this page