Kafka connector

Security

Reach a secured cluster without putting a secret in the config file: SASL mechanisms, OAUTHBEARER token providers, TLS and mTLS, and where credentials live instead.

How the Kafka connector authenticates to your cluster, and where every secret comes from. All of it lives in the kafka: block except the token provider and the secret files, which are Duraton's own surface.

Every property here is checked at startup, so validate tells you whether the combination holds before anything dials a broker.

SASL

security.protocol decides transport encryption and authentication independently: PLAINTEXT, SSL (TLS only), SASL_PLAINTEXT (authentication only), SASL_SSL (both). Setting a SASL_* protocol requires sasl.mechanism.

sasl.mechanismRequired with it
PLAINsasl.username, plus a password from the environment or a file
SCRAM-SHA-256same
SCRAM-SHA-512same
OAUTHBEARERa token provider, below

Exactly one mechanism is installed, so a mismatch with the broker is an authentication failure rather than a silent downgrade to something weaker.

GSSAPI and AWS_MSK_IAM are refused, each with its own sentence rather than a bare list. For IAM in particular, the answer is not a mechanism at all:

  - AWS_MSK_IAM is a Java login module and cannot be used outside a JVM. Authenticate to a cluster
    using IAM with security.protocol: SASL_SSL, sasl.mechanism: OAUTHBEARER and the AWS token
    provider, which is declared in this connector's contract but not built in this release.
  - security.protocol SASL_SSL requires sasl.mechanism

The second bullet always accompanies the first: refusing the mechanism leaves sasl.mechanism unresolved, and every fault in a configuration is reported in one pass rather than one per run.

OAUTHBEARER tokens

A Kafka client cannot fetch a token on its own - upstream expresses token acquisition only as a Java class name - so the connector supplies one, selected by duraton.sasl.tokenProvider:

ProviderBehaviour
oidcClient-credentials grant against sasl.oauthbearer.token.endpoint.url, with sasl.oauthbearer.client.credentials.client.id and its secret, and optional sasl.oauthbearer.scope. The token is cached until 30 seconds before it expires. This is the default when a token endpoint URL is set.
staticThe token is read from KAFKA_OAUTH_TOKEN, or from the file at duraton.sasl.tokenFile, on every SASL session - so a rotated token takes effect without a restart.

sasl.oauthbearer.extensions takes a comma-separated list of key=value pairs.

TLS and mTLS

PropertyDefaultMeaning
ssl.ca.locationsystem trust storePath to a PEM certificate authority.
ssl.ca.pem-The authority inline as PEM. Mutually exclusive with ssl.ca.location.
ssl.certificate.location-Client certificate for mTLS. Requires ssl.key.location.
ssl.certificate.pem-The client certificate inline as PEM. Requires ssl.key.pem. Mutually exclusive with ssl.certificate.location.
ssl.key.location-Its private key. Requires ssl.certificate.location.
ssl.key.pem-The private key inline as PEM. A secret: supply it in KAFKA_SSL_KEY_PEM, never in the file. Requires ssl.certificate.pem. Mutually exclusive with ssl.key.location.
enable.ssl.certificate.verificationtrueSetting it false turns broker certificate checking off.
ssl.endpoint.identification.algorithmhttpsnone (or the empty string) turns host checking off.
ssl.protocolTLSv1.2Lowest version negotiated. TLSv1.3 also accepted.

mTLS is the certificate and its key together; setting one without the other is an error, so mutual TLS is never half-applied:

  - ssl.certificate.location requires ssl.key.location: a client certificate and its key are set
    together

The pair is given either as two paths or as two inline PEMs, never as a mixture. Inline is what a deployment with no mounted secret volume uses: the certificate goes in the configuration and the key in KAFKA_SSL_KEY_PEM.

Server-name indication needs no configuration: the broker host being dialled is used automatically.

Either spelling that disables verification - enable.ssl.certificate.verification: false or ssl.endpoint.identification.algorithm: none - is accepted, because test clusters need it, and each emits a startup warning naming what it costs: the broker's certificate is no longer checked and the connection can be intercepted. Neither is ever a default.

Encrypted private keys are not supported. Use an unencrypted key file protected by filesystem permissions.

Where secrets come from

A secret is never written in the configuration file. Each one comes from an environment variable or from a file whose path you name:

SecretEnvironment variableFile
Duraton API keyDURATON_API_KEYduraton.keyFile
SASL passwordKAFKA_SASL_PASSWORDsasl.password.file
OAuth client secretKAFKA_OAUTH_CLIENT_SECRETsasl.oauthbearer.client.credentials.client.secret.file
OAUTHBEARER token (static)KAFKA_OAUTH_TOKENduraton.sasl.tokenFile
Schema Registry credentialKAFKA_SCHEMA_REGISTRY_AUTHduraton.schemaRegistry.authFile
TLS private keyKAFKA_SSL_KEY_PEMssl.key.location

The file companion is the secret's own name with .file appended, except for the TLS private key: a key on disk is already ssl.key.location, so there is no ssl.key.pem.file.

Writing sasl.password, sasl.oauthbearer.client.credentials.client.secret or duraton.schemaRegistry.auth into the file is an error naming both sources:

  - sasl.password must never carry the password itself. Supply it in KAFKA_SASL_PASSWORD, or point
    sasl.password.file at a file holding it.

A secret file may end with one trailing newline, so a value written by a shell redirect and the same value written by a secret manager are treated identically. No secret is logged at any level, and the resolved-configuration listing prints <redacted> in place of the value.

Next

On this page