Confluent config keys
The confluent-kafka shim reads a confluent-kafka config dict.
Each key it knows is in the table below, with what it does on a Producer and on a
Consumer. A key outside the table raises KafkaException (_INVALID_ARG) with
confluent’s text, No such configuration property: "key", when the client is built.
- Translated: the native client has the setting and the value is passed on.
- Alias: librdkafka’s other spelling of a key; it is read as that key, and the later of two spellings in the dict wins.
- Shim: implemented in the shim’s Python.
- Fixed: the native client behaves one way. The listed value is accepted silently. Any other value is refused or ignored with a warning, as the row says.
- Ignored: accepted, changes nothing, and logs one warning per process. The warning
goes to the config’s
loggerwhen it has one, else to thepyrula.kafka.compat.confluentlogger. - Refused: raises
KafkaException(_INVALID_ARG) because ignoring it would change data, placement, durability or security, or the feature is absent. - Refused (as librdkafka): confluent refuses the key itself, with the same text.
- pyrula only: accepted by the shim, refused by confluent.
- Other client: librdkafka accepts a producer key on a consumer, and the reverse, with a warning and no effect. The shim does the same, so one dict can configure both.
- Unknown: not a key of that client.
Booleans and numbers are read as librdkafka reads them: true, t, 1, false, f,
0 for a boolean, and a number or a string holding one for a number.
The shim does not check a value’s range or the relation between keys (the range of
session.timeout.ms, max.poll.interval.ms >= session.timeout.ms, message.max.bytes):
librdkafka refuses those at construction, and the native client sees them later, or not.
request.timeout.ms is honoured on a consumer, unlike librdkafka, where it is a producer
key: it bounds a broker request and the waits before the first connection.
| Key | Producer | Consumer | What pyrula does |
|---|---|---|---|
acks | Translated | Other client | |
allow.auto.create.topics | Fixed at true; other values ignored | Translated | Consumer: false, librdkafka’s consumer default, never creates a missing topic; true lets the broker’s auto.create.topics.enable decide for a manual assign(), while subscribe() never creates one. Producer: the native producer’s metadata requests always let the broker auto-create a topic, so false is not honoured: a topic the producer names is created when the broker’s auto.create.topics.enable allows it. |
api.version.fallback.ms | Ignored | Ignored | Only used with brokers older than 0.10. |
api.version.request | Fixed at true; other values ignored | Fixed at true; other values ignored | API versions are always negotiated. |
api.version.request.timeout.ms | Ignored | Ignored | Only used with brokers older than 0.10. |
auto.commit.enable | Other client | Alias of enable.auto.commit | |
auto.commit.interval.ms | Other client | Translated | |
auto.offset.reset | Other client | Translated | Values: earliest, smallest, beginning, latest, largest, end, none, error. |
background_event_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “background_event_cb” must be set through dedicated ..set..() function. |
batch.num.messages | Translated | Other client | |
batch.size | Translated | Other client | Capped at message.max.bytes, as librdkafka caps it, rather than refused. |
bootstrap.servers | Translated | Translated | At least one broker is required: the shim cannot connect to none. |
broker.address.family | Fixed at any; other values ignored | Fixed at any; other values ignored | Addresses of both families are tried. |
broker.address.ttl | Ignored | Ignored | The native client manages its own connections and retries. |
broker.version.fallback | Ignored | Ignored | Only used with brokers older than 0.10. |
builtin.features | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
check.crcs | Other client | Fixed (accepts true or false) | Record CRCs are always verified. |
client.dns.lookup | Fixed at use_all_dns_ips; other values ignored | Fixed at use_all_dns_ips; other values ignored | Every address a name resolves to is tried. |
client.id | Translated | Translated | |
client.rack | Ignored | Ignored | The consumer always fetches from the partition leader (no KIP-392 follower fetching). |
client.software.name | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
client.software.version | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
closesocket_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “closesocket_cb” must be set through dedicated ..set..() function. |
compression.codec | Alias of compression.type | Other client | |
compression.level | Ignored | Other client | Throughput tuning with no native counterpart; delivery is unchanged. |
compression.type | Translated | Other client | |
connect_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “connect_cb” must be set through dedicated ..set..() function. |
connections.max.idle.ms | Ignored | Ignored | The native client manages its own connections and retries. |
consume.callback.max.messages | Other client | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
consume_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “consume_cb” must be set through dedicated ..set..() function. |
coordinator.query.interval.ms | Other client | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
debug | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
default.topic.config | Shim | Shim | A dict flattened into the config; its entries are applied first, so a top-level key wins. |
default_topic_conf | Refused (as librdkafka) | Refused (as librdkafka) | Property “default_topic_conf” must be set through dedicated ..set..() function. |
delivery.report.only.error | Shim | Other client | Successful reports are taken off the queue without calling their callback or counting in poll()’s return; librdkafka never queues them. |
delivery.timeout.ms | Alias of message.timeout.ms | Other client | |
dr_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “dr_cb” must be set through dedicated ..set..() function. |
dr_msg_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “dr_msg_cb” must be set through dedicated ..set..() function. |
enable.auto.commit | Other client | Translated | |
enable.auto.offset.store | Other client | Shim | The shim stores each delivered offset itself; with it off only store_offsets() moves the stored offset. |
enable.gapless.guarantee | Fixed at false; other values refused | Other client | Gap-less ordering is not supported. |
enable.idempotence | Translated | Other client | |
enable.metrics.push | Ignored | Ignored | Client telemetry (KIP-714) is not sent. |
enable.partition.eof | Other client | Shim | With true, poll() and consume() return a message whose error().code() is _PARTITION_EOF when the fetch position reaches the end of a partition, once per partition until it is sought, assigned or grows. |
enable.random.seed | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
enable.sasl.oauthbearer.unsecure.jwt | Fixed at false; other values refused | Fixed at false; other values refused | There is no unsecured-JWT builder. |
enable.sparse.connections | Ignored | Ignored | The native client manages its own connections and retries. |
enable.ssl.certificate.verification | Fixed at true; other values refused | Fixed at true; other values refused | Certificates are always verified. |
enable_sasl_queue | Ignored | Ignored | The native client’s SASL handling has no separate queue. |
enabled_events | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
error_cb | Translated | Translated | Called from poll() on the application’s thread, as librdkafka calls it. |
fetch.error.backoff.ms | Other client | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
fetch.max.bytes | Other client | Translated | |
fetch.message.max.bytes | Other client | Alias of max.partition.fetch.bytes | |
fetch.min.bytes | Other client | Translated | |
fetch.queue.backoff.ms | Other client | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
fetch.wait.max.ms | Other client | Translated | |
group.id | Other client | Translated | |
group.instance.id | Other client | Translated | |
group.protocol | Other client | Fixed at classic; other values refused | The KIP-848 consumer protocol is not supported. |
group.protocol.type | Other client | Ignored | Used only by the KIP-848 consumer group protocol, which is refused. |
group.remote.assignor | Other client | Ignored | Used only by the KIP-848 consumer group protocol, which is refused. |
heartbeat.interval.ms | Other client | Translated | |
https.ca.location | Ignored | Ignored | Used only by librdkafka’s built-in OIDC token request; under sasl.oauthbearer.method=oidc it is refused, since the token request verifies against the system trust store. |
https.ca.pem | Ignored | Ignored | Used only by librdkafka’s built-in OIDC token request; under sasl.oauthbearer.method=oidc it is refused, since the token request verifies against the system trust store. |
interceptors | Refused (as librdkafka) | Refused (as librdkafka) | Internal property “interceptors” not settable. |
internal.termination.signal | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
isolation.level | Other client | Translated | |
linger.ms | Translated | Other client | |
log.connection.close | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
log.queue | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
log.thread.name | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
log_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “log_cb” must be set through dedicated ..set..() function. |
log_level | Ignored | Ignored | A diagnostic; pass logger= and set its level for the native client’s logs. |
logger | Translated | Translated | Also where this page’s warnings go. |
max.in.flight | Alias of max.in.flight.requests.per.connection | Alias of max.in.flight.requests.per.connection | |
max.in.flight.requests.per.connection | Translated | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
max.partition.fetch.bytes | Other client | Translated | |
max.poll.interval.ms | Other client | Translated | |
max.poll.records | Other client | Translated | The records one native poll fetches; poll() still hands them out one at a time. |
message.copy.max.bytes | Ignored | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
message.max.bytes | Translated | Ignored | Producer: caps a produce request and any one record; a record over it is refused at produce(). Consumer: throughput tuning with no native counterpart; delivery is unchanged. |
message.send.max.retries | Translated | Other client | |
message.timeout.ms | Translated | Other client | |
metadata.broker.list | Alias of bootstrap.servers | Alias of bootstrap.servers | |
metadata.max.age.ms | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
metadata.recovery.rebootstrap.trigger.ms | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
metadata.recovery.strategy | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
metadata.request.timeout.ms | Ignored | Ignored | The native client manages its own connections and retries. |
msg_order_cmp | Refused (as librdkafka) | Refused (as librdkafka) | Property “msg_order_cmp” must be set through dedicated ..set..() function. |
oauth_cb | Shim | Shim | Called as oauth_cb(sasl.oauthbearer.config), returning (token, expiry[, principal[, extensions]]). |
oauthbearer_token_refresh_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “oauthbearer_token_refresh_cb” must be set through dedicated ..set..() function. |
offset.store.method | Other client | Fixed at broker; other values refused | The file offset store is not supported. |
offset.store.path | Other client | Ignored | Used only by the file offset store, which is refused. |
offset.store.sync.interval.ms | Other client | Ignored | Used only by the file offset store, which is refused. |
offset_commit_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “offset_commit_cb” must be set through dedicated ..set..() function. |
on_commit | Unknown | Shim | Called as on_commit(err, partitions) from poll(), consume() and close() for an asynchronous commit() and for each automatic commit. A commit(asynchronous=False) never calls it. |
on_delivery | Shim | Unknown | The default delivery callback; a callback= or on_delivery= given to produce() overrides it. |
opaque | Refused (as librdkafka) | Refused (as librdkafka) | Property “opaque” must be set through dedicated ..set..() function. |
open_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “open_cb” must be set through dedicated ..set..() function. |
partition.assignment.strategy | Other client | Translated | |
partitioner | Translated | Other client | murmur2_random and consistent_random route like murmur2 and consistent; they differ only for records with no key, which the native client spreads itself. Values: murmur2, murmur2_random, consistent, consistent_random; refused: fnv1a, fnv1a_random, random. |
partitioner_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “partitioner_cb” must be set through dedicated ..set..() function. |
plugin.library.paths | Fixed at empty; other values ignored | Fixed at empty; other values ignored | Librdkafka plugins are not loaded (the usual one is the Control Center monitoring interceptor). |
produce.offset.report | Ignored | Other client | A deprecated no-op in librdkafka too. |
queue.buffering.backpressure.threshold | Ignored | Other client | Throughput tuning with no native counterpart; delivery is unchanged. |
queue.buffering.max.kbytes | Translated | Other client | |
queue.buffering.max.messages | Translated | Other client | |
queue.buffering.max.ms | Alias of linger.ms | Other client | |
queued.max.messages.kbytes | Other client | Translated | Kibibytes in, bytes out. |
queued.min.messages | Other client | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
queuing.strategy | Fixed at fifo; other values refused | Other client | lifo would reorder records. |
rebalance.timeout.ms | Unknown | pyrula only | A pyrula extension; confluent refuses it. Sent in every JoinGroup. |
rebalance_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “rebalance_cb” must be set through dedicated ..set..() function. |
receive.message.max.bytes | Ignored | Ignored | Throughput tuning with no native counterpart; delivery is unchanged. |
reconnect.backoff.jitter.ms | Ignored | Ignored | The native client manages its own connections and retries. |
reconnect.backoff.max.ms | Ignored | Ignored | The native client manages its own connections and retries. |
reconnect.backoff.ms | Ignored | Ignored | The native client manages its own connections and retries. |
request.required.acks | Alias of acks | Other client | |
request.timeout.ms | Translated | Translated | In librdkafka it is producer-only and a consumer ignores it; the shim honours it as the bound of one broker request, and of the waits before the connection is up. |
resolve_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “resolve_cb” must be set through dedicated ..set..() function. |
retries | Alias of message.send.max.retries | Other client | |
retry.backoff.max.ms | Ignored | Ignored | The native client manages its own connections and retries. |
retry.backoff.ms | Translated | Ignored | The native client manages its own connections and retries. |
sasl.jaas.config | Refused (as librdkafka) | Refused (as librdkafka) | Java JAAS configuration is not supported, see https://github.com/confluentinc/librdkafka/wiki/Using-SASL-with-librdkafka for more information. |
sasl.kerberos.keytab | Ignored | Ignored | GSSAPI only, and GSSAPI is refused as the SASL mechanism. |
sasl.kerberos.kinit.cmd | Ignored | Ignored | GSSAPI only, and GSSAPI is refused as the SASL mechanism. |
sasl.kerberos.min.time.before.relogin | Ignored | Ignored | GSSAPI only, and GSSAPI is refused as the SASL mechanism. |
sasl.kerberos.principal | Ignored | Ignored | GSSAPI only, and GSSAPI is refused as the SASL mechanism. |
sasl.kerberos.service.name | Ignored | Ignored | GSSAPI only, and GSSAPI is refused as the SASL mechanism. |
sasl.mechanism | Translated | Translated | A mechanism the native client lacks (GSSAPI) is refused when the client is built. |
sasl.mechanisms | Alias of sasl.mechanism | Alias of sasl.mechanism | |
sasl.oauthbearer.assertion.algorithm | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.aud | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.exp.seconds | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.iss | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.jti.include | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.nbf.seconds | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.claim.sub | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.file | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.jwt.template.file | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.private.key.file | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.private.key.passphrase | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.assertion.private.key.pem | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.client.credentials.client.id | Alias of sasl.oauthbearer.client.id | Alias of sasl.oauthbearer.client.id | |
sasl.oauthbearer.client.credentials.client.secret | Alias of sasl.oauthbearer.client.secret | Alias of sasl.oauthbearer.client.secret | |
sasl.oauthbearer.client.id | Shim | Shim | Used under sasl.oauthbearer.method=oidc; inert, with a warning, under default. |
sasl.oauthbearer.client.secret | Shim | Shim | Used under sasl.oauthbearer.method=oidc; inert, with a warning, under default. |
sasl.oauthbearer.config | Shim | Shim | Passed to oauth_cb as its argument. Without oauth_cb and without sasl.oauthbearer.method=oidc it is refused: there is no built-in unsecured-JWT retriever. |
sasl.oauthbearer.extensions | Shim | Shim | Used under sasl.oauthbearer.method=oidc; inert, with a warning, under default. |
sasl.oauthbearer.grant.type | Fixed at client_credentials; other values refused | Fixed at client_credentials; other values refused | The JWT-bearer grant is not supported. |
sasl.oauthbearer.metadata.authentication.type | Fixed at none; other values refused | Fixed at none; other values refused | Azure IMDS and AWS IAM token retrieval are not supported. |
sasl.oauthbearer.method | Shim | Shim | default uses oauth_cb. oidc builds a client-credentials token provider from the sasl.oauthbearer.client.*, .token.endpoint.url, .scope and .extensions keys when oauth_cb is absent. Values: default, oidc. |
sasl.oauthbearer.scope | Shim | Shim | Used under sasl.oauthbearer.method=oidc; inert, with a warning, under default. |
sasl.oauthbearer.sub.claim.name | Ignored | Ignored | Used only by the JWT-bearer grant, which is refused. |
sasl.oauthbearer.token.endpoint.url | Shim | Shim | Used under sasl.oauthbearer.method=oidc; inert, with a warning, under default. |
sasl.password | Translated | Translated | |
sasl.username | Translated | Translated | |
security.protocol | Translated | Translated | |
session.timeout.ms | Other client | Translated | |
share.acknowledgement.mode | Other client | Ignored | Used only by share consumers. |
socket.blocking.max.ms | Ignored | Ignored | The native client manages its own connections and retries. |
socket.connection.setup.timeout.ms | Ignored | Ignored | The native client manages its own connections and retries. |
socket.keepalive.enable | Ignored | Ignored | The native client manages its own connections and retries. |
socket.max.fails | Ignored | Ignored | The native client manages its own connections and retries. |
socket.nagle.disable | Ignored | Ignored | The native client manages its own connections and retries. |
socket.receive.buffer.bytes | Ignored | Ignored | The native client manages its own connections and retries. |
socket.send.buffer.bytes | Ignored | Ignored | The native client manages its own connections and retries. |
socket.timeout.ms | Ignored | Translated | Consumer: bounds one broker request unless request.timeout.ms is given, as librdkafka bounds a consumer’s. Producer: a produce request is bounded by request.timeout.ms, as in librdkafka. |
socket_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “socket_cb” must be set through dedicated ..set..() function. |
ssl.ca.certificate.stores | Refused | Refused | Windows certificate stores are not read. |
ssl.ca.location | Translated | Translated | The value probe means the system trust store, which an unset native field uses. |
ssl.ca.pem | Refused | Refused | Inline PEM is not supported; write it to a file and use the ssl.*.location key. |
ssl.certificate.location | Translated | Translated | |
ssl.certificate.pem | Refused | Refused | Inline PEM is not supported; write it to a file and use the ssl.*.location key. |
ssl.certificate.verify_cb | Refused (as librdkafka) | Refused (as librdkafka) | Property “ssl.certificate.verify_cb” must be set through dedicated ..set..() function. |
ssl.cipher.suites | Refused | Refused | The native TLS stack’s own parameter set is used; a policy cannot be narrowed. |
ssl.crl.location | Refused | Refused | CRL checks are not supported. |
ssl.curves.list | Refused | Refused | The native TLS stack’s own parameter set is used; a policy cannot be narrowed. |
ssl.endpoint.identification.algorithm | Fixed at https; other values refused | Fixed at https; other values refused | The native TLS setup always verifies the broker’s hostname against its certificate, librdkafka’s https; none is not supported. |
ssl.engine.id | Refused | Refused | OpenSSL providers and engines are not available. |
ssl.engine.location | Refused | Refused | OpenSSL providers and engines are not available. |
ssl.key.location | Translated | Translated | |
ssl.key.password | Refused | Refused | Encrypted private keys are not supported; decrypt the key file. |
ssl.key.pem | Refused | Refused | Inline PEM is not supported; write it to a file and use the ssl.*.location key. |
ssl.keystore.location | Refused | Refused | PKCS#12 keystores are not supported; use PEM files. |
ssl.keystore.password | Refused | Refused | PKCS#12 keystores are not supported; use PEM files. |
ssl.providers | Refused | Refused | OpenSSL providers and engines are not available. |
ssl.sigalgs.list | Refused | Refused | The native TLS stack’s own parameter set is used; a policy cannot be narrowed. |
ssl.truststore.location | Refused (as librdkafka) | Refused (as librdkafka) | Java TrustStores are not supported, use ssl.ca.location and a certificate file instead. See https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka for more information. |
ssl_ca | Refused (as librdkafka) | Refused (as librdkafka) | Internal property “ssl_ca” not settable. |
ssl_certificate | Refused (as librdkafka) | Refused (as librdkafka) | Internal property “ssl_certificate” not settable. |
ssl_engine_callback_data | Refused (as librdkafka) | Refused (as librdkafka) | Property “ssl_engine_callback_data” must be set through dedicated ..set..() function. |
ssl_key | Refused (as librdkafka) | Refused (as librdkafka) | Internal property “ssl_key” not settable. |
statistics.interval.ms | Translated | Translated | Needs stats_cb; the callback gets librdkafka-style JSON from poll(). |
stats_cb | Translated | Translated | Called from poll() with a JSON string. |
sticky.partitioning.linger.ms | Translated | Other client | |
test.mock.broker.rtt | Refused | Refused | Librdkafka’s mock cluster is not available; run a broker. |
test.mock.num.brokers | Refused | Refused | Librdkafka’s mock cluster is not available; run a broker. |
throttle_cb | Ignored | Ignored | The native client does not report broker throttling; it is never called. |
topic.blacklist | Ignored | Refused | Producer: a producer reads no subscription. Consumer: excluding topics from a subscription is not supported. |
topic.metadata.propagation.max.ms | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
topic.metadata.refresh.fast.cnt | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
topic.metadata.refresh.fast.interval.ms | Ignored | Translated | The native metadata cache has its own refresh cadence. |
topic.metadata.refresh.interval.ms | Translated | Translated | |
topic.metadata.refresh.sparse | Ignored | Ignored | The native metadata cache has its own refresh cadence. |
topics | Unknown | pyrula only | A pyrula extension; confluent refuses it. The topics a consumer subscribes to or reads from. |
transaction.timeout.ms | Fixed at 60000; other values refused | Other client | The native producer’s transaction timeout is fixed at librdkafka’s default. |
transactional.id | Translated | Other client |