Skip to content
Pyrula
Pre-release: the packages are not on PyPI yet, so pip install does not resolve. Build from source to try it today.

Confluent config keys

The confluent-kafka shim reads a confluent-kafka config dict. Each key it knows is in the table below, with what it does on a Producer and on a Consumer. A key outside the table raises KafkaException (_INVALID_ARG) with confluent’s text, No such configuration property: "key", when the client is built.

  • Translated: the native client has the setting and the value is passed on.
  • Alias: librdkafka’s other spelling of a key; it is read as that key, and the later of two spellings in the dict wins.
  • Shim: implemented in the shim’s Python.
  • Fixed: the native client behaves one way. The listed value is accepted silently. Any other value is refused or ignored with a warning, as the row says.
  • Ignored: accepted, changes nothing, and logs one warning per process. The warning goes to the config’s logger when it has one, else to the pyrula.kafka.compat.confluent logger.
  • Refused: raises KafkaException (_INVALID_ARG) because ignoring it would change data, placement, durability or security, or the feature is absent.
  • Refused (as librdkafka): confluent refuses the key itself, with the same text.
  • pyrula only: accepted by the shim, refused by confluent.
  • Other client: librdkafka accepts a producer key on a consumer, and the reverse, with a warning and no effect. The shim does the same, so one dict can configure both.
  • Unknown: not a key of that client.

Booleans and numbers are read as librdkafka reads them: true, t, 1, false, f, 0 for a boolean, and a number or a string holding one for a number.

The shim does not check a value’s range or the relation between keys (the range of session.timeout.ms, max.poll.interval.ms >= session.timeout.ms, message.max.bytes): librdkafka refuses those at construction, and the native client sees them later, or not.

request.timeout.ms is honoured on a consumer, unlike librdkafka, where it is a producer key: it bounds a broker request and the waits before the first connection.

KeyProducerConsumerWhat pyrula does
acksTranslatedOther client
allow.auto.create.topicsFixed at true; other values ignoredTranslatedConsumer: false, librdkafka’s consumer default, never creates a missing topic; true lets the broker’s auto.create.topics.enable decide for a manual assign(), while subscribe() never creates one. Producer: the native producer’s metadata requests always let the broker auto-create a topic, so false is not honoured: a topic the producer names is created when the broker’s auto.create.topics.enable allows it.
api.version.fallback.msIgnoredIgnoredOnly used with brokers older than 0.10.
api.version.requestFixed at true; other values ignoredFixed at true; other values ignoredAPI versions are always negotiated.
api.version.request.timeout.msIgnoredIgnoredOnly used with brokers older than 0.10.
auto.commit.enableOther clientAlias of enable.auto.commit
auto.commit.interval.msOther clientTranslated
auto.offset.resetOther clientTranslatedValues: earliest, smallest, beginning, latest, largest, end, none, error.
background_event_cbRefused (as librdkafka)Refused (as librdkafka)Property “background_event_cb” must be set through dedicated ..set..() function.
batch.num.messagesTranslatedOther client
batch.sizeTranslatedOther clientCapped at message.max.bytes, as librdkafka caps it, rather than refused.
bootstrap.serversTranslatedTranslatedAt least one broker is required: the shim cannot connect to none.
broker.address.familyFixed at any; other values ignoredFixed at any; other values ignoredAddresses of both families are tried.
broker.address.ttlIgnoredIgnoredThe native client manages its own connections and retries.
broker.version.fallbackIgnoredIgnoredOnly used with brokers older than 0.10.
builtin.featuresIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
check.crcsOther clientFixed (accepts true or false)Record CRCs are always verified.
client.dns.lookupFixed at use_all_dns_ips; other values ignoredFixed at use_all_dns_ips; other values ignoredEvery address a name resolves to is tried.
client.idTranslatedTranslated
client.rackIgnoredIgnoredThe consumer always fetches from the partition leader (no KIP-392 follower fetching).
client.software.nameIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
client.software.versionIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
closesocket_cbRefused (as librdkafka)Refused (as librdkafka)Property “closesocket_cb” must be set through dedicated ..set..() function.
compression.codecAlias of compression.typeOther client
compression.levelIgnoredOther clientThroughput tuning with no native counterpart; delivery is unchanged.
compression.typeTranslatedOther client
connect_cbRefused (as librdkafka)Refused (as librdkafka)Property “connect_cb” must be set through dedicated ..set..() function.
connections.max.idle.msIgnoredIgnoredThe native client manages its own connections and retries.
consume.callback.max.messagesOther clientIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
consume_cbRefused (as librdkafka)Refused (as librdkafka)Property “consume_cb” must be set through dedicated ..set..() function.
coordinator.query.interval.msOther clientIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
debugIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
default.topic.configShimShimA dict flattened into the config; its entries are applied first, so a top-level key wins.
default_topic_confRefused (as librdkafka)Refused (as librdkafka)Property “default_topic_conf” must be set through dedicated ..set..() function.
delivery.report.only.errorShimOther clientSuccessful reports are taken off the queue without calling their callback or counting in poll()’s return; librdkafka never queues them.
delivery.timeout.msAlias of message.timeout.msOther client
dr_cbRefused (as librdkafka)Refused (as librdkafka)Property “dr_cb” must be set through dedicated ..set..() function.
dr_msg_cbRefused (as librdkafka)Refused (as librdkafka)Property “dr_msg_cb” must be set through dedicated ..set..() function.
enable.auto.commitOther clientTranslated
enable.auto.offset.storeOther clientShimThe shim stores each delivered offset itself; with it off only store_offsets() moves the stored offset.
enable.gapless.guaranteeFixed at false; other values refusedOther clientGap-less ordering is not supported.
enable.idempotenceTranslatedOther client
enable.metrics.pushIgnoredIgnoredClient telemetry (KIP-714) is not sent.
enable.partition.eofOther clientShimWith true, poll() and consume() return a message whose error().code() is _PARTITION_EOF when the fetch position reaches the end of a partition, once per partition until it is sought, assigned or grows.
enable.random.seedIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
enable.sasl.oauthbearer.unsecure.jwtFixed at false; other values refusedFixed at false; other values refusedThere is no unsecured-JWT builder.
enable.sparse.connectionsIgnoredIgnoredThe native client manages its own connections and retries.
enable.ssl.certificate.verificationFixed at true; other values refusedFixed at true; other values refusedCertificates are always verified.
enable_sasl_queueIgnoredIgnoredThe native client’s SASL handling has no separate queue.
enabled_eventsIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
error_cbTranslatedTranslatedCalled from poll() on the application’s thread, as librdkafka calls it.
fetch.error.backoff.msOther clientIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
fetch.max.bytesOther clientTranslated
fetch.message.max.bytesOther clientAlias of max.partition.fetch.bytes
fetch.min.bytesOther clientTranslated
fetch.queue.backoff.msOther clientIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
fetch.wait.max.msOther clientTranslated
group.idOther clientTranslated
group.instance.idOther clientTranslated
group.protocolOther clientFixed at classic; other values refusedThe KIP-848 consumer protocol is not supported.
group.protocol.typeOther clientIgnoredUsed only by the KIP-848 consumer group protocol, which is refused.
group.remote.assignorOther clientIgnoredUsed only by the KIP-848 consumer group protocol, which is refused.
heartbeat.interval.msOther clientTranslated
https.ca.locationIgnoredIgnoredUsed only by librdkafka’s built-in OIDC token request; under sasl.oauthbearer.method=oidc it is refused, since the token request verifies against the system trust store.
https.ca.pemIgnoredIgnoredUsed only by librdkafka’s built-in OIDC token request; under sasl.oauthbearer.method=oidc it is refused, since the token request verifies against the system trust store.
interceptorsRefused (as librdkafka)Refused (as librdkafka)Internal property “interceptors” not settable.
internal.termination.signalIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
isolation.levelOther clientTranslated
linger.msTranslatedOther client
log.connection.closeIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
log.queueIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
log.thread.nameIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
log_cbRefused (as librdkafka)Refused (as librdkafka)Property “log_cb” must be set through dedicated ..set..() function.
log_levelIgnoredIgnoredA diagnostic; pass logger= and set its level for the native client’s logs.
loggerTranslatedTranslatedAlso where this page’s warnings go.
max.in.flightAlias of max.in.flight.requests.per.connectionAlias of max.in.flight.requests.per.connection
max.in.flight.requests.per.connectionTranslatedIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
max.partition.fetch.bytesOther clientTranslated
max.poll.interval.msOther clientTranslated
max.poll.recordsOther clientTranslatedThe records one native poll fetches; poll() still hands them out one at a time.
message.copy.max.bytesIgnoredIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
message.max.bytesTranslatedIgnoredProducer: caps a produce request and any one record; a record over it is refused at produce(). Consumer: throughput tuning with no native counterpart; delivery is unchanged.
message.send.max.retriesTranslatedOther client
message.timeout.msTranslatedOther client
metadata.broker.listAlias of bootstrap.serversAlias of bootstrap.servers
metadata.max.age.msIgnoredIgnoredThe native metadata cache has its own refresh cadence.
metadata.recovery.rebootstrap.trigger.msIgnoredIgnoredThe native metadata cache has its own refresh cadence.
metadata.recovery.strategyIgnoredIgnoredThe native metadata cache has its own refresh cadence.
metadata.request.timeout.msIgnoredIgnoredThe native client manages its own connections and retries.
msg_order_cmpRefused (as librdkafka)Refused (as librdkafka)Property “msg_order_cmp” must be set through dedicated ..set..() function.
oauth_cbShimShimCalled as oauth_cb(sasl.oauthbearer.config), returning (token, expiry[, principal[, extensions]]).
oauthbearer_token_refresh_cbRefused (as librdkafka)Refused (as librdkafka)Property “oauthbearer_token_refresh_cb” must be set through dedicated ..set..() function.
offset.store.methodOther clientFixed at broker; other values refusedThe file offset store is not supported.
offset.store.pathOther clientIgnoredUsed only by the file offset store, which is refused.
offset.store.sync.interval.msOther clientIgnoredUsed only by the file offset store, which is refused.
offset_commit_cbRefused (as librdkafka)Refused (as librdkafka)Property “offset_commit_cb” must be set through dedicated ..set..() function.
on_commitUnknownShimCalled as on_commit(err, partitions) from poll(), consume() and close() for an asynchronous commit() and for each automatic commit. A commit(asynchronous=False) never calls it.
on_deliveryShimUnknownThe default delivery callback; a callback= or on_delivery= given to produce() overrides it.
opaqueRefused (as librdkafka)Refused (as librdkafka)Property “opaque” must be set through dedicated ..set..() function.
open_cbRefused (as librdkafka)Refused (as librdkafka)Property “open_cb” must be set through dedicated ..set..() function.
partition.assignment.strategyOther clientTranslated
partitionerTranslatedOther clientmurmur2_random and consistent_random route like murmur2 and consistent; they differ only for records with no key, which the native client spreads itself. Values: murmur2, murmur2_random, consistent, consistent_random; refused: fnv1a, fnv1a_random, random.
partitioner_cbRefused (as librdkafka)Refused (as librdkafka)Property “partitioner_cb” must be set through dedicated ..set..() function.
plugin.library.pathsFixed at empty; other values ignoredFixed at empty; other values ignoredLibrdkafka plugins are not loaded (the usual one is the Control Center monitoring interceptor).
produce.offset.reportIgnoredOther clientA deprecated no-op in librdkafka too.
queue.buffering.backpressure.thresholdIgnoredOther clientThroughput tuning with no native counterpart; delivery is unchanged.
queue.buffering.max.kbytesTranslatedOther client
queue.buffering.max.messagesTranslatedOther client
queue.buffering.max.msAlias of linger.msOther client
queued.max.messages.kbytesOther clientTranslatedKibibytes in, bytes out.
queued.min.messagesOther clientIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
queuing.strategyFixed at fifo; other values refusedOther clientlifo would reorder records.
rebalance.timeout.msUnknownpyrula onlyA pyrula extension; confluent refuses it. Sent in every JoinGroup.
rebalance_cbRefused (as librdkafka)Refused (as librdkafka)Property “rebalance_cb” must be set through dedicated ..set..() function.
receive.message.max.bytesIgnoredIgnoredThroughput tuning with no native counterpart; delivery is unchanged.
reconnect.backoff.jitter.msIgnoredIgnoredThe native client manages its own connections and retries.
reconnect.backoff.max.msIgnoredIgnoredThe native client manages its own connections and retries.
reconnect.backoff.msIgnoredIgnoredThe native client manages its own connections and retries.
request.required.acksAlias of acksOther client
request.timeout.msTranslatedTranslatedIn librdkafka it is producer-only and a consumer ignores it; the shim honours it as the bound of one broker request, and of the waits before the connection is up.
resolve_cbRefused (as librdkafka)Refused (as librdkafka)Property “resolve_cb” must be set through dedicated ..set..() function.
retriesAlias of message.send.max.retriesOther client
retry.backoff.max.msIgnoredIgnoredThe native client manages its own connections and retries.
retry.backoff.msTranslatedIgnoredThe native client manages its own connections and retries.
sasl.jaas.configRefused (as librdkafka)Refused (as librdkafka)Java JAAS configuration is not supported, see https://github.com/confluentinc/librdkafka/wiki/Using-SASL-with-librdkafka for more information.
sasl.kerberos.keytabIgnoredIgnoredGSSAPI only, and GSSAPI is refused as the SASL mechanism.
sasl.kerberos.kinit.cmdIgnoredIgnoredGSSAPI only, and GSSAPI is refused as the SASL mechanism.
sasl.kerberos.min.time.before.reloginIgnoredIgnoredGSSAPI only, and GSSAPI is refused as the SASL mechanism.
sasl.kerberos.principalIgnoredIgnoredGSSAPI only, and GSSAPI is refused as the SASL mechanism.
sasl.kerberos.service.nameIgnoredIgnoredGSSAPI only, and GSSAPI is refused as the SASL mechanism.
sasl.mechanismTranslatedTranslatedA mechanism the native client lacks (GSSAPI) is refused when the client is built.
sasl.mechanismsAlias of sasl.mechanismAlias of sasl.mechanism
sasl.oauthbearer.assertion.algorithmIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.audIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.exp.secondsIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.issIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.jti.includeIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.nbf.secondsIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.claim.subIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.fileIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.jwt.template.fileIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.private.key.fileIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.private.key.passphraseIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.assertion.private.key.pemIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.client.credentials.client.idAlias of sasl.oauthbearer.client.idAlias of sasl.oauthbearer.client.id
sasl.oauthbearer.client.credentials.client.secretAlias of sasl.oauthbearer.client.secretAlias of sasl.oauthbearer.client.secret
sasl.oauthbearer.client.idShimShimUsed under sasl.oauthbearer.method=oidc; inert, with a warning, under default.
sasl.oauthbearer.client.secretShimShimUsed under sasl.oauthbearer.method=oidc; inert, with a warning, under default.
sasl.oauthbearer.configShimShimPassed to oauth_cb as its argument. Without oauth_cb and without sasl.oauthbearer.method=oidc it is refused: there is no built-in unsecured-JWT retriever.
sasl.oauthbearer.extensionsShimShimUsed under sasl.oauthbearer.method=oidc; inert, with a warning, under default.
sasl.oauthbearer.grant.typeFixed at client_credentials; other values refusedFixed at client_credentials; other values refusedThe JWT-bearer grant is not supported.
sasl.oauthbearer.metadata.authentication.typeFixed at none; other values refusedFixed at none; other values refusedAzure IMDS and AWS IAM token retrieval are not supported.
sasl.oauthbearer.methodShimShimdefault uses oauth_cb. oidc builds a client-credentials token provider from the sasl.oauthbearer.client.*, .token.endpoint.url, .scope and .extensions keys when oauth_cb is absent. Values: default, oidc.
sasl.oauthbearer.scopeShimShimUsed under sasl.oauthbearer.method=oidc; inert, with a warning, under default.
sasl.oauthbearer.sub.claim.nameIgnoredIgnoredUsed only by the JWT-bearer grant, which is refused.
sasl.oauthbearer.token.endpoint.urlShimShimUsed under sasl.oauthbearer.method=oidc; inert, with a warning, under default.
sasl.passwordTranslatedTranslated
sasl.usernameTranslatedTranslated
security.protocolTranslatedTranslated
session.timeout.msOther clientTranslated
share.acknowledgement.modeOther clientIgnoredUsed only by share consumers.
socket.blocking.max.msIgnoredIgnoredThe native client manages its own connections and retries.
socket.connection.setup.timeout.msIgnoredIgnoredThe native client manages its own connections and retries.
socket.keepalive.enableIgnoredIgnoredThe native client manages its own connections and retries.
socket.max.failsIgnoredIgnoredThe native client manages its own connections and retries.
socket.nagle.disableIgnoredIgnoredThe native client manages its own connections and retries.
socket.receive.buffer.bytesIgnoredIgnoredThe native client manages its own connections and retries.
socket.send.buffer.bytesIgnoredIgnoredThe native client manages its own connections and retries.
socket.timeout.msIgnoredTranslatedConsumer: bounds one broker request unless request.timeout.ms is given, as librdkafka bounds a consumer’s. Producer: a produce request is bounded by request.timeout.ms, as in librdkafka.
socket_cbRefused (as librdkafka)Refused (as librdkafka)Property “socket_cb” must be set through dedicated ..set..() function.
ssl.ca.certificate.storesRefusedRefusedWindows certificate stores are not read.
ssl.ca.locationTranslatedTranslatedThe value probe means the system trust store, which an unset native field uses.
ssl.ca.pemRefusedRefusedInline PEM is not supported; write it to a file and use the ssl.*.location key.
ssl.certificate.locationTranslatedTranslated
ssl.certificate.pemRefusedRefusedInline PEM is not supported; write it to a file and use the ssl.*.location key.
ssl.certificate.verify_cbRefused (as librdkafka)Refused (as librdkafka)Property “ssl.certificate.verify_cb” must be set through dedicated ..set..() function.
ssl.cipher.suitesRefusedRefusedThe native TLS stack’s own parameter set is used; a policy cannot be narrowed.
ssl.crl.locationRefusedRefusedCRL checks are not supported.
ssl.curves.listRefusedRefusedThe native TLS stack’s own parameter set is used; a policy cannot be narrowed.
ssl.endpoint.identification.algorithmFixed at https; other values refusedFixed at https; other values refusedThe native TLS setup always verifies the broker’s hostname against its certificate, librdkafka’s https; none is not supported.
ssl.engine.idRefusedRefusedOpenSSL providers and engines are not available.
ssl.engine.locationRefusedRefusedOpenSSL providers and engines are not available.
ssl.key.locationTranslatedTranslated
ssl.key.passwordRefusedRefusedEncrypted private keys are not supported; decrypt the key file.
ssl.key.pemRefusedRefusedInline PEM is not supported; write it to a file and use the ssl.*.location key.
ssl.keystore.locationRefusedRefusedPKCS#12 keystores are not supported; use PEM files.
ssl.keystore.passwordRefusedRefusedPKCS#12 keystores are not supported; use PEM files.
ssl.providersRefusedRefusedOpenSSL providers and engines are not available.
ssl.sigalgs.listRefusedRefusedThe native TLS stack’s own parameter set is used; a policy cannot be narrowed.
ssl.truststore.locationRefused (as librdkafka)Refused (as librdkafka)Java TrustStores are not supported, use ssl.ca.location and a certificate file instead. See https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka for more information.
ssl_caRefused (as librdkafka)Refused (as librdkafka)Internal property “ssl_ca” not settable.
ssl_certificateRefused (as librdkafka)Refused (as librdkafka)Internal property “ssl_certificate” not settable.
ssl_engine_callback_dataRefused (as librdkafka)Refused (as librdkafka)Property “ssl_engine_callback_data” must be set through dedicated ..set..() function.
ssl_keyRefused (as librdkafka)Refused (as librdkafka)Internal property “ssl_key” not settable.
statistics.interval.msTranslatedTranslatedNeeds stats_cb; the callback gets librdkafka-style JSON from poll().
stats_cbTranslatedTranslatedCalled from poll() with a JSON string.
sticky.partitioning.linger.msTranslatedOther client
test.mock.broker.rttRefusedRefusedLibrdkafka’s mock cluster is not available; run a broker.
test.mock.num.brokersRefusedRefusedLibrdkafka’s mock cluster is not available; run a broker.
throttle_cbIgnoredIgnoredThe native client does not report broker throttling; it is never called.
topic.blacklistIgnoredRefusedProducer: a producer reads no subscription. Consumer: excluding topics from a subscription is not supported.
topic.metadata.propagation.max.msIgnoredIgnoredThe native metadata cache has its own refresh cadence.
topic.metadata.refresh.fast.cntIgnoredIgnoredThe native metadata cache has its own refresh cadence.
topic.metadata.refresh.fast.interval.msIgnoredTranslatedThe native metadata cache has its own refresh cadence.
topic.metadata.refresh.interval.msTranslatedTranslated
topic.metadata.refresh.sparseIgnoredIgnoredThe native metadata cache has its own refresh cadence.
topicsUnknownpyrula onlyA pyrula extension; confluent refuses it. The topics a consumer subscribes to or reads from.
transaction.timeout.msFixed at 60000; other values refusedOther clientThe native producer’s transaction timeout is fixed at librdkafka’s default.
transactional.idTranslatedOther client

View this page as Markdown