Kafka runtime guarantees

The Kafka runtime provides exactly-once processing and at-least-once output delivery by default. @kafka_agent uses the same runtime contracts as @kafka_workflow with an additional LLM loop.

yes means the guarantee holds; partial means it holds within the boundary under Limits; no means it does not hold and Limits says why.

Binding	processing	output delivery	poison	handler error
kafka_workflow	yes	partial	yes	yes
kafka_workflow[exactly_once]	yes	yes	yes	yes
kafka_workflow[durable=False]	no	partial	yes	yes
kafka_agent	yes	partial	yes	yes
kafka_agent[exactly_once]	yes	yes	yes	yes

Column definitions

• processing: Exactly-once: a redelivered record whose run completed is not re-run.
  • Its result is read back from the run store.
  • A retried record whose run is partial resumes from its checkpointed steps; completed steps do not re-execute.
  • durable=False rows are at-least-once: the handler and every inline step re-run.
• output delivery: A failed delivery never commits past its record.
  • at_least_once rows: a failed delivery commits nothing, so the record is redelivered and re-produced.
  • exactly_once rows: output records and input offsets commit in one transaction; an abort leaves neither.
• poison: A record that fails decode goes to the DLQ topic, with the error, before the workflow runs.
  • The run store never sees it; the rest of the batch proceeds and commits.
• handler error: After max_retries, the on_error disposition holds.
  • dlq routes the record with the error; fail stops the worker without committing past it; skip commits past it.
  • A run that pauses (ctx.sleep, ctx.interrupt, ctx.wait_for_signal) is an error on this path and never stalls its partition.

Install and tested versions

All bindings: built in. Supported brokers: Kafka 3.9+ (tested on 3.9 and 4.4), Confluent Platform 7.9+.

Notes

• kafka_workflow[durable=False]: durable=False has no run store: the handler re-runs on every retry or redelivery, by design

Limits

• kafka_workflow - output delivery; kafka_workflow[durable=False] - output delivery; kafka_agent - output delivery: at-least-once: a crash between produce and commit re-produces on redelivery; dedupe downstream on key, or choose delivery=“exactly_once”

Runner behavior

• Handler error: on_error="fail" stops the runner’s worker, not only the one record.
• Delivery: a failed produce, offset commit or aborted transaction rewinds the partition to the uncommitted offset and retries with a backoff instead of stopping the runner; a fatal error, or eight failures in a row, stops it and run() raises KafkaRunnerFailed so the process exits non-zero. A rewind re-runs the handler for the whole drain, including records that went to the dead-letter topic (a run that ended in error is not reused), so a failing record repeats its handler attempts on each retry - at-least-once processing, not a leak.
• Processing: kafka_workflow(durable=False) runs with no run store and reruns on redelivery.
• Poison: a poison record sharing a drain with good records does not stall the batch or take the consumer down with it.
• Processing: an existing @workflow stacked under @kafka_workflow still coerces its persisted input the same way a bare handler does.
• Output delivery: exactly-once holds across a cooperative rebalance between live members: every input id on the output exactly once, read-committed.
• Output delivery: each partition has its own transactional producer, with transactional id group:topic:partition, and init_transactions fences any earlier producer using that id. This is per-partition transactional-id fencing (KIP-98) only. The offset commit goes through the producer using the consumer’s group id and coordinator, but the TxnOffsetCommit request is pinned to v0 and carries no generation or member id, so KIP-447 group-generation fencing is not implemented. The producer is created on a partition’s first delivery, not at assignment.