Benchmarks

pyrula.kafka against the common Python Kafka clients — confluent-kafka (librdkafka), aiokafka, and kafka-python. Reproducible from benchmarks/kafka/benchmark.py.

Scorecard

100,000 records, 1 KB payloads. Higher is better (rec/s).

Suite	pyrula	confluent	aiokafka	kafka-python	vs confluent
Produce bytes	159k	115k	93k	6.1k	1.38x
Produce JSON	142k	124k	91k	5.5k	1.15x
Transactional produce	89k	78k	n/a	n/a	1.14x
Produce Avro	151k	138k	n/a	n/a	1.09x
Consume records	147k	149k	117k	6.3k	0.99x
Consume raw bytes	147k	147k	n/a	n/a	1.00x
Consume + JSON decode	132k	130k	85k	5.7k	1.01x
Consume Avro	141k	34k	n/a	n/a	4.18x

n/a = not separately benchmarked, not necessarily unsupported: kafka-python has no transaction API; aiokafka transactions and Avro-via-fastavro exist but were not measured; raw-bytes consume for both is the same path as record consume.

Produce

Bulk produce_many_* batches across partitions and serializes in Rust (no per-record Python json.dumps). For Avro, use the fused produce_many_avro so the Rust encode overlaps the broker send.

Consume

Parity with confluent-kafka on byte and JSON consume. Avro is the standout: pyrula deframes and decodes the whole batch in parallel Rust (AvroRegistrySerde.deserialize_many_objects) against a per-record fastavro loop.

Methodology

• Apple M1 Max, single local cp-kafka broker, 12 partitions, no replication.
• 100,000 records, 1 KB payloads. Topics pre-loaded; consumers use manual assign() and seek to zero between runs (no group-join in the measured path).
• One warmup, best of three. A prime step consumes the first record before the clock so cold fetch latency is not billed. Same treatment for every client.
• confluent-kafka 2.x, aiokafka 0.14, kafka-python 2.3, fastavro 1.x.

docker compose up -d kafka schema-registry
python benchmarks/kafka/benchmark.py \
  --broker localhost:9092 --schema-registry http://localhost:8081 \
  --suites P1,P2,P3,C1,C2,C3,A1,A2 --sizes 100000 --payload-bytes 1024 \
  --partitions 12 --fetch-mode per_broker

Localhost removes the network; over a real link produce and consume become bandwidth and round-trip bound. The durable pyrula advantages are where Rust does real work: bulk serde, Avro, transactions, and JSON.