Observability

The durable engine emits OpenTelemetry spans for the work it does, and the Kafka runner exposes pull-based throughput and latency stats. Both are no-ops until you opt in, so there is no overhead in a default install.

Tracing

Every durable step and stream operation opens a span on the global OTel tracer:

Span	Attributes
pyrula.step	pyrula.step.id, pyrula.step.kind
pyrula.stream.batch	pyrula.stream.source, pyrula.stream.partition, pyrula.stream.offset
pyrula.stream.commit	pyrula.stream.source, pyrula.stream.partition

Without the opentelemetry package installed the span helper is a no-op. With it, spans go through whatever tracer provider is registered globally, so they appear in the pipeline you already run.

If you have no provider yet, the simplest setup is the one the agent runtime ships. Install pyrula-agents[otel] and call configure() once at startup; it honors the standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS variables and sets up the global provider that these workflow spans then flow through. See Agents → Observability for the full setup, the GenAI semantic conventions, and W3C trace-context propagation across Kafka.

For a pure pyrula-workflows deployment with no agent runtime, register your own provider with the standard OpenTelemetry SDK; the engine does not own the provider, it only opens spans on it.

Metrics

The Kafka runner keeps lightweight counters and merges them with the Kafka client’s own metrics(). Read them as a plain dict for a lag or throughput dashboard:

stats = runner.stats()
# {
#   "processed": 12043, "dlq": 2, "errors": 0, "in_flight": 8,
#   "last_latency_ms": 4.1,
#   "latency_p50_ms": 3.7, "latency_p95_ms": 9.2, "latency_p99_ms": 14.0,
#   "consumer": {...}, "producer": {...},   # raw pyrula-kafka client metrics
# }

These are pull-based counters, separate from the OTel trace path. Scrape them on whatever interval your dashboard uses. The dlq counter tracks records routed to the dead-letter topic; see the Kafka runner guide for DLQ configuration.

Worker metrics

WorkflowWorker emits nine OTel instruments covering claim throughput, run lifecycle, latency, and queue state. They flow through the same OTel pipeline as any other metric in your process — Prometheus, Grafana, or whatever your MeterProvider exports to.

Requirement: install the [otel] extra and register a MeterProvider before starting the worker. Without both, all instruments are no-ops and have zero overhead.

pip install 'pyrula-workflows[otel]'

Configure the provider the same way you would for the agent runtime (see Agents → Observability for a minimal example using OTEL_EXPORTER_OTLP_ENDPOINT).

Instruments

All instruments carry a pyrula.workflow.name attribute scoped to the workflow name. The outcome attribute on completed-run and duration instruments is either success or error.

Instrument	Type	Attributes	Notes
pyrula.workflows.runs.claimed	counter	per workflow	incremented each time the worker claims a run
pyrula.workflows.runs.active	updowncounter	per workflow	in-flight run count; goes up on claim, down on finish
pyrula.workflows.runs.completed	counter	per workflow, outcome	terminal completions, split by success/error
pyrula.workflows.run.attempt.duration	histogram (s)	per workflow, outcome	wall time from claim to terminal event
pyrula.workflows.claim.latency	histogram (s)	per workflow	time to acquire the claim lock
pyrula.workflows.run.schedule_to_start.duration	histogram (s)	per workflow	time from submit to first claim (queue wait)
pyrula.workflows.queue.depth	gauge	per workflow	pending (unclaimed) run count
pyrula.workflows.orphans.recovered	counter	per workflow	runs reclaimed after a worker crash
pyrula.workflows.runs.stranded	counter	per workflow	runs with no compatible worker version

schedule_to_start.duration and queue.depth match the surfaces that Temporal, Inngest, Hatchet, and KEDA expose for queue-pressure autoscaling. Both require a store that records submission time and supports a pending-count query (Valkey and Postgres stores); they are silently omitted when the capability is absent.

The queue.depth gauge is observable (polled on each collection cycle); errors from the store are swallowed so a transient store failure does not break the metrics pipeline.