Observability

The durable engine emits OpenTelemetry spans for the work it does, and the Kafka runner exposes pull-based throughput and latency stats. Both are no-ops until you opt in, so there is no overhead in a default install.

Tracing

Every durable step and stream operation opens a span on the global OTel tracer:

Span	Attributes
pyrula.step	pyrula.step.id, pyrula.step.kind
pyrula.stream.batch	pyrula.stream.source, pyrula.stream.partition, pyrula.stream.offset
pyrula.stream.commit	pyrula.stream.source, pyrula.stream.partition

Without the opentelemetry package installed the span helper is a no-op. With it, spans go through whatever tracer provider is registered globally, so they appear in the pipeline you already run.

If you have no provider yet, the simplest setup is the one the agent runtime ships. Install pyrula-agents[otel] and call configure() once at startup; it honors the standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS variables and sets up the global provider that these workflow spans then flow through. See Agents → Observability for the full setup, the GenAI semantic conventions, and W3C trace-context propagation across Kafka.

For a pure pyrula-workflows deployment with no agent runtime, register your own provider with the standard OpenTelemetry SDK; the engine does not own the provider, it only opens spans on it.

Metrics

The Kafka runner keeps lightweight counters and merges them with the Kafka client’s own metrics(). Read them as a plain dict for a lag or throughput dashboard:

stats = runner.stats()
# {
#   "processed": 12043, "dlq": 2, "errors": 0, "in_flight": 8,
#   "last_latency_ms": 4.1,
#   "latency_p50_ms": 3.7, "latency_p95_ms": 9.2, "latency_p99_ms": 14.0,
#   "consumer": {...}, "producer": {...},   # raw pyrula-kafka client metrics
# }

These are pull-based counters, separate from the OTel trace path. Scrape them on whatever interval your dashboard uses. The dlq counter tracks records routed to the dead-letter topic; see the Kafka runner guide for DLQ configuration.

Worker metrics

WorkflowWorker emits nine OTel instruments covering claim throughput, run lifecycle, latency, and queue state. They flow through the same OTel pipeline as any other metric in your process — Prometheus, Grafana, or whatever your MeterProvider exports to.

Requirement: install the [otel] extra and register a MeterProvider before starting the worker. Without both, all instruments are no-ops and have zero overhead.

pip install 'pyrula-workflows[otel]'

Configure the provider the same way you would for the agent runtime (see Agents → Observability for a minimal example using OTEL_EXPORTER_OTLP_ENDPOINT).

Instruments

All instruments carry a pyrula.workflow.name attribute scoped to the workflow name. The outcome attribute on completed-run and duration instruments is either success or error.

Instrument	Type	Attributes	Notes
pyrula.workflows.runs.claimed	counter	per workflow	incremented each time the worker claims a run
pyrula.workflows.runs.active	updowncounter	per workflow	in-flight run count; goes up on claim, down on finish
pyrula.workflows.runs.completed	counter	per workflow, outcome	terminal completions, split by success/error
pyrula.workflows.run.attempt.duration	histogram (s)	per workflow, outcome	wall time from claim to terminal event
pyrula.workflows.claim.latency	histogram (s)	per workflow	time to acquire the claim lock
pyrula.workflows.run.schedule_to_start.duration	histogram (s)	per workflow	time from submit to first claim (queue wait)
pyrula.workflows.queue.depth	gauge	per workflow	pending (unclaimed) run count
pyrula.workflows.orphans.recovered	counter	per workflow	runs reclaimed after a worker crash
pyrula.workflows.runs.stranded	counter	per workflow	runs with no compatible worker version

schedule_to_start.duration and queue.depth match the surfaces that Temporal, Inngest, Hatchet, and KEDA expose for queue-pressure autoscaling. Both require a store that records submission time and supports a pending-count query (Valkey and Postgres stores); they are silently omitted when the capability is absent.

The queue.depth gauge is observable (polled on each collection cycle); errors from the store are swallowed so a transient store failure does not break the metrics pipeline.

Local dev UI

pyrula dev serves a run browser at /ui. It is observability only — it reads the event log and never sends a command back to a job or a worker. There are two modes:

pyrula dev --store valkey://localhost:6379   # attach read-only to a shared store
pyrula dev path/to/module.py:my_agent        # run a worker in-process, plus the UI

Attach mode mounts nothing but the UI, over an inert control object: no cancel, no resume, no signal. A workflow target runs the workflow to completion inline and exposes that same read-only UI, nothing more. An agent target additionally mounts the full agent HTTP API, because it is driving a worker in-process — that API is how the worker is driven, and it is why an agent target is for your machine only. Both bind 127.0.0.1 by default.

The UI shows every event the log holds except heartbeats, including the step:* family and run:init (which carries the run’s params). That is deliberately more than the SSE wire exposes: the wire is for external consumers, this is for debugging. Use the “hide step events” toggle on a noisy run.

The run index filters by status and run id, and states what it filtered over — showing 3 of 50 loaded means the filter applied to the 50 runs on the page, not to every run in the store. “Load more” raises that window to 500.