Observability

Tracing

Every durable step and stream operation opens a span on the global OTel tracer:

Span	Attributes
pyrula.step	pyrula.step.id, pyrula.step.kind
pyrula.stream.batch	pyrula.stream.source, pyrula.stream.partition, pyrula.stream.offset
pyrula.stream.commit	pyrula.stream.source, pyrula.stream.partition, pyrula.stream.offset

No opentelemetry package, no spans. The engine doesn’t own the provider: register one with the OTel SDK, or install "pyrula[agents,otel]" and call configure() once at startup (reads OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS). Agents observability has the full setup and trace-context propagation across Kafka.

Kafka runner stats

The Kafka runner merges its counters with the Kafka client’s metrics(). Pull-based, separate from OTel:

stats = runner.stats()
# {
#   "processed": 12043, "dlq": 2, "errors": 0, "in_flight": 8, "held": 0,
#   "last_latency_ms": 4.1,
#   "latency_p50_ms": 3.7, "latency_p95_ms": 9.2, "latency_p99_ms": 14.0,
#   "consumer": {...}, "producer": {...},   # raw pyrula-kafka client metrics
# }

Worker metrics

WorkflowWorker emits OTel instruments for claims, run lifecycle, latency and queue state. Install the [otel] extra and register a MeterProvider before the worker starts, else they’re no-ops.

pip install "pyrula[workflows,otel]"

All instruments carry pyrula.workflow.name. outcome is success or error.

Instrument	Type	Attributes	Notes
pyrula.workflows.runs.claimed	counter	workflow	each claim
pyrula.workflows.runs.active	updowncounter	workflow	in-flight runs
pyrula.workflows.runs.completed	counter	workflow, outcome	terminal completions
pyrula.workflows.run.attempt.duration	histogram (s)	workflow, outcome	claim to terminal event
pyrula.workflows.claim.latency	histogram (s)	workflow	time to acquire the claim lock
pyrula.workflows.run.schedule_to_start.duration	histogram (s)	workflow	submit to first claim (queue wait)
pyrula.workflows.queue.depth	gauge	workflow	pending run count
pyrula.workflows.orphans.recovered	counter	workflow	runs reclaimed after a worker crash
pyrula.workflows.runs.stranded	counter	workflow	runs with no compatible worker version
pyrula.workflows.runs.requeued	counter	workflow, reason	runs the store put back on the queue
pyrula.workflows.schedules.skipped	counter	workflow, reason	schedule occurrences not submitted
pyrula.workflows.loop.restarts	counter	pyrula.worker.loop, workflow on claim loops	background-loop restarts after a store or network error; the loop kind is claim, timer, orphan, lease, sweep, schedule, heartbeat and so on

schedule_to_start.duration and queue.depth suit queue-pressure autoscaling (KEDA etc.). They need a store that records submit time and counts pending runs (MemoryStore, ValkeyStore, or an ArchivingStore over either); otherwise omitted. queue.depth is polled each collection cycle and store errors are swallowed.

Local dev UI

pyrula dev serves a read-only run browser at /ui. It reads the event log and sends no commands.

pyrula dev --attach --store valkey://localhost:6379   # attach read-only to a shared store
pyrula dev path/to/module.py:my_agent        # run a worker in-process, plus the UI

--attach mounts only the UI. A workflow target runs inline with the same read-only UI. An agent target also mounts the full agent HTTP API, so keep it on your machine. Both bind 127.0.0.1 (--host + --insecure to override); the port is --port or PYRULA_DEV_PORT.

The UI shows every event except heartbeats, including step:* and run:init with the run’s params, more than the SSE wire exposes. The index filters by status and run id over the loaded runs (showing 3 of 50 loaded); “Load more” raises the window to 500.