# Observability

> OpenTelemetry spans, runner metrics, and worker instruments for the durable engine.

Source: https://docs.pyrula.io/workflows/guides/observability/

## Tracing

Every durable step and stream operation opens a span on the global OTel tracer:

| Span | Attributes |
|---|---|
| `pyrula.step` | `pyrula.step.id`, `pyrula.step.kind` |
| `pyrula.stream.batch` | `pyrula.stream.source`, `pyrula.stream.partition`, `pyrula.stream.offset` |
| `pyrula.stream.commit` | `pyrula.stream.source`, `pyrula.stream.partition`, `pyrula.stream.offset` |

No `opentelemetry` package, no spans. The engine doesn't own the provider: register one with the
OTel SDK, or install `"pyrula[agents,otel]"` and call `configure()` once at startup (reads
`OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS`). [Agents observability](https://docs.pyrula.io/agents/observability.md)
has the full setup and trace-context propagation across Kafka.

## Kafka runner stats

The Kafka runner merges its counters with the Kafka client's `metrics()`. Pull-based, separate
from OTel:

```python
stats = runner.stats()
# {
#   "processed": 12043, "dlq": 2, "errors": 0, "in_flight": 8, "held": 0,
#   "last_latency_ms": 4.1,
#   "latency_p50_ms": 3.7, "latency_p95_ms": 9.2, "latency_p99_ms": 14.0,
#   "consumer": {...}, "producer": {...},   # raw pyrula-kafka client metrics
# }
```

## Worker metrics

`WorkflowWorker` emits OTel instruments for claims, run lifecycle, latency and queue state.
Install the `[otel]` extra and register a `MeterProvider` before the worker starts, else they're
no-ops.

```
pip install "pyrula[workflows,otel]"
```

All instruments carry `pyrula.workflow.name`. `outcome` is `success` or `error`.

| Instrument | Type | Attributes | Notes |
|---|---|---|---|
| `pyrula.workflows.runs.claimed` | counter | workflow | each claim |
| `pyrula.workflows.runs.active` | updowncounter | workflow | in-flight runs |
| `pyrula.workflows.runs.completed` | counter | workflow, `outcome` | terminal completions |
| `pyrula.workflows.run.attempt.duration` | histogram (s) | workflow, `outcome` | claim to terminal event |
| `pyrula.workflows.claim.latency` | histogram (s) | workflow | time to acquire the claim lock |
| `pyrula.workflows.run.schedule_to_start.duration` | histogram (s) | workflow | submit to first claim (queue wait) |
| `pyrula.workflows.queue.depth` | gauge | workflow | pending run count |
| `pyrula.workflows.orphans.recovered` | counter | workflow | runs reclaimed after a worker crash |
| `pyrula.workflows.runs.stranded` | counter | workflow | runs with no compatible worker version |
| `pyrula.workflows.runs.requeued` | counter | workflow, `reason` | runs the store put back on the queue |
| `pyrula.workflows.schedules.skipped` | counter | workflow, `reason` | schedule occurrences not submitted |
| `pyrula.workflows.loop.restarts` | counter | `pyrula.worker.loop`, workflow on claim loops | background-loop restarts after a store or network error; the loop kind is `claim`, `timer`, `orphan`, `lease`, `sweep`, `schedule`, `heartbeat` and so on |

`schedule_to_start.duration` and `queue.depth` suit queue-pressure autoscaling (KEDA etc.). They
need a store that records submit time and counts pending runs (`MemoryStore`, `ValkeyStore`, or an
`ArchivingStore` over either); otherwise omitted. `queue.depth` is polled each collection cycle and
store errors are swallowed.

## Local dev UI

`pyrula dev` serves a read-only run browser at `/ui`. It reads the event log and sends no commands.

```bash
pyrula dev --attach --store valkey://localhost:6379   # attach read-only to a shared store
pyrula dev path/to/module.py:my_agent        # run a worker in-process, plus the UI
```

`--attach` mounts only the UI. A workflow target runs inline with the same read-only UI. An agent
target also mounts the full agent HTTP API, so keep it on your machine. Both bind `127.0.0.1`
(`--host` + `--insecure` to override); the port is `--port` or `PYRULA_DEV_PORT`.

The UI shows every event except heartbeats, including `step:*` and `run:init` with the run's
params, more than the SSE wire exposes. The index filters by status and run id over the loaded runs
(`showing 3 of 50 loaded`); "Load more" raises the window to 500.