Observability

The agent runtime emits OpenTelemetry spans for every turn, LLM call, tool call, and durable step. Span names and attributes follow the GenAI semantic conventions, so backends that understand them (Langfuse, LangSmith, Arize Phoenix, Datadog) render token counts, models, and tool calls without any mapping work on your side.

Without the otel extra installed, all of this is a no-op. There is no overhead and nothing to turn off.

Setup

pip install 'pyrula-agents[otel]'

Then one call at startup, before the worker starts:

from pyrula.agents.observability import configure

configure()

With no arguments, configure() honors the standard OTLP environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <token>"

That covers any OTLP backend, including a local collector. For the common SaaS backends there are shortcuts that fill in the endpoint and auth headers:

from pyrula.agents.observability import configure_langfuse

# reads LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST
configure_langfuse()

from pyrula.agents.observability import configure_langsmith

# reads LANGSMITH_API_KEY and LANGSMITH_PROJECT
configure_langsmith()

from pyrula.agents.observability import configure_datadog

# exports to the local Datadog Agent's OTLP intake (enable otlp_config in the Agent)
configure_datadog()

Keys can also be passed explicitly (configure_langfuse(public_key=..., secret_key=...)). To check what is being emitted without a backend, print spans to stdout:

configure(console=True)

If your application already sets up an OTel tracer provider, skip configure() entirely. Pyrula’s spans go through the global provider, so they show up in whatever pipeline you already have.

What gets traced

One agent turn produces a span tree like:

invoke_agent billing_agent
├── chat claude-sonnet-4-6          gen_ai.usage.input_tokens=812 output_tokens=94
│   └── execute_tool lookup_invoice  gen_ai.tool.name=lookup_invoice
└── chat claude-sonnet-4-6          gen_ai.usage.input_tokens=931 output_tokens=187

Spans carry gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name, and token usage. Durable steps and stream batches get pyrula.* spans nested in the same trace.

Prompt and completion content

Message content (prompts, completions, tool arguments and results) is recorded on spans only when content capture is on. configure() turns it on by default, since calling it is an explicit opt-in to tracing. Turn it off for backends that must not see message content:

configure_datadog(capture_content=False)

Without configure(), capture follows the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env var and defaults to off. Captured values are truncated at 32 KB.

Metrics

configure() also exports the two GenAI client metrics:

• gen_ai.client.token.usage (histogram, by model, provider, and token type)
• gen_ai.client.operation.duration (histogram, seconds, tagged with error.type on failure)

Langfuse and LangSmith only ingest traces, so their shortcuts disable metrics export automatically. Pass metrics=False to configure() for any other trace-only backend.

Tracing across Kafka

KafkaAgentRunner propagates W3C trace context through Kafka headers, so a pipeline of agents on different topics shows up as one trace:

• Consuming a record starts a process {topic} consumer span, parented to the traceparent header stamped by the upstream producer.
• Records produced by the turn carry the processing span’s context, so the next consumer chains to this turn rather than skipping it.
• DLQ records keep the source record’s traceparent. From a dead letter you can jump straight to the trace of the run that failed.

None of this requires configuration beyond configure(). Without OTel, the runner falls back to forwarding the inbound traceparent header verbatim, so trace continuity survives a mixed fleet where only some services are instrumented.

For manual produce paths, build headers from the active span yourself:

from pyrula.agents.observability.otel import inject_current_context

headers = {k: v.encode() for k, v in inject_current_context().items()}

Conversations, usage, and cost in SQL

Tracing answers “what happened in this one run”. For the other questions, how many tokens did this agent burn last week, what did each model cost, show me the full transcript of run X, you want SQL. Pyrula can fold the durable event log into three read tables you can query directly:

• pyrula_agent_messages — one row per conversation turn (role, JSONB content).
• pyrula_agent_usage — one row per LLM call with token counts.
• pyrula_agent_cost — a view that joins usage to a price table and sums cost per (name, run_id, thread_id, model).

This is a read-only projection. The event log stays the system of record; the tables are derived from it and can be rebuilt at any time. Your agent code and ctx never read them, so there is no way for a query to affect a live run.

Enable it

Pass project=True to the agent Postgres store. That runs the projection migrations once and registers the projectors; the worker drains new events into the tables every 60 seconds.

from pyrula.agents import PyrulaWorker
from pyrula.agents.stores.postgres import PostgresStore

store = PostgresStore(
    dsn="postgresql://localhost/pyrula",
    valkey_url="redis://localhost:6379",
    project=True,
)
worker = PyrulaWorker(agents=[support], store=store, llm=...)
worker.run()

The projection uses its own migration chain (pyrula_agent_alembic_version), so it can share a database with the workflow archive without colliding.

Query it

Spend per model, last 7 days:

SELECT model, SUM(input_tokens) AS in_tok, SUM(output_tokens) AS out_tok, SUM(cost) AS usd
FROM pyrula_agent_cost
WHERE run_id IN (SELECT run_id FROM pyrula_agent_usage WHERE created_at > NOW() - INTERVAL '7 days')
GROUP BY model
ORDER BY usd DESC;

Full transcript of one run, in order:

SELECT role, content
FROM pyrula_agent_messages
WHERE name = 'support' AND run_id = '<run-id>'
ORDER BY msg_seq;

Prices live in the pyrula_agent_price table (USD per 1K tokens), seeded for the current Claude models. Edit a row to reprice; unpriced models show up with cost = 0 rather than disappearing. pyrula.agents.projection.price_map.DEFAULT_PRICE_MAP mirrors the seeded values.

Backfill

Runs archived before you turned projection on (or after a price change) are filled in with pyrula reproject:

pyrula reproject                                 # every archived run, every agent
pyrula reproject --agent support                 # one agent
pyrula reproject --agent support --run-id <id>   # one run (--run-id needs --agent)