Agent Evals

pyrula.agents.evals is an experimental local-first regression harness for @agent code. It runs each case through the durable production path (InlineWorker -> RunExecutor -> AgentProfile) rather than a separate fake engine, so you observe the same output validation, tool routing, replay, and event-log behavior your real turns use.

The V1 focus is practical OSS use:

• golden routing and tool-use regression suites
• deterministic and custom evaluators
• CI-friendly exit codes and JSON artifacts
• offline ownership with no database or hosted service requirement

It is not an online eval product, experiment registry, tracing console, or benchmarking platform.

Quick example

The runnable example is fully offline and writes a versioned report artifact:

python examples/agent_evals.py

from pyrula.agents.evals import EvalCase, EvalDataset, EvalSuite, equals_expected, tool_called

Use it when you want to lock down behavior such as:

• “refund prompts still route to billing”
• “the agent still calls lookup_order before answering”
• “a refactor did not silently change output shape”

Why this exists

Most agent eval tooling splits in two directions:

• lightweight mock-based tests that are fast but do not exercise the real turn machinery
• hosted eval products that assume remote storage, tracing, or a managed control plane

Pyrula’s position is narrower and more defensible:

• local/offline by default
• durable runtime semantics preserved
• readable artifacts suitable for OSS CI
• bring your own LLM client, evaluator logic, and fixtures

That means the scope is intentionally smaller than hosted products. V1 does not claim statistical significance, cross-provider ranking authority, online replay against archived traces, or exactly-once external side effects across concurrent eval runs.

Authoring a suite

An eval suite is just Python: an ordered dataset, one agent, and a set of evaluators.

from pyrula.agents.evals import EvalCase, EvalDataset, EvalSuite, equals_expected, tool_called

suite = EvalSuite(
    name="support-router",
    agent=support_router,
    dataset=EvalDataset(
        name="golden",
        cases=[
            EvalCase(
                id="refund-001",
                inputs={"message": "Please refund order 42", "order_id": "42"},
                expected={"route": "billing"},
            ),
        ],
    ),
    evaluators=(
        equals_expected(),
        tool_called("lookup_order"),
    ),
    llm_factory=build_fake_llm,
)

Cases and expectations

• inputs are the keyword arguments passed to the agent
• expected is optional
• expected=MISSING and expected=None are distinct
• dataset order is preserved in the report and in the dataset digest

Use metadata for non-authoritative labels such as scenario, owner, or fixture origin.

Evaluators

V1 supports:

• deterministic built-ins such as equals_expected(), tool_called(), tool_sequence(), contains_text(), max_tokens(), and max_latency()
• custom sync evaluators
• custom async evaluators
• user-authored LLM judges, if you want them

Evaluators run sequentially inside a completed trial against an immutable observation: output, tool call summaries, token usage, latency, and the captured event view.

Running suites

From Python:

report = await suite.run()

The returned EvaluationReport is immutable and includes:

• suite / dataset / agent identity
• evaluator definitions
• selected case IDs
• effective run config
• per-case and per-trial outcomes
• redacted case snapshots for reproducible artifacts
• canonical dataset digest

From the CLI:

pyrula eval myapp.evals:suite \
  --case refund-001 \
  --max-concurrency 4 \
  --timeout 30 \
  --output eval-report.json

Exit codes are stable:

• 0: pass
• 1: quality failure
• 2: import / usage / preflight error
• 3: execution or evaluator error

--fail-under can tighten a run’s case-pass threshold for CI without mutating the suite definition.

Repetitions and nondeterminism

If your agent uses a real provider or any intentionally variable behavior, use repetitions > 1 and reason about pass rates instead of single outcomes.

• required evaluator pass rates gate trial and case success
• optional evaluators remain visible in reports but do not fail the suite
• mean scores are descriptive only; they are not a second pass authority

V1 does not include confidence intervals or experiment-comparison statistics.

Factories for non-thread-safe clients

Static llm, deps, and memory_backend objects remain caller-owned. If a resource should be fresh per trial, use *_factory instead.

This is the right default for:

• fake/scripted LLM clients
• ephemeral dependency objects
• non-thread-safe test doubles
• stateful memory backends you do not want shared across trials

Write-tool safety

Write tools are denied by default in eval mode. This is deliberate: V1 is for regression confidence, not side-effect execution.

To allow write tools, two things must both be true:

1. the suite declares allow_writes=True
2. the CLI invocation passes --allow-writes

If either side is missing, writes stay denied. A blocked write becomes a typed execution error rather than a model-visible tool failure.

Privacy and artifacts

Default JSON artifacts are privacy-minimal:

• case payloads are redacted
• tool payloads are omitted by default
• full event payloads are excluded by default
• evaluator error text is bounded and sanitized

Opt into events with include_events=True or --include-events. Event payloads still go through redaction and payload capping before serialization.

This makes the V1 report suitable for OSS CI logs and artifacts without requiring a database or archive service.

Using a real provider

You can run the same suite against a real provider-backed client:

from pyrula.agents.llm import AnthropicLLM

suite = EvalSuite(
    name="support-router",
    agent=support_router,
    dataset=dataset,
    evaluators=evaluators,
    llm=AnthropicLLM(...),
    repetitions=3,
)

Use this when you want provider-real behavior, but keep expectations grounded: V1 does not persist experiments or compare providers for you.

V1 boundaries

Shipped in V1:

• local-first suite execution
• immutable reports and JSON artifacts
• built-in and custom evaluators
• deterministic offline examples
• CLI execution with stable exit codes

Deferred beyond V1:

• online / production eval orchestration
• archived-trace replay or export workflows
• signal / HITL / continuation eval scenarios
• first-party hosted LLM judges
• cost scoring
• confidence intervals and statistical comparison
• experiment persistence, registry, or UI
• semantic diffing between runs
• generic workflow-target evals

Stability

pyrula.agents.evals is exported as an experimental subpackage. Keep eval names out of your top-level compatibility assumptions and pin a version if you depend on specific report or CLI details during the 0.x line.