Agent Evals

Generated from the type stubs and docstrings. Do not edit by hand.

Types

CaseReport

All repetition reports and aggregate counts for one dataset case.

Fields:

• case_id: str
• status: OutcomeStatus
• trials: Sequence[TrialReport]
• inputs: Mapping[str, JSONValue] = field(default_factory=dict)
• expected: Optional[JSONValue | _Missing] = MISSING
• counts: Mapping[str, int] = field(default_factory=dict)
• evaluator_pass_rates: Mapping[str, float] = field(default_factory=dict)
• passed: bool

---

EvalCase

One named agent input and optional reference output.

Fields:

• id: str
• inputs: Mapping[str, Any]
• expected: Any = MISSING
• metadata: Mapping[str, JSONValue] = field(default_factory=dict)

---

EvalDataset

An ordered collection of uniquely identified evaluation cases.

Fields:

• name: str
• cases: Sequence[EvalCase]
• description: Optional[str] = None

from_jsonl

from_jsonl(cls, path: Path, name: Optional[str] = None) -> EvalDataset

to_jsonl

to_jsonl(self, path: Path) -> None

---

EvalSuite

Fields:

• name: str
• agent: Callable[..., Any]
• dataset: EvalDataset
• evaluators: Sequence[Evaluator]
• llm: Any = None
• llm_factory: Optional[Factory] = None
• deps: Any = None
• deps_factory: Optional[Factory] = None
• memory_backend: Any = None
• memory_backend_factory: Optional[Factory] = None
• repetitions: int = 1
• min_trial_pass_rate: float = 1.0
• min_case_pass_rate: float = 1.0
• max_concurrency: int = 4
• timeout: Optional[float] = None
• allow_writes: bool = False
• metadata: Mapping[str, Any] = field(default_factory=dict)

run

run(self, case_ids: Optional[Sequence[str]] = None, max_concurrency: Optional[int] = None, timeout: Optional[float] = None, allow_writes: Optional[bool] = None) -> EvaluationReport

---

EvaluationConfigurationError

The suite is invalid and no trials were started.

---

EvaluationContext

The complete immutable observation passed to an evaluator.

Fields:

• case: EvalCase
• output: JSONValue
• run_id: str
• trial_index: int
• agent_name: str
• agent_version: Optional[str]
• tool_calls: tuple[ToolCallView, ...]
• usage: TokenUsage
• latency_ms: float
• events: tuple[TurnEventView, ...]

---

EvaluationError

Base error for evaluation configuration and execution.

---

EvaluationExecutionError

A trial could not produce a completed observation.

---

EvaluationReport

The immutable aggregate result of an evaluation suite execution.

Fields:

• schema: str
• evaluation_id: str
• suite_name: str
• dataset_name: str
• agent_name: str
• status: OutcomeStatus
• cases: Sequence[CaseReport]
• started_at: Optional[str] = None
• finished_at: Optional[str] = None
• agent_version: Optional[str] = None
• dataset_digest: Optional[str] = None
• selected_case_ids: Sequence[str] = field(default_factory=tuple)
• evaluators: Sequence[EvaluatorDefinition] = field(default_factory=tuple)
• run_config: Optional[RunConfig] = None
• counts: Mapping[str, int] = field(default_factory=dict)
• case_pass_rate: Optional[float] = None
• mean_scores: Mapping[str, float] = field(default_factory=dict)
• metadata: Mapping[str, JSONValue] = field(default_factory=dict)
• passed: bool

---

EvaluationResult

A score and optional evaluator diagnostics before status normalization.

Fields:

• score: float
• reason: Optional[str] = None
• details: Mapping[str, JSONValue] = field(default_factory=dict)

---

EvaluationWriteBlocked

A write-capable tool was denied by evaluation policy.

---

Evaluator

One named synchronous or asynchronous evaluator definition.

Fields:

• name: str
• evaluate: SyncOrAsyncEvaluator
• threshold: float = 1.0
• required: bool = True
• timeout: float = 30.0
• version: Optional[str] = None

---

EvaluatorDefinition

Evaluator identity and thresholds captured once per report.

Fields:

• name: str
• threshold: float
• required: bool
• timeout: float
• version: Optional[str] = None

---

EvaluatorError

An evaluator failed or returned an invalid result.

---

EvaluatorReport

The normalized status and diagnostics for one evaluator invocation.

Fields:

• name: str
• status: EvaluatorStatus
• required: bool = True
• score: Optional[float] = None
• threshold: float = 1.0
• reason: Optional[str] = None
• details: Mapping[str, JSONValue] = field(default_factory=dict)
• error_type: Optional[str] = None
• error: Optional[str] = None
• passed: bool

---

EvaluatorStatus

The outcome of one evaluator for one completed trial.

Fields:

• PASSED = 'passed'
• FAILED = 'failed'
• SKIPPED = 'skipped'
• ERROR = 'error'

---

OutcomeStatus

The aggregate outcome of a trial, case, or complete evaluation.

Fields:

• PASSED = 'passed'
• FAILED = 'failed'
• ERROR = 'error'

---

RunConfig

The effective run configuration for one evaluation execution.

Fields:

• repetitions: int
• max_concurrency: int
• min_trial_pass_rate: float
• min_case_pass_rate: float
• timeout: Optional[float]
• allow_writes: bool

---

TokenUsage

Token counters folded from a trial’s LLM completion events.

Fields:

• input_tokens: int = 0
• output_tokens: int = 0
• cache_creation_input_tokens: int = 0
• cache_read_input_tokens: int = 0

---

ToolCallView

An immutable summary of one tool call in the completed trial journal.

Fields:

• tool_use_id: str
• name: str
• input: Optional[JSONValue]
• status: str
• result: Optional[JSONValue]

---

TrialContext

The immutable identity supplied to a trial-local resource factory.

Fields:

• case: EvalCase
• repetition: int

---

TrialReport

A completed or failed trial and its evaluator outcomes.

Fields:

• case_id: str
• repetition: int
• run_id: Optional[str]
• status: OutcomeStatus
• output: Optional[JSONValue]
• latency_ms: float
• usage: TokenUsage
• tool_calls: Sequence[ToolCallView] = field(default_factory=tuple)
• events: Sequence[TurnEventView] = field(default_factory=tuple)
• evaluators: Sequence[EvaluatorReport] = field(default_factory=tuple)
• model: Optional[str | tuple[str, ...]] = None
• provider: Optional[str] = None
• error_phase: Optional[str] = None
• error_code: Optional[str] = None
• error_type: Optional[str] = None
• error: Optional[str] = None
• passed: bool

---

TurnEventView

A JSON-safe immutable event view available to local evaluators.

Fields:

• run_seq: int
• kind: str
• payload: Mapping[str, JSONValue]

Constants

• MISSING: Final = _Missing()

Functions

contains_text

contains_text(text: Optional[str] = None) -> Evaluator

equals_expected

equals_expected() -> Evaluator

max_latency

max_latency(milliseconds: float) -> Evaluator

max_tokens

max_tokens(limit: int) -> Evaluator

report_to_dict

report_to_dict(report: EvaluationReport, include_events: bool = False, max_payload_bytes: int = DEFAULT_MAX_PAYLOAD_BYTES) -> dict[str, Any]

run_evaluator

run_evaluator(evaluator: Evaluator, context: EvaluationContext) -> EvaluatorReport

Run one evaluator with timeout and normalize its result.

run_evaluators

run_evaluators(evaluators: Sequence[Evaluator], context: EvaluationContext) -> tuple[EvaluatorReport, ...]

tool_called

tool_called(name: str, min_calls: int = 1, max_calls: Optional[int] = None) -> Evaluator

tool_not_called

tool_not_called(name: str) -> Evaluator

tool_sequence

tool_sequence(names: Sequence[str], exact: bool = False) -> Evaluator

write_report_json

write_report_json(path: Path, report: EvaluationReport, include_events: bool = False, max_payload_bytes: int = DEFAULT_MAX_PAYLOAD_BYTES) -> None