Record once, iterate free

pyrula dev keeps the newest finished run of its target as a case file in .pyrula/last/: inputs, journal and every recorded step, model and tool result. pyrula case run reruns it against your current code, answering matching calls from the recording, so a new system prompt or model can be tried without calling a tool again. A write the recording cannot answer is refused. Copied into your repository, the same file becomes a pytest test. None of this needs a Cloud account or a network.

This page uses Python 3.11 to 3.15 and a POSIX shell, in an empty directory.

1. The agent

pip install "pyrula[agents]"

Save this as app.py. It is the agent pyrula init writes: one tool and a scripted model, so it runs with no key. TOOL_CALLS counts the times the tool really runs.

app.py

import asyncio
from typing import Any

from pyrula.agents import AgentContext, ToolContext, agent, tool
from pyrula.agents.testing import FakeLLM, FakeResponse
from pyrula.agents.testing.runner import AgentRunner

TOOL_CALLS: list[str] = []


@tool(read_only=True)
async def lookup_order(ctx: ToolContext, order_id: str) -> dict[str, Any]:
    """Look up an order by id."""
    TOOL_CALLS.append(order_id)
    return {"status": "shipped", "total_cents": 4200}


exchange = [
    FakeResponse(
        tool_calls=[{"name": "lookup_order", "id": "t1", "input": {"order_id": "A-1042"}}]
    ),
    FakeResponse(text="Order A-1042 has shipped. Total $42.00."),
]
llm: Any = FakeLLM(exchange * 5)


@agent(llm=llm, timeout=60)
async def support(ctx: AgentContext, message: str = "Where is order A-1042?") -> str:
    return await ctx.llm.run([{"role": "user", "content": message}], tools=[lookup_order])


if __name__ == "__main__":
    asyncio.run(AgentRunner(support, llm=llm).run(message="Where is order A-1042?"))
    print("tool calls:", TOOL_CALLS)

2. Record a run under pyrula dev

pyrula dev app.py

dev runs the agent once at startup. When a run finishes it writes .pyrula/last/support.case and prints where; the newest finished run replaces the file. The dev UI shows the pyrula case run command for each run. Pass --no-save-last to keep nothing.

The commands from here on run in a second terminal. In a script, start dev in the background and wait for the file:

pyrula dev app.py --no-ui > dev.log 2>&1 &
for _ in $(seq 150); do [ -f .pyrula/last/support.case ] && break; sleep 0.2; done
kill $!
ls .pyrula/last

3. Rerun it with a new prompt

Write the system prompt to try into a file and rerun the case with it:

printf 'Answer in one short sentence.\n' > prompt.txt
pyrula case run .pyrula/last/support.case --target app.py:support --from-start \
  --prompt-file prompt.txt

{"key": "llm#0", "kind": "llm", "provenance": "live", "type": "call"}
{"key": "t1", "kind": "tool", "provenance": "reused", "type": "call"}
{"key": "llm#1", "kind": "llm", "provenance": "live", "type": "call"}
{"changed_calls": [], ..., "live_calls": 2, "provenance": {"changed": 0, "live": 2, "reused": 1, "simulated": 0}, ..., "status": "complete", "type": "outcome", ...}

--from-start reruns the whole recorded run. With --prompt-file (or --model ID), every model call runs live on the agent’s own client, because a recorded answer was made under the old prompt. These are real provider calls and are billed. The tool call asks for the same order, so its recorded result answers it (reused). A tool call whose input changed runs live only when the tool is declared read_only=True; any other tool is a write, and the run stops with write_not_allowed unless you pass --allow-write tool:NAME.

Other flags: --llm-ref MODULE:ATTR runs on another client, --replay-only answers every call from the recording and stops at the first it cannot, and --live-budget N or --max-cost-usd X bound what runs live.

4. Keep it as a test

Copy the case into the repository and run it with pytest:

mkdir -p cases
cp .pyrula/last/support.case cases/
pytest -q --pyrula-cases cases

--pyrula-cases DIR collects every .case file under DIR as a test. Each replays the case’s recorded journal against the code registered under its target name and fails on the first difference, such as a step in another order or a recorded call the code no longer makes. Nothing runs live. Targets are imported from app.py and agents.py in the current directory; name other modules with --pyrula-app FILE.

For a hand-written test, the case_fixture fixture loads a case file, and assert_case_compatible runs the same check:

from pyrula.workflows.testing.cases import assert_case_compatible


def test_support_case(case_fixture):
    case = case_fixture("cases/support.case")
    assert case.manifest.target_name == "support"
    assert_case_compatible(case)

In CI, the same pytest --pyrula-cases cases line needs no model key, store or Cloud account. Without --pyrula-cases the plugin collects nothing.

Case file contents

A case file holds inputs, every journal event and every recorded model and tool result, as zstd-compressed JSON, written owner-only (mode 0600) in .pyrula/, which pyrula init adds to .gitignore. Read a case before you copy it into a repository: a recorded prompt or tool result can carry customer data or a secret. A target with capture level none (capture="none" or PYRULA_CLOUD_CAPTURE=none) is never saved. A failed save is reported on stderr and pyrula dev tries again after the next run.