Replay checks in CI

pyrula ci run checks that a PR’s code still does what it did in the runs you saved as cases. It runs in your CI job against your checkout, using only each case’s recording. Cloud pins the suite and keeps the result.

What a pass means

The code gets every recorded model and tool result it asks for; a call the recording can’t answer stops the case. A case passes when the code made the same calls with the same inputs, reached the same end, and every pinned assertion held. It says nothing about how a live model would answer today.

Outcome	Meaning	Fails the build
compatible	Same calls, inputs and end; assertions held.	no
diverged	A call does not match the recording: input_changed, unexpected_call or missing_call.	yes
raised	Your code raised where the recording did not.	yes
assertion_failed	The run matched, but a pinned assertion did not hold.	yes
error	The case could not be checked (bundle_missing, store_mismatch, target_not_registered, kind_mismatch, case_not_ready, recorded_error or internal).	yes (exit 3)
timeout	The case or the suite ran out of time.	yes (exit 5)

A tool that failed in the recorded run replays as that failure and is not itself a failure. Anything that is not your change, such as a flaky store, is an error, never a fail.

Set up

1. Save the runs to guard as cases, from the dashboard or pyrula case save (Saved cases).
2. Make a suite: a named list of cases for one project and environment. Cases join explicitly, so a new case never gates builds by accident.
3. Mint a CI token in Settings, API tokens (it can read the project and start and report suite runs). Store it as the secret PYRULA_TOKEN.
4. Store the fixture store URL as PYRULA_FIXTURE_STORE_URL. It is never sent to Cloud. A store that is not the one a case was saved to is store_mismatch.
5. Map each case’s target (a registered workflow or agent) to an import path in pyproject.toml:

[tool.pyrula.ci]
app_dir = "src"          # added to sys.path before targets are imported
deps = "app.deps:deps"   # optional: the deps object your agents expect
llm = "app.llm:client"   # optional: a replay-only run never calls it

[tool.pyrula.ci.targets]
support_agent = "app.agents:support_agent"
checkout = "app.workflows:checkout"

--target NAME=MODULE:ATTR overrides a configured target. An unlisted target ends as error with target_not_registered.

6. Add the Action. Install your project first; the check imports your code.

- uses: actions/setup-python@v5
  with: {python-version: "3.13"}
- run: pip install -e .
- uses: pyrula-io/pyrula/actions/replay-check@v0.1.0
  with:
    suite: checkout-regressions
    project: ${{ vars.PYRULA_PROJECT }}
    token: ${{ secrets.PYRULA_TOKEN }}
    fixture-store-url: ${{ secrets.PYRULA_FIXTURE_STORE_URL }}

The Action installs pyrula-cloud-client if absent (version from client-version), adds the Markdown report to the job summary, annotates each failing case, and sets outputs verdict, suite-run-url, json-path, junit-path. Other inputs: environment (default production), cloud-url, timeout, idempotency-suffix.

Outside GitHub (--environment defaults to production):

export PYRULA_TOKEN=...            # never a flag
export PYRULA_FIXTURE_STORE_URL=...
pyrula ci run --suite checkout-regressions --project proj_123 \
  --json report.json --junit report.xml --summary report.md

Timeouts and idempotency

Cases run one at a time, replay-only with a model-less runtime. --case-timeout defaults to 120 seconds; --timeout for the suite defaults to the deadline less a minute, and cases starting after it are timeout. Keep targets async: Python cannot interrupt synchronous code that never yields. --deadline-seconds (60 to 21600, default 1800) is how long Cloud holds the run open; a late report is refused. --verbose adds engine logs and tracebacks.

The run key is GITHUB_RUN_ID-GITHUB_RUN_ATTEMPT-GITHUB_JOB. Legs of one matrix job share it, so give each a distinct idempotency-suffix input (--idempotency-suffix) or the second leg’s report is refused. Outside GitHub pass --idempotency-key, or each invocation is a new run. A key already reported exits 3 with the run’s link. A suite run counts against the plan’s evaluation runs per month.

Exit codes

Code	Meaning
0	Every case is compatible.
1	Any case diverged, raised or assertion_failed.
2	Usage or configuration error, found before any network call.
3	A case could not be checked, with no failures. Also a run key already reported.
4	Infrastructure: Cloud unreachable, token or plan refusal, unreachable fixture store, or rejected report, with no failures.
5	A timeout, with no failures. Also a run key that expired before reporting.

When several apply, precedence is 2, 1, 4, 5, 3.

Fork pull requests

The job runs the PR’s code with your token and fixture store credential, so run it only on code you’d let read them. GitHub withholds secrets from a fork’s pull_request workflow, so the command sees no PYRULA_TOKEN and exits 4. Skip forks (if: github.event.pull_request.head.repo.full_name == github.repository) or gate behind a protected environment after reading the diff. Never use pull_request_target for this. Use a read-only fixture store credential, so a malicious change can’t alter the recordings that gate later builds.

Outputs

No output holds a recorded or produced value, only case names, targets, call keys, reason codes and counts, so public job logs are safe.

JSON report

--json writes schema: "pyrula.replay-ci/1":

{
  "schema": "pyrula.replay-ci/1",
  "check": "replay_compatibility",
  "statement": "Deterministic replay-compatibility check: ...",
  "verdict": "fail",
  "exitCode": 1,
  "suite": {"id": "suite_1", "name": "checkout-regressions"},
  "suiteRunId": "run_1",
  "url": "https://cloud.example/...",
  "project": "proj_123",
  "environment": "production",
  "startedAt": "2026-10-26T10:00:00Z",
  "finishedAt": "2026-10-26T10:00:05Z",
  "code": {"sha": "abc123", "ref": "refs/pull/7/merge", "pullRequest": 7},
  "pyrulaVersion": "0.1.0",
  "counts": {"total": 3, "passed": 2, "failed": 1, "errored": 0, "timedOut": 0},
  "cases": [
    {
      "caseId": "case_1", "name": "refund flow", "kind": "agent", "target": "support_agent",
      "fixtureHash": "9f2c...", "outcome": "diverged", "reasonCode": "diverged",
      "reasonMessage": "tool:toolu_01: called with different input",
      "divergence": {"key": "tool:toolu_01", "reason": "input_changed"},
      "calls": {"reused": 3, "simulated": 0, "unexpected": 1, "unconsumed": 0},
      "assertions": [{"index": 0, "op": "status_completed", "passed": true, "reason": null}],
      "durationMs": 12
    }
  ]
}

verdict is pass, fail, error, timeout or infrastructure. A run that never started has no cases and an error object with code and message. calls.unconsumed counts recorded calls the code never made.

JUnit

One <testsuite> named for the suite and one <testcase classname="SUITE.TARGET" name="CASE"> per case. A fail is a <failure type="diverged:input_changed"> (or raised:CODE, assertion_failed); an error or timeout is an <error type="REASON">.

Ordering assertions

actions_match_original and action_not_executed compare the run’s actions with the original’s, which Cloud supplies at start. Tool calls count as tool:NAME, workflow writes as step:ID, and a call served from the recording is reused, so a recorded write never counts as executed. If Cloud no longer has the original history, the case is an error.

See also

Saved cases covers pyrula case run --replay-only for one case locally. Versioning and deploys covers replay_check, which asks whether in-flight runs can resume on new code.