agent-reliability-ci

Did this change make your agent less reliable?

ARCI (agent-reliability-ci) runs your baseline and candidate repeatedly under the same predeclared tasks and injected tool faults, grades each run with an independent success check, returns PASS, BLOCK, INCONCLUSIVE or ERROR with an exit code, and can replay a recorded failure offline.

Try it without cloning (seeded fixture, no model call):

uvx --from agent-reliability-ci arci demo

Replace these example references with your project's importable Python functions:

uv tool install agent-reliability-ci && arci init checkout-retry --baseline app.v1:run --candidate app.v2:run --toolset tests.world:make_world --oracle tests.world:passed && arci run arci.yaml

baseline + candidate → same predeclared tasks/faults → repeated trials → verdict → minimise/replay

PASS means only this: for this predeclared experiment, the evidence supports that any reliability drop from baseline to candidate is smaller than the allowed drop. It is not a claim of absolute reliability.

A small first run will often be INCONCLUSIVE. That is a valid result; use arci plan before choosing a larger confirmatory run.

ARCI observes calls that cross its tool boundary. It is not a sandbox, it does not replay model inference, and trace divergence shows where runs differ, not why.

Try the offline demo

Python 3.11+, POSIX and uv. Installation needs network; the fixture makes no model calls.

uvx --from agent-reliability-ci arci demo

arci demo runs a packaged copy of the seeded inventory fixture (arci.demo): the same world, agents, seeds and sample size as the recorded run below, so it computes 192/200 baseline successes and 132/200 retry-removed successes fresh on your machine and says whether they matched the recorded run. It is a reproduction, not the recorded evidence. The six steps go through the real CLI and the Python API, each trial in a fresh process with a separate grader process. Runtime depends on the machine and current load (measured: 38 s on a 15-core Apple Silicon Mac with 8 workers; 72 s through uvx from a local wheel including environment creation; CI runners are slower). just demo runs the checkout script examples/retry_agent/hero_demo.py, the same six steps. Lines from the recorded run:

=== 2. The frozen experiment, N=200 per arm, fault: tool_timeout on reserve
  VERDICT: BLOCK (exit 1)   [A vs agent_b]
  VERDICT: PASS (exit 0)   [A vs agent_a]

=== 3. Where do a passing and a failing run part ways?
  left: tool_start tool="reserve" arguments={"order_id":"order-7","quantity":2,"sku":"widget"}
  right: tool_start tool="confirm" arguments={"order_id":"order-7","sku":"widget"}
  after_injection: tool_timeout

=== 4. Shrink the failing condition (3 injected faults)
  kept: tool_timeout
  removed: empty_result, tool_error_once
  minimality: 1-minimal

=== 5. Replay the reduced bundle offline
  REPRODUCED: failure reproduced

Copied from the unedited, committed log hero-demo-2026-10-05.txt, without the ok lines. Step 1 passes both agents on one clean run; step 6 passes the repaired agent C on the exact reproducer and on its own frozen experiment; the run ends All six steps held, exit 0. The experiment is also a frozen acceptance test (test_hero.py).

Agent B is not rigged with dice. It is agent A with the retry around reserve removed, a plausible refactoring slip: when reserve times out it carries on to confirm and still reports success. Its failure rate comes from the environment. Reservations already exist with configured probability 65% (135 of the default 200 seeds, plus 8 scenarios with a naturally flaky confirm; shares set in world.py, counts asserted in test_retry_example.py), and in those the missing retry never matters. That is exactly why one run hides it.

Recorded result and evidence

Recorded fixture: 200 trials per arm; timeout on reserve. Baseline: 192/200. Retry removed: 132/200. Verdict: BLOCK (exit 1). Recorded results · Trace, reduction and replay

Evidence: 18 archived decisions re-derived · Frozen contracts · Design records · CI and macOS statistics checks

Re-derived means each archived decision.json regenerates byte for byte from its committed inputs, whatever its verdict: reproducing a BLOCK, INCONCLUSIVE or ERROR counts, and no archived run is turned green (reproduction recipe, CITATION.cff).

Review a retry-policy change before merging

In the recorded inventory-reservation fixture, the candidate loses its retry after a timeout. ARCI reports BLOCK, locates the trace divergence, reduces three injected faults to one, and reproduces the failure offline. The repaired candidate passes the reproducer and its own separately frozen experiment.

Scaffold your own experiment with arci init (below), or start from the example’s manifest builder. Define the oracle, fault conditions, margin and sample size before running. A small experiment may remain INCONCLUSIVE.

Manifest builder of the recorded run: examples/retry_agent/experiment.py.

Verdicts and exit codes

Every gating command ends in a verdict and an exit code. Nothing else decides.

An agent crash, timeout, blown budget or uncaught tool fault is a FAIL. A broken environment, injector, grader or event sink is an ERROR, and it invalidates the experiment instead of quietly leaving the denominator. arci replay uses its own codes: 0 reproduced, 1 not reproduced, 3 invalid.

How it works

In words: the manifest (task, toolset, oracle, contract, both arms, fault conditions, alpha, delta, n_per_arm, seeds) is sealed before the run. Each trial runs as a Python worker or a command agent behind an MCP boundary, with a separate grader process; the tool boundary applies budgets and seeded faults and records every call into sealed trials.jsonl. The gate is a pure function of manifest and trials. Diagnosis reads the same records, whatever the verdict: diff (first divergence), minimize (ddmin over injected faults, same failure fingerprint), bundle and replay (re-serves recorded boundary calls: REPRODUCED, NOT_REPRODUCED or INVALID).

A Python agent is a plain function, agent(task, tools, rng) -> dict. It calls tools.call("name", ...) and may annotate its reasoning with tools.note_model_step(...). Success is decided by an oracle over the environment's final state, never by what the agent says about itself.

A gate that can say "I do not know". The rule is fixed-sample and deliberately boring (docs/STATISTICS.md): per arm, an exact Clopper-Pearson interval at tail alpha / (4K); bounds on the difference by Bonferroni; PASS iff the lower bound is above -delta, BLOCK iff the upper bound is below it. No peeking, no extending a run, no rerunning until green. The decision rule is bound into every trial's identity, so a manifest resealed after the run with a different rule no longer matches its trials and the gate returns ERROR: choosing the rule after the data is detected, not merely forbidden. Earlier runs on the same candidate are listed in prior_runs and printed in the decision and the Markdown report; arci does not discover them or enforce a cross-run error budget for you. Wilson intervals are shown for readability and never decide.

Command agents. An agent does not have to be a Python function: it can be any program that speaks MCP over stdio. arci owns the one MCP server behind it, so every tool call is recorded, budgeted, fault-injected and replayable from outside the agent's process.

your agent (any argv) --> arci.mcp_shim --> arci.mcp_boundary --> MCP server (the environment)
                          byte relay        recorder, faults,     yours, or any python toolset via
                                            budgets, latches      python -m arci.mcp_toolset_server

docs/REAL_AGENTS.md has the recipe and the local-model worked example.

Decision calls. A command agent may call a System One decision endpoint (TypeSafe's Jev API). The same boundary serves POST /v1/systemone on loopback, records each admitted request as a tool call named decision:systemone, and applies budgets, perturbations, diff, minimisation, bundles and exact replay to it. The agent is redirected with TYPESAFE_BASE_URL and a per-trial token while any real key stays in the harness; the official Python and JavaScript SDKs are exercised unchanged by the acceptance tests. arci preflight checks key, endpoint, pinned model and one smoke decision before a live run.

Run against Jev on 2026-10-03. Built from documentation dated 2026-09-22, the boundary ran the pre-registered day one live. Four things did not survive contact and are fixed: alias-only model lists (arci preflight --allow-unlisted-model), smoke questions needing instructions, CONNECT-only egress (ARCI_HTTPS_PROXY), and the console script needing the repository root on PYTHONPATH. Prices and provider limits are still as documented, not measured, and the gateway route is still untested. See docs/DECISIONS.md.

Where it has been run

All runs are the maintainer's own; no external users yet. Failed and invalid attempts stay in the linked records.

• Seeded retry demo (2026-09-23): the result above; A against itself and the repaired C both PASS (retry-demo-n200.md).
• Support-triage agent, live Jev endpoint (2026-10-03, jev-1.13.0 pinned, Python and Node SDKs unchanged; planted regression, the harness caps confidence and B stops escalating): A vs B at N=200, 200/200 vs 0/200, bounds [-1.000, -0.957], BLOCK; one minimised failure replays REPRODUCED; 902 recorded decisions (docs/results/jev). Offline against a seeded fixture: .venv/bin/python examples/jev_triage_agent/hero_demo.py --n 50.
• The tool-call guard of the maintainer's other project, frontier-scout (2026-10-05, hook_runtime.py verbatim at 58b0b17, run through the harness boundary, not inside a coding-agent session; outages injected): fail-closed against fail-open under decision_unavailable at N=200, 200/200 vs 130/200, bounds [-0.4304, -0.2537], BLOCK; diffed, minimised (reduced) and replayed REPRODUCED (docs/results/guardrail).
• A local model, one sentence of prompt (qwen3.5:4b-mlx through Ollama, one machine, exploratory): N=400, 376/400 vs 308/400, bounds [-0.2445, -0.0921], INCONCLUSIVE; BLOCK needed the upper bound below -0.10 and missed by 0.008. Not re-run until it blocks (docs/results).

What went wrong and was kept.

• Ollama run 1: a harness bug (tools advertised with no parameters), found from the failure clusters (ollama-run1-n30-bridge-bug.md). Run 3: the model server died and one ERROR trial invalidated the run (ollama-run3-n400-invalid.md); hence infra_exit_codes (75 by default), which turns an agent's own infrastructure failure into ERROR.
• Jev attempt 1: every trial crashed on an import error, so equally broken arms "passed"; the invalid stores are kept in attempt-1-invalid.
• Guardrail at N=50: 50/50 vs 38/50, bounds [-0.4016, -0.0348], INCONCLUSIVE against the pre-registered BLOCK, reported as is (docs/results/guardrail).

Is the model's confidence something you can gate on? A separate, pre-registered audit of jev-1.13.0 on CLINC150 and Banking77 answers that for intent routing, not permission calibration: zero-shot accuracy 0.921 with ECE 0.024 on CLINC150, and 0.801 with ECE 0.084 on Banking77, where confidence is over-stated in the middle of the range. Write-up, full table and limits: docs/results/jev-calibration; protocol and code: bench/jev_calibration.

The gate, measured against itself

just selfcheck (or .venv/bin/python bench/selfcheck.py) computes the gate's own operating characteristics by exact enumeration (alpha 0.05, delta 0.10, one condition). Probabilities of PASS / BLOCK / INCONCLUSIVE:

• False PASS at the margin and false BLOCK under no change are both far below alpha (worst 0.004 and 0.001 over the boundary sweep). The price is conservatism: two equal agents at 80% reach PASS only 21.8% of the time at N=200.
• Twenty runs decide almost nothing. At N=20 a 95% -> 65% collapse is still INCONCLUSIVE 99.2% of the time, and showing a failure rate below 0.5% needs at least 598 clean trials. A twenty-point drop from 0.95 to 0.75 BLOCKs only 36.5% of the time at N=200.
• interval_method: newcombe (about twice the power, calibrated by exact enumeration rather than proved) and pre-registered looks must be frozen before the run; arci plan shows a design's verdict probabilities before you spend trials. Details: docs/STATISTICS.md, report section 3.

Gate your own agent

Scaffold one experiment from four references, then run it:

uv tool install agent-reliability-ci
arci init checkout-retry --baseline app.v1:run --candidate app.v2:run --toolset tests.world:make_world --oracle tests.world:passed
arci run arci.yaml

arci init imports the four references once (in a child process, as a trial would), checks that every fault names a tool the toolset provides, and writes arci.yaml: a thin, schema-checked descriptor that arci run seals into the same JSON manifest the Python builder produces. The scaffold is a smoke design (8 trials per arm): under the default rule two perfect arms cannot PASS below N=42 and a total collapse cannot BLOCK below N=8, so expect INCONCLUSIVE, then plan a confirmatory N with arci plan before spending trials. Without --fault the scaffold writes one clean condition (no injected fault); --fault tool_timeout:reserve adds a timeout on the first reserve call, and --task '{...}' sets the task. A hand-written descriptor with one fault looks like this:

experiment_id: checkout-retry
task: {order_id: "order-7"}
toolset: tests.world:make_world
oracle: tests.world:passed
baseline: app.v1:run
candidate: app.v2:run
conditions:
  - id: reserve-timeout
    fault: {name: tool_timeout, tool: reserve, occurrence: 0}
alpha: 0.05
delta: 0.10
n_per_arm: 8

The descriptor expands documented shorthands (id, fault, occurrence, oracle) into a sealed manifest; the key reference lists its defaults. arci schema prints the JSON Schema, and docs/DESCRIPTOR.md has the key reference, command-agent experiments and the YAML quoting rule. The toolset is (task, seed) -> object with .tools and .snapshot(), the oracle (task, final_state) -> bool, and each agent a plain function agent(task, tools, rng) -> dict.

Advanced: the Python manifest builder

A manifest is a sealed JSON document built in Python. The retry example's builder, examples/retry_agent/experiment.py, is the template; its shape:

Manifest.create(
    experiment_id="retry-agent-agent_b-reserve-timeout-200",
    task_id="confirm-inventory-order",
    task={"order_id": "order-7", "sku": "widget", "quantity": 2, "stock": 10},
    toolset="examples.retry_agent.world:make_world",  # (task, seed) -> tools + snapshot
    contract=ContractSpec(
        oracle="examples.retry_agent.world:oracle"
    ),  # (task, final_state) -> bool
    baseline=ArmSpec(label="agent_a", agent="examples.retry_agent.agents:agent_a"),
    candidate=ArmSpec(label="agent_b", agent="examples.retry_agent.agents:agent_b"),
    conditions=(
        Condition(
            condition_id="reserve-timeout",
            faults=(
                FaultSpec(
                    name="tool_timeout", bucket=Bucket.FALSIFY, tool="reserve", at_occurrence=0
                ),
            ),
        ),
    ),
    n_per_arm=200,
    base_seed=12_000,
    budgets=Budgets(max_tool_calls=10, max_model_steps=10, max_seconds=15.0),
)

A behaviour contract names the oracle plus the invariants the tool boundary can observe (max_tool_calls, forbidden_tools, required_tools, max_calls_per_tool). A contract that asks for something the boundary cannot see is rejected, not silently skipped. Fault conditions are tagged falsify (a robust agent should survive), benign (must not change the outcome) or ceiling (known-unrecoverable: reported, never gated on rates). Built in: tool_timeout, tool_error_once and empty_result at the tool boundary, plus decision_low_confidence and decision_unavailable at the decision boundary; a "module:function" reference plugs in your own. For a command agent, replace toolset with mcp_server and the arms' agent with command, as docs/REAL_AGENTS.md shows.

arci init NAME --baseline M:F --candidate M:F --toolset M:F --oracle M:F   # write arci.yaml (smoke N=8)
arci schema                                        # the descriptor's JSON Schema, generated from code
arci demo                                          # the packaged six-step retry story, no clone
arci plan --help                                   # verdict probabilities of a design, before any trial
arci preflight manifest.json                       # live decision upstreams: key, model, smoke call
arci run arci.yaml --workers 8                     # run + gate; exit code is the verdict (--out defaults to runs)
arci gate runs/<experiment> --junit junit.xml --markdown report.md
arci report runs/<experiment>                      # gate, failure clusters, tokens, estimated cost
arci diff runs/<experiment> <passing_trial> <failing_trial>    # --all-steps to include annotations
arci minimize runs/<experiment> <failing_trial> --out min.json
arci bundle runs/<experiment> <failing_trial> --out bundle.json --root . --include my_agent.py
arci replay bundle.json                            # 0 reproduced, 1 not reproduced, 3 invalid

A run store is one directory per experiment with the sealed manifest, events.jsonl and trials.jsonl. Keep it with the evidence it supports; it can hold the agent's full task state and, for decision calls, the full request state, so treat stores and bundles as sensitive data. Records sealed by an earlier schema version are not readable after an upgrade (v0.2 and v0.5 changed the schema); re-run the experiment rather than trust an unreadable store.

GitHub Actions

This example pins v0.8.0, the released feature set (the Action installs that tag's source; the same code is on PyPI as agent-reliability-ci==0.8.0). See CHANGELOG.md.

- uses: actions/setup-python@v7
  with: { python-version: "3.12" }
- run: pip install -e .            # your agent, importable
- uses: ajaysurya1221/agent-reliability-ci@v0.8.0
  id: gate
  with:
    manifest: reliability/manifest.json
    workers: "4"                   # out: arci-runs by default
- run: echo "verdict=${{ steps.gate.outputs.verdict }}"

This assumes your agent's repository is checked out and installed, and that you pin action versions you have verified. The report lands in the job summary (a start-up failure shows only in the step's stderr). BLOCK and ERROR always fail the job; INCONCLUSIVE fails it unless you set allow-inconclusive: "true". The verdict output (PASS, BLOCK, INCONCLUSIVE or ERROR) is there for steps that run after the gate. This repository's own CI runs the test suite, re-derives the archived decisions, smoke-tests the built wheel on Linux and macOS, and runs the Action on a committed fixture that must BLOCK.

What it does not do

• It is not a sandbox and not a security boundary. Python agents share a process with their tool boundary. Command agents use a separate harness-owned MCP boundary, which is isolation from accident, not containment. Both assume buggy, non-hostile agent code; a hostile agent could still forge or bypass its record. See docs/TRUST_MODEL.md.
• It only sees the tool boundary. Files, network and subprocesses the agent touches directly are invisible, as is any decision call that does not go through the injected base URL.
• It replays the boundary, not the model. Deterministic replay is guaranteed for seeded, fixture-backed agents. A live model will not repeat itself token for token, so replaying its bundle is usually INVALID; what survives is the recorded trace and the reduced fault condition.
• INCONCLUSIVE is common at small N. That is the honest answer, not a failure of the run: at N=20 even a 95% -> 65% collapse is INCONCLUSIVE 99.2% of the time, and one real 400-trial run missed BLOCK by 0.008 and stayed INCONCLUSIVE.
• A bundle embeds code, and replaying it runs that code. Hashes are integrity checks, not signatures. Replay bundles only from sources you trust. A bundle embeds only the files you --include (the minimiser's output embeds none), so portable replay needs the referenced code and a compatible environment.
• A divergence is evidence, not proof of cause. It shows where two runs part ways.
• The decision boundary has been run against one vendor model, on one day, on three seeded tickets. The live model answered every clean decision with confidence 1.0, so the runs in docs/results/jev show the harness, the pacing, both SDKs and the gate working end to end and the planted regression caught; they say nothing about the model's calibration or about ambiguous tickets. Live answers are sampled, so live minimisation is reported as reduced, never 1-minimal, and a decision model's confidence never enters a verdict except through the agent's actions.
• Reference validation executes user code. arci init validates Python references by importing them, and calls the toolset factory once, in a child process; import-time code runs.
• Declaring a fault does not guarantee injection. A fault names a tool; if the agent never calls that tool at that occurrence, the condition runs clean and nothing says so. Inspect the recorded calls to confirm injection.
• Command agents get one stdio MCP server (revision 2026-07-28), serial tool calls, no HTTP transport, no concurrent decisions.
• No importers (OTLP, Claude Code, Codex), no HTML report yet. POSIX only: on Windows the package installs and arci --help runs, but experiments are unsupported.

Related work: Documentation comparison, checked 2026-10-05; not a matched benchmark.

Independence. ARCI's job is preregistered stochastic regression testing. The manifest is a prospective contract supplied by its author: it fixes the task, fault conditions, sample size and decision rule before the trials run, and a rule changed after the fact is detected, not merely forbidden. It does not independently authenticate when the manifest was written or that every earlier run was disclosed; that discipline stays with the author. A PASS reports relative non-inferiority against the baseline within the registered margin under those conditions, never absolute reliability or a general quality score; BLOCK, INCONCLUSIVE and ERROR are defined above. ARCI is one of the maintainer's independent projects alongside the earlier evalopt-graph experiment, whose standalone feature development has ended. ARCI declares no package or runtime dependency on evalopt and imports nothing from it, and it shares no runtime layer with the maintainer's sibling projects Actseal, Dorian and Frontier Scout. NOTICE records two adaptations from evalopt under MIT with attribution: the canonical JSON hashing helpers in src/arci/hashing.py and the self-sealing decision pattern in src/arci/schema.py, both attributed source adaptation inside the frozen manifest, not a dependency. ARCI is not a successor to, conformance target for or drop-in replacement for evalopt, and nothing here claims adoption, performance or superiority.

Roadmap

Most valuable first: per-request pacing through parent IPC; counterfactual replay of recorded decisions, if it can be done without overstating what a re-perturbed recording proves; extending the intent-routing calibration audit to labelled coding-agent permission decisions, with a separate protocol and explicit limits. Full list in docs/ROADMAP.md.

Development

How it was built. One maintainer, September to October 2026, with Claude Code and Codex as pair-programmers (recorded in the commit messages). The experiment design, the decision rule and its calibration, the trust model and the acceptance tests are the maintainer's own; the retry demo, the calibration table and the test suite reproduce from a clean clone. Issues and PRs welcome.

uv sync
just check        # ruff format --check, ruff, basedpyright strict, pytest
just frozen       # the frozen contract files are unchanged
just selfcheck    # the calibration table above
just demo
npm ci --prefix examples/jev_triage_agent   # only for the Node agent and its test (Node 20+)
python docs/assets/src/make_figures.py --check   # the README figures match their generator

The acceptance suite is hash-pinned in FROZEN.sha256. CI checks formatting, lint, strict typing and tests on Python 3.11 and 3.14, with separate macOS statistics checks. See the workflow for coverage and the evidence report for measured results. Design records live in docs/design, the changelog in CHANGELOG.md.

Availability and project status

Availability: v0.8.0 is on PyPI as agent-reliability-ci and as the Action tag v0.8.0; it adds the packaged demo (arci demo), the descriptor (arci.yaml, arci schema), arci init, the planner (arci plan), opt-in pass^k reporting (arci report --pass-k) and portability fixes over v0.7.0. Boundary: PASS means relative non-inferiority, not absolute reliability. This is not a sandbox.

Security policy

Apache-2.0. See NOTICE for adapted code.