agent-crucible

Agent Crucible

A statistically rigorous A/B testing tool for measuring whether Agent Skills actually improve task outcomes. Deterministic assertions first, repeated runs, a corrected statistical comparison, and cost-per-success — for Claude Code.

A SKILL.md is a prompt you ship to other people. This tool answers the only question that matters about one: does it actually make the agent better, or does it just feel like it does?

It runs the same task twice — once with a baseline agent, once with the skill installed — repeats that N times, checks the results with deterministic assertions, and reports the difference with a confidence interval attached.

Real results are in BENCHMARK.md — including a skill that only becomes conclusive at N=10, two skills with no measurable effect, and an honest surprise about what trigger detection can and cannot see.

                    Without       With
Task success            50%       100%
Assertion score        0.83       1.00
Tokens                8,700      7,100
Time                     55s        41s

IMPACT   (95% bootstrap CI on the delta)

  Task success        +50.0pp   [+20.0, +80.0]   p=0.033
~ Tokens              -23.0%    [-41.2%, +2.1%]

CONFIDENCE  MEDIUM  (N=5 per condition)
~ marks a delta not distinguishable from run-to-run noise.

PER TASK  (success rate)

task-001                  0%       100%
task-002                100%       100%

The per-task breakdown matters: an aggregate +50pp driven entirely by one task is a different finding from one spread across all of them.

Install

pip install agent-crucible

Or from a clone: pip install -e .

Requires Python 3.10+, git, and an authenticated Claude Code CLI on PATH. Tested on Linux, macOS, and Windows (CI runs the suite on all three across Python 3.10–3.12). Verify the agent side works before benchmarking anything:

claude -p "reply with OK" --output-format json

If that returns "Failed to authenticate", log in first — every run in this tool shells out to that same command.

Use

crucible init my-skill        # scaffold SKILL.md + evals/
crucible validate my-skill    # static checks, no tokens spent
crucible test my-skill        # the A/B benchmark
crucible report my-skill      # re-render the last run
crucible regression my-skill  # did the latest run drop below the baseline?

1. Validate

Runs before every benchmark, and refuses to spend tokens on a broken suite.

  [ok]   SKILL.md exists
  [ok]   valid YAML frontmatter
  [ok]   frontmatter has 'name'
  [ok]   frontmatter has 'description'
  [ok]   referenced files exist (2/2)
  [ok]   [task-001] prompt is customised, not the scaffold placeholder
  [warn] model 'sonnet' is a pinned version (aliases drift between releases)

2. Write eval tasks

id: task-001
name: Make calc errors typed

prompt: |
  The test suite in tests/ fails. Fix src/calc.py so every test passes.
  Do not modify anything under tests/.

fixture: fixtures/task-001

assertions:
  - type: command          # did it solve the task?
    run: "python -m pytest -q"
    expect_exit: 0

  - type: file_contains    # did it follow the convention the skill teaches?
    path: src/calc.py
    pattern: "raise CalcError\\([^)]*code="

  - type: file_unchanged   # anti-gaming: editing the tests does not count
    path: tests/test_calc.py

Assertion types: command, file_exists, file_absent, file_contains, file_changed, file_unchanged.

Before burning tokens, check two things — neither spends a cent:

crucible test my-skill --dry-run    # can the suite even measure work?
crucible test my-skill --estimate   # what will a real run cost?

--dry-run runs the assertions against the untouched fixture. Anything that passes there is flagged [TRIVIAL] — it would pass even if the agent did nothing. (Assertions that check for the absence of something — file_absent, file_unchanged, or a file_contains with expect: false — are guards, and correctly pass on the pristine fixture.)

--estimate projects cost and wall time as a range. With no history it is a coarse prior; after one real run it uses your own recorded per-run cost. A real test prints the estimate first, then runs.

3. Benchmark

crucible test my-skill -n 10

How it works

Per run:

1. The task fixture is copied into a throwaway temp directory.
2. That directory is git init-ed and committed, which is what makes file_changed / file_unchanged assertions possible.
3. In the with-skill condition only, the skill is copied to .claude/skills/<name>/ inside that workspace. The evals/ directory is excluded — leaking it would hand the agent its own grading criteria.
4. claude -p "<prompt>" --output-format json runs with the workspace as cwd, with --setting-sources project. (Note: in practice this does not fully hide your personal ~/.claude skills — see "Known gap" above. They appear in both conditions, so the delta stays attributable to the skill under test.)
5. Assertions run against the resulting workspace; tokens, cost and wall time come from the agent's own JSON result.
6. The workspace is deleted (--keep-workspace to inspect it).

Results land in <skill>/.crucible/runs/*.json, with one summary line per invocation appended to <skill>/.crucible/history.jsonl.

Design decisions worth knowing

No LLM judge. Scoring is deterministic: assertions pass or they don't. An LLM grader adds a second noisy component on top of an already noisy agent, and you can no longer tell which one moved. Quality rubrics are a later addition, not the foundation.

Every delta carries an uncertainty estimate. Agent runs are non-deterministic; the same prompt gives different token counts and sometimes different outcomes. A single number invites false regression alarms. Anything not separable from noise is printed with ~ and called inconclusive.

One primary metric decides the verdict; the rest are secondary. The report is split on purpose. Task success is the single pre-registered primary outcome — it alone drives the verdict and the CI gate, judged by Fisher's exact test (p<0.05). Everything else (assertion score, tokens, time, cost) is secondary: reported, but never the thing that passes or fails a run.

Task success uses Fisher's exact test, not the bootstrap. This was forced by testing the tool against a deliberately noisy stub with a weak 60%-vs-50% effect: the percentile bootstrap called that a conclusive regression on the first try, because five Bernoulli samples per condition is exactly where it is anti-conservative. Fisher's exact test on the 2×2 table has no such problem, and still detects a real effect at N=5 (0/5 vs 5/5 gives p=0.008).

Secondary metrics are Holm-Bonferroni corrected. Testing four secondary metrics at 0.05 each inflates the false-positive rate — roughly one report in seven would carry a bogus secondary claim. Each secondary metric gets a permutation p-value, Holm-corrected across the family; a secondary claim is only made on the adjusted p. The verdict never rests on them regardless. (The tool's own maths — Fisher, Holm, permutation, cost-per-success — is pinned to known values in tests/test_statistics.py, so the evaluator is itself under test.)

Cost-per-success, not just success. A higher success rate bought with much more spend is not obviously a win. The report shows cost, tokens, and time per successful task for each condition, so a skill that lifts success 10% while tripling cost-per-success is visible as the trade it is. It stays secondary — different users weight cost differently.

Held-out evaluation. Iterating skill → benchmark → skill overfits the skill to its own eval suite. Define task groups in evals.yaml and confirm the effect on a set you never tuned against:

groups:
  development: [task-001, task-002]
  holdout: [task-003]

crucible test my-skill --group development   # iterate here
crucible test my-skill --group holdout       # confirm here, untouched

Confidence is labelled by sample size. N<5 is LOW and says so in the verdict. The tool will not pretend five runs are evidence — at N=1 it reports the observed delta and then explicitly refuses to conclude from it.

Pin your model. model: sonnet is an alias that moves between releases. A comparison against a run from last month is meaningless if the model changed underneath it. validate warns about this.

The baseline carries Claude Code's bundled skills. They appear in both conditions, so the delta stays attributable to the skill under test, and they are identical across installs of the same version. isolation: bare removes them for a clean-room baseline (needs ANTHROPIC_API_KEY).

This is decision support, not an oracle. --strict exists for CI, but consider making the CI job a warning rather than a hard block, and gate it on SKILL.md actually having changed — every invocation spends real tokens. N tasks × N repeats × 2 conditions agent runs is not free the way a unit test is free.

Examples

Complete, runnable skills live in examples/, each mixing a "did it work" command assertion with a "did it the right way" content assertion, and each carrying trigger cases:

crucible validate examples/sql-parameterization
crucible test examples/sql-parameterization --dry-run
crucible test examples/sql-parameterization --estimate
crucible test examples/sql-parameterization -n 5

Each positive example is built so the difference is visible: a baseline agent usually makes the tests pass, but only the skilled agent reliably follows the convention. "Tests pass" and "did it the right way" are different questions — which is exactly why a single assertion is not enough.

negative-control is the opposite on purpose: its skill only changes docstring wording, so it cannot affect whether the tests pass. Running it should report no conclusive effect — a measurement instrument you can trust reads zero when the effect is zero.

CI

examples/ci/skill-eval.yml is a GitHub Actions template. It triggers only when SKILL.md or evals/ changed, and posts the report as a PR comment instead of failing the build.

Validated against a live agent

Run end-to-end against the real claude CLI, not just a stub. Two things the example run demonstrated, both worth internalising before you trust any number:

• A lucky small sample lies, and the tool catches it. At N=3 the example skill scored a clean 3/3 vs 0/3 — an apparent +100pp. At N=5 the same skill, same prompt, scored 1/5 (+20pp, p=1.000). The effect is real but weak and inconsistent; three runs happened to catch a good streak. A tool that reports one number would have published "+100% success". This one reported "not distinguishable from noise" and asked for more runs. That is the whole point.

• Availability is not invocation. A skill in .claude/skills/ is discovered by the agent but only used when its description matches the task. The prompt "make the tests pass" did not trigger an error-handling skill; "add proper error handling following this project's conventions" did. Your eval prompt is a real variable — the same skill looks useless or useful depending on it.

Baseline isolation, found in the same run: the headless agent always sees the skills that ship with Claude Code (design, code-review, …). They are not your personal ~/.claude skills — they are bundled with the install, so they are the same for anyone on the same Claude Code version, and they appear in both conditions, cancelling out of the delta. For a true clean-room baseline set isolation: bare in evals.yaml (runs the agent with --bare); it needs ANTHROPIC_API_KEY, since bare mode never reads your OAuth login. The default keeps the bundled skills and works with OAuth.

4. Regression

Each test writes a run record. regression compares the newest run against an earlier baseline — by default the most recent run whose skill content differs, so it answers "did my last edit make the skill worse?"

crucible regression my-skill

With-skill success:  86%  ->  62%   (-24pp, p=0.021)
--------------------------------------------------------
x  REGRESSION — with-skill success dropped beyond run-to-run noise.

Crucially, it is judged by the same Fisher exact test as the benchmark, not by subtracting two scores. A drop from 100% to 80% on five runs prints ~ PASS (inconclusive) with p=1.000, because one flipped run is exactly what noise produces — a naive point-vs-point check would have raised a false alarm. It exits 1 only on a conclusive drop, so a CI gate blocks a genuine regression without failing on the dice:

- run: crucible test ./my-skill -n 10
- run: crucible regression ./my-skill   # exit 1 blocks the merge

5. Trigger evaluation

A skill can be excellent when used and still be worthless because the agent never reaches for it — or noisy because it fires on unrelated prompts. trigger measures that directly. In evals/triggers.yaml you label prompts:

cases:
  - prompt: "Add error handling to src/calc.py."
    should_trigger: true
  - prompt: "What is 27 times 34?"
    should_trigger: false

crucible trigger my-skill

  Precision (when it fired, was it right): 100%   FP=0
  Recall (of prompts that should fire it):  75%   FN=1
  ! 1 prompt that should have used the skill did not — available but not reached for.

Detection is deterministic, not an LLM judging "was it probably used": the run uses --output-format stream-json, and a trigger is a Skill tool-use event for this skill in the stream.

Scope, learned from real runs (see BENCHMARK.md): this measures explicit Skill-tool invocation. That is the right signal for skills designed to be invoked as a procedure. But Claude Code activates a simple convention skill by putting it in context, not by making the model call the Skill tool — so such a skill can measurably change behaviour (proven by the A/B test) while its explicit-invocation rate is zero. Read trigger recall as a lower bound on influence, not as "the skill did nothing"; the A/B test is what measures influence. The false-positive side is still meaningful: a skill firing on unrelated prompts is a real defect this catches.

Real results

Four skills were run against a live agent (Claude Code, claude-sonnet-4-5, N=5). The full write-up with methodology and raw data is in BENCHMARK.md; the short version:

• No skill reached statistical significance on task success at N=5. Two showed promising but inconclusive swings (0→40%, 40→80%, both p≈0.5); the tool reported them as inconclusive rather than publishing the point estimate.
• The negative control read exactly zero (0.00 delta, p=1.000) — the instrument does not manufacture improvements.
• The only conclusive deltas were token costs — including a skill whose advice the baseline agent already followed, so it added tokens and nothing else.
• Trigger recall was 0–50%: the skills often were not invoked even when clearly relevant, which likely explains the weak outcome effects. Availability is not invocation.

This is the intended behaviour: run it on your own skill and it will tell you the truth, including "no effect" or "costs more than it helps."

Reproducibility

Every test writes a full run record to .crucible/runs/*.json — not just the summary, but every per-run outcome, and an environment block capturing what the numbers depend on: tool version, agent (claude) version, model, OS, Python version, and separate fingerprints for the skill and the eval suite. The suite fingerprint exists because a result can flip when you reword a prompt without touching SKILL.md; comparing across a changed suite fingerprint is comparing two different experiments. crucible report prints the environment block so a reader knows the conditions.

Limitations

Read these before quoting any number from this tool.

• Results are model-, harness-, task-, and environment-dependent. A correct claim is "on this eval suite, under this model and Claude Code version, the skill raised task success by N percentage points" — not "this skill makes the AI N% better". The run record stores exactly those conditions so the scoped claim is the easy one to make.
• A benchmark is only as good as its eval suite. The tool measures whether the agent satisfied your assertions on your tasks. A weak or unrepresentative suite produces a confident, meaningless number. --dry-run guards against assertions that pass on the untouched fixture; nothing guards against tasks that don't represent real use.
• Cost. Each run is a real agent session spending real tokens. N tasks × N repeats × 2 conditions is not free the way a unit test is; --estimate prints the projected spend before you commit.
• Small N is a smoke test, not evidence. Below ~5 runs per condition the tool labels the result LOW confidence and refuses to call effects conclusive, on purpose. --estimate prints a power-vs-cost table so you pick N from the effect size you expect: N=5 can only detect near-total effects (≥80pp even in the best case), N=10 reaches ~50pp, ~20 reaches ~25pp.
• The suite is your responsibility. The tool proves a skill changed an outcome; it cannot know the tasks represent real use. See docs/writing-good-evals.md — the honest limit of the whole approach, and how to work within it.

Status

Claude Code only: init, validate, test (with --dry-run, --estimate, --strict), report, regression, trigger. Tested on Linux/macOS/Windows across Python 3.10–3.12; the test suite stubs the agent, so CI needs no API key.

Roadmap — as separate layers, not core

These are deliberately not in core, and not because they're unimportant. The differentiator is a small, deterministic, statistically honest A/B engine; bolting these into it would dilute exactly that. They belong in optional layers:

• Cross-agent (crucible compare across Claude Code / Codex / Gemini) — the biggest future expansion, but it needs an agent-adapter abstraction, transcript normalization, and cross-agent comparability. A project of its own.
• Semantic / LLM-judge scoring — a secondary signal layered on top of the deterministic core, never replacing it. Core stays: task → agent → assertions → statistics.
• Security evaluation (secret leakage, filesystem/network boundaries, dangerous commands) — valuable as a separate crucible-security extension.

So "why no Gemini / no security scan?" has a straight answer: this tool measures, rigorously, whether a skill improves task outcomes on Claude Code. That is the whole promise, and it keeps it.

License

MIT