57b-agenteval

agenteval

Pick the right model — or agent, or combination of both — for each step of your AI workflow, with evidence you can check.

You give agenteval a task: some test inputs, the candidates to compare, and how to judge them. It runs every candidate on every input, grades the results with deterministic checks and (optionally) a judge model, and tells you which candidate to use, why, what it cost, and what it could not observe.

• Compare models on one prompt, coding agents on one job, or one agent across several models.
• Grade with a check you write (any program that exits 0 or 1), a TypeScript compile, and/or a judge model from a different vendor.
• Decide by explicit gates and an operating mode (lowest-cost, highest-assurance, …), not by eyeballing a leaderboard.
• Validate workflows: chain steps, then test the per-step picks end to end against a single-model baseline.
• Keep the evidence: every run writes its scores, costs, gaps and a self-contained report.html beside the task. A local dashboard browses them.

Where to start

Whichever path: agenteval plan <task> and agenteval check-task <task> are free and say what a run would do before anything is billed, and agenteval run without --yes only prints that preview.

Install

Published on npm as 57b-agenteval. Requires Node.js 22 or newer.

npm install -g 57b-agenteval        # installs the `agenteval` command
agenteval --version

Or without installing: npx 57b-agenteval <command>. To update, run the install again; to remove it, npm uninstall -g 57b-agenteval.

API key

Models are called through OpenRouter, with one key: OPENROUTER_API_KEY. You only need it for tasks that call models; plan, init and offline tasks never do.

export OPENROUTER_API_KEY=sk-or-...                   # this shell
echo 'export OPENROUTER_API_KEY=sk-or-...' >> ~/.zshrc  # every new shell
echo 'OPENROUTER_API_KEY=sk-or-...' > .env.local       # this directory only

A variable already set in your environment wins over .env.local, which is read from the directory you run in. Keep .env.local out of git (echo .env.local >> .gitignore). To check the key works: agenteval models <task> sends each model one tiny request (a fraction of a cent); --catalog-only checks the ids and prices without a key.

An agent candidate's own key (e.g. ANTHROPIC_API_KEY for Claude Code) is read by the agent, not by agenteval. On the host it inherits your environment; in a container, name it in the spec: env: { ANTHROPIC_API_KEY: "" } ("" passes the host's value through).

Quick start (free, offline)

mkdir my-evals && cd my-evals

agenteval init my-task              # creates tasks/my-task/, runnable as-is
agenteval plan my-task              # what a run would do — no key, no calls
agenteval run my-task --yes --html  # run it
agenteval dash                      # browse tasks and runs at http://127.0.0.1:4173

The generated task compares two local stand-in commands and grades them with a TypeScript compile, so it costs nothing. The run prints where it wrote its evidence — open report.html there, or use the dashboard.

A real one (about a cent)

export OPENROUTER_API_KEY=sk-or-...
agenteval init real --models        # two cheap models, a tsc gate, a judge
agenteval run real --yes --html

Two models from different vendors write the same small function; tsc gates what they wrote and a judge from a third vendor compares the ones that compile. The recommendation comes from the gates and the operating mode (here: lowest cost); the judge's preference is reported beside it, and the report says when the two disagree. Budget is capped at $0.05.

From here, edit tasks/<name>/spec.yaml: swap the models or use your own agent, add inputs, and raise the trials. See Writing a task.

Build a task with Claude Code

The fastest way to an evaluation you can trust is to let Claude Code build it with you. agenteval install-skill adds the /agenteval-new-task skill to your workspace; the skill interviews you, writes the files, and proves the task with the free checks before anything is billed.

mkdir my-evals && cd my-evals        # or your project: any directory works
agenteval install-skill              # adds .claude/skills/agenteval-new-task/
claude                               # open Claude Code here

Then, in Claude Code, start with the decision you need to make:

/agenteval-new-task which model should triage our support tickets, and is a cheaper one good enough?

What happens next:

1. The decision. One sentence on the choice the task informs, what one input is, and what a candidate is (a model, or an agent command).
2. Inputs. It asks for real cases first (logs, past failures, cases you write) and synthesizes only from examples you give it. You read every input and sign off.
3. How right is decided. A check script, reference answers it must pass, and a judge where a script can't see the quality. It runs agenteval check-task and shows you the check grading real answers; you sign off again.
4. Candidates, gates, budget. Then the plan, the cost, and one question: run it? Nothing is billed without your yes.

You end up with tasks/<name>/ (spec, inputs, check, references, rubric) and, once you run it, a report that opens with what the task evaluates. Bring three to five real examples and a sentence on what makes a case hard; that is what makes the task worth its numbers. For a pipeline (which model should run each stage?), say so: the skill builds one step at a time with its own sign-offs.

The skill is a copy: after upgrading agenteval, run agenteval install-skill --force to get the new version (it replaces any edits you made to the copy).

Tutorial: compare two models on your own prompt

This builds a task from scratch: two models summarise two short texts, a check script you write rejects summaries that are too long, and a judge compares the rest. One run costs well under a cent. Every file below is complete — copy them as they are, then change them.

my-evals/
  tasks/summary/
    spec.yaml            what to run and how to decide
    prompts/summary.md   the prompt; {{input}} is replaced by each input
    inputs/release.txt   test case 1
    inputs/incident.txt  test case 2
    rubrics/summary.md   what the judge compares
    checks/length.mjs    the pass/fail gate

prompts/summary.md:

Summarise the text below for a busy reader in at most three sentences.
Keep every number that matters. Reply with the summary only.

{{input}}

inputs/release.txt and inputs/incident.txt hold one text each — paste in any paragraph you like. Each input is one test case, and every candidate answers every input.

rubrics/summary.md — one numbered item per dimension the judge compares; the bold names must match judged_dimensions in the spec:

# summary rubric

Judge two summaries of the same text.

1. **faithfulness** — says nothing the text does not; numbers are right.
2. **coverage** — keeps what a busy reader needs: the main change or event,
   its impact, and the key numbers.
3. **concision** — no filler; shorter wins when both are equally faithful
   and complete.

checks/length.mjs — runs in each trial's working directory, where output.txt is the candidate's answer. Exit 0 passes, exit 1 fails the candidate; the JSON it prints is kept as evidence:

import fs from "node:fs";

const text = fs.readFileSync("output.txt", "utf8").trim();
const words = text.split(/\s+/).filter(Boolean).length;
const sentences = text.split(/[.!?](?:\s|$)/).filter((s) => s.trim()).length;
const ok = text.length > 0 && words <= 80 && sentences <= 3;

console.log(JSON.stringify({
  evidence: `${sentences} sentence(s), ${words} words`,
  ...(ok ? {} : { reason: "longer than three sentences or 80 words" }),
}));
process.exit(ok ? 0 : 1);

spec.yaml:

protocol_version: "0.3"
run_name: summary
budget_usd: 0.05                  # hard ceiling for the whole run

workflow:
  control_candidate: deepseek-flash
  steps:
    - id: summarise
      version: "1"
      producer: prompt            # send prompt_file to each model
      prompt_file: prompts/summary.md
      rubric_file: rubrics/summary.md
      test_set:
        id: summaries-v1
        inputs: [release, incident]         # inputs/release.txt, inputs/incident.txt
      candidate_ids: [deepseek-flash, gpt-5-nano]
      required_checks: [length]
      judged_dimensions: [faithfulness, coverage, concision]
      success_criteria:
        mandatory_checks: all     # a trial succeeds when every required check passes
      operating_mode: lowest-cost # among those that pass the gates, pick the cheapest
      eligibility:                # gates: fall below one and you are out, with the reason
        minimum_reliability: 1.0
        minimum_required_check_pass_rate: 1.0
      maximum_completion_time_seconds: 120

candidates:
  - id: deepseek-flash
    model: deepseek/deepseek-v4-flash
    provider_route: openrouter
  - id: gpt-5-nano
    model: openai/gpt-5-nano
    provider_route: openrouter

evaluators:
  required_checks:
    length:
      kind: command
      argv: [node, checks/length.mjs]
      version_files: [checks/length.mjs]    # editing the check changes its version
      timeout_seconds: 30
  judge:
    model: google/gemini-2.5-flash          # must be a third vendor
    provider_route: openrouter
    rubric_file: rubrics/summary.md
    methods: [pairwise-swap]                # compare pairs, in both orders

execution:
  trials_per_case: 1
  concurrency: 2

x-harness:                                  # required; repeats the producer
  producer: prompt
  prompt_file: prompts/summary.md

Then, from my-evals/:

agenteval models summary --catalog-only   # are the model ids right? free
agenteval plan summary                    # 4 generations, 4 judge calls, $0.05 ceiling; free
agenteval run summary --yes --html        # billed: well under a cent

What you get: each model answers each input; length passes or fails each answer; the judge compares the two models' answers on each input; the recommendation is the cheapest model that passed every gate. The terminal ends with a table like

  candidate       success  check  duels     cost    p50
  deepseek-flash      2/2    2/2    1/2  $0.0001   4.8s
  gpt-5-nano          2/2    2/2    0/2  $0.0010  12.6s
  → deepseek-flash (directional) · total $0.0052

— success is trials that passed the check, duels is pairwise wins, cost is generation cost per success. Open report.html in the run directory it prints, or agenteval dash. Reading a run explains the words in it.

To let the judge alone decide, see Judge-only grading. To put your own program in as a candidate, see Command candidates.

Samples

The package ships seven sample tasks. Copy one — or all of them — into your workspace and run it by name:

mkdir -p tasks
cp -R "$(npm root -g)/57b-agenteval/examples/tasks/custom-check" tasks/   # one
cp -R "$(npm root -g)/57b-agenteval/examples/tasks/"* tasks/             # or all seven
agenteval ls                                                        # what you now have
agenteval run custom-check --yes --html

Copying keeps their runs in your workspace instead of in the global install.

Before a billed sample, look first — both are free:

agenteval models compare-models --catalog-only   # every model id listed, and its price
agenteval plan compare-models                    # how many calls, and the budget ceiling

The agent samples run those agents unsandboxed on your machine — read the warnings in their spec.yaml and see Safety first. What an agent spends on its own provider is invisible to agenteval and its budget.

Datasets

The repository (not the npm package) holds four evaluation datasets in datasets/: Medicare claim denials, KYC beneficial owners, contract terms, and agb's own flow. Each has an answer key, the rules behind it where rules decide the answer, and one finished run, so you can read its results without paying for a run. From a clone:

agenteval dash --workspace datasets        # browse the four sets and their reports
open datasets/tasks/*/runs/*/report.html   # or open the reports directly

datasets/README.md says what each set evaluates, where its answers come from, and what its run found.

Concepts

• Workspace — a directory with a tasks/ folder. Commands find it by walking up from where you stand; with none found, the current directory is the workspace. --workspace DIR names one explicitly.
• Task — tasks/<name>/, self-contained: spec.yaml plus the inputs, prompts, rubrics, agents and checks it refers to, and its own runs/. Copy the directory anywhere and it still runs.
• Step — one thing being evaluated, e.g. "write release notes". A task has one or more.
• Candidate — what competes on a step: a model (model: vendor/name) or a command (adapter: agent-cli), such as claude -p or your own script.
• Input — a test case, inputs/<id>.txt. Every candidate runs every input, trials_per_case times.
• Required check — the deterministic gate that decides success or failure: a tsc compile, or any program you write.
• Judge — an optional model that compares outputs pairwise (in both orders) and/or scores them 1–5 against a rubric.
• Recommendation — per step: which candidates passed the eligibility gates, and which one the step's operating_mode picks among them.

Commands

Every command takes --help.

<task> is a task name in the workspace (my-task), a task directory (./somewhere/my-task), or a path to a spec.yaml.

Useful run options: --html (write report.html), --json (one JSON event per line, for scripts), --reuse (reuse identical earlier trials instead of paying for them again — whole, including their check results, so do not use it right after changing a check), --concurrency N. On an interactive terminal run --yes shows a live progress block; piped, redirected or with --plain it prints one line per event.

plan reads the spec and counts; it makes no call and needs no key. Per step:

▶ <task> / <step> [<producer>] — <candidates> candidates × <inputs> inputs × <trials> trials
  generations N · pairwise P pairs (2P judge calls, up to 3 attempts each) · absolute S calls
  judge <model> · concurrency C · budget $B
  benchmark <mode> · cache <mode> · directional yes|no

pairwise off and absolute off mean no judge call will be made. plan counts calls; models gives per-token prices, and budget_usd is the ceiling.

Exit codes: 0 done · 1 failed · 2 usage error · 3 finished but partial (the budget stopped it, or the machine slept and durations are unreliable).

Writing a task

The tutorial has a complete spec; this section is the reference. Run agenteval plan <task> after every edit — it validates the spec and names the field it rejects.

Before you pay for a run, run agenteval check-task <task>. It calls no model and needs no key:

• Inputs: it counts them and flags empty or duplicated ones. It also says how far a pass rate can move by chance at that size. With 1 input × 2 trials the swing is about ±71 points, so aim for 10 or more inputs.
• Check: it feeds the required check an empty answer, which must fail. It also feeds it a correct answer, which must pass. Put correct answers in references/<input>/ as the files a good candidate would leave, with an optional output.txt, or in references/<input>.txt as the deliverable text. A check that passes nothing-at-all, or rejects a correct answer, blocks.
• Candidates: a declared script that does not exist blocks. A command that runs without image: gets a warning, because it can read checks/, rubrics/ and references/.

It exits 1 when anything blocks, and --json prints the findings for scripts.

Building a task with Claude Code. agenteval install-skill adds the /agenteval-new-task skill, with this README beside it as reference.md; see Build a task with Claude Code.

Rules plan enforces

• protocol_version, run_name, workflow, candidates, evaluators and execution are required.
• evaluators.judge is required even when no judge runs: give it a model and methods: [].
• The judge's vendor (the part before / in the model id) must differ from every candidate's, so a judge cannot favour its own vendor's output.
• x-harness.producer is required, and must name the producer the steps use (see below); with producer: prompt it also needs prompt_file.
• A step's required_checks names checks declared under evaluators.required_checks, at most one per step.
• Candidate ids are yours to choose; every id in candidate_ids must be declared under candidates.

Producers — how a candidate's answer is made:

• prompt — each model gets prompt_file with {{input}} replaced by the input's text. Its answer is the deliverable (output.txt in the check's directory). Use it for models answering in text.
• codegen — for code: models are asked for files in ```file:<name> blocks, which are parsed out and written to disk, so a tsc check can compile them. Command candidates (below) also use codegen: the files they leave behind are their deliverable. No prompt_file needed.

Operating modes — how the pick is made among candidates that passed every gate: lowest-cost (cheapest per success), fastest-within-cost-ceiling (fastest median time under eligibility.cost_ceiling_per_success_usd), highest-assurance (highest success rate), judge-preference (the judge's favourite).

Eligibility gates — minimum_reliability (share of trials that succeeded), minimum_required_check_pass_rate, maximum_p95_ms. A candidate below any gate is removed, and the report says which gate and by how much.

Judge-only grading

With no check, no trial can succeed or fail — the outcome is undetermined — so the minimum_reliability and minimum_required_check_pass_rate gates would remove every candidate. To let the judge alone decide:

      required_checks: []
      judged_dimensions: [faithfulness, coverage, concision]
      operating_mode: judge-preference   # no eligibility, no success_criteria
# ...
evaluators:
  required_checks: {}
  judge: { model: google/gemini-2.5-flash, provider_route: openrouter,
           rubric_file: rubrics/summary.md, methods: [pairwise-swap] }

The judge's favourite is recommended, and the success column reads 0/2: without a check nothing was measured as right or wrong. Such a verdict is at most directional.

Human review

A required check says whether an answer is right. It cannot say which of two right answers is better. For that, review a run blind, in the dashboard:

evaluators:
  human:
    rubric_file: rubrics/review.md          # what the reviewer reads
    show: [".ab/tasks.json", "*.md"]        # files to compare, in each trial's work dir
    dimensions: [coverage, granularity]     # optional; defaults to judged_dimensions

agenteval review <run>                      # opens the dashboard on the review page

The page shows one pair per screen: the input, then two candidates' answers as Left and Right. Model names, cost, time and check results are not shown, and the server does not send them until you reveal. For each pair you give an overall verdict (left, right, tie or both bad), optional verdicts per dimension, and a note. Shortcuts: a/b/t/x, Enter to save, j/k to move between pairs.

Each verdict is saved as you give it, to runs/<run>/reviews/<you>.jsonl. A reload resumes where you stopped. Pairs where one side failed its check are hidden unless you tick the box. Reveal shows which candidate was which, and makes your verdicts read-only.

report.html and the dashboard then show a Human win rate column. If a judge also ran, they also show how often the judge agreed with you. Below 90% agreement, GAPS.md in the report calls the judge uncalibrated. Human verdicts inform the report and do not change the recommendation. Review works on runs made before the human block was added: it reads the task's current spec, and stamps the rubric's hash on every verdict.

Command candidates

Any program can compete:

  - id: my-agent
    adapter: agent-cli
    cli:
      argv: [node, agents/run.mjs, "{{input}}"]

It runs in a fresh, empty work dir. {{input}} is replaced by the test case (also written to .eval-input.txt) and {{workdir}} by the work dir's path. The files it leaves behind are its deliverable; if it leaves none, its stdout is. It must exit 0: a non-zero exit is recorded as a failed attempt (with the exit code as the reason), even if it left files behind. Relative paths resolve against the task directory. The judge reads every file it left; when that is a whole project, evaluators.judge.show (globs such as [README.md, "src/**"]) narrows what the judge reads, and what the report shows, to those files. The check still gets every file. Use x-harness: { producer: codegen }. Add image: to run it in Docker (see Safety); agenteval init and the custom-check sample are working examples.

agenteval cannot see what a command spends, so its cost is recorded as unobserved: unobserved in the tables, null per success, named in GAPS.md. A command that knows its spend can report it. Declare reports_usage: true under cli: and have the command write .eval-usage.json in its work dir:

{ "cost_usd": 0.0727, "prompt_tokens": 60180, "completion_tokens": 4070 }

Declare it only for a command you trust to write that file. The agent under evaluation can write to its work dir too, so an agent you run directly could price itself. A wrapper that copies the figure from a harness it ran, as tasks/harbor-spike does with Harbor, is the intended use. The figure is recorded as estimated, and GAPS.md names the candidates that reported their own cost.

An agent tool's own cost figure assumes its own vendor's prices (claude prices any model it does not know at Opus rates). To have agenteval do the pricing, declare reports_usage: tokens, have the command report tokens only, and give the candidate the model it runs (an OpenRouter id, priced from the OpenRouter catalog) or a price (USD per million tokens) for a direct provider:

  - id: deepseek-v4-pro
    adapter: agent-cli
    price: { input: 1.32, output: 3.96, cache_read: 0.044 }   # or: model: deepseek/deepseek-v4-pro-0813
    cli:
      argv: [node, agents/run.mjs, deepseek-v4-pro, "{{max_cost_usd}}"]
      reports_usage: tokens
      env: { ANTHROPIC_API_KEY: "${DEEPSEEK_API_KEY}" }

{ "prompt_tokens": 1200, "completion_tokens": 3400, "cache_read_tokens": 81000, "cache_write_tokens": 9000 }

A command that exits non-zero fails its trial whatever the check says. When the command is a harness with verdicts of its own (its gates, its verifier), declare exit_code: ignore on the step (or on the candidate's cli): the exit code is kept as evidence and the check decides. combine and judge apply it to runs recorded before it was declared.

prompt_tokens is input not read from or written to a cache. A step's max_cost_per_trial_usd is handed to the command as {{max_cost_usd}} and EVAL_MAX_COST_USD, with the model's price as EVAL_MODEL_PRICE (JSON, per million tokens) so a wrapper can convert the cap into its tool's units; agenteval cannot stop an agent mid-run, so the command enforces it. ${NAME} in env is read from the environment (or .env.local) when the command starts, so the spec and the run's records never hold the secret; check-task blocks when one is not set.

Check scripts

A kind: command check runs in the trial's work dir: the candidate's files, plus output.txt (the deliverable), input.txt (the test case) and meta.json ({step, candidate, input, trial}). For a prompt model output.txt is its answer as text; for code and command candidates it holds the files as ```file: blocks, so read the files themselves.

Exit 0 passes, exit 1 fails the candidate; anything else — a crash, a timeout — is recorded as an evaluator error, never blamed on the candidate. Print {"evidence": "...", "reason": "..."} and both are kept with the trial. version_files hashes the script into the check's version, so editing it shows up in the evidence. kind: tsc with scaffold_dir: scaffold instead compiles the produced TypeScript against scaffold/tsconfig.json.

Saying what a task evaluates

A report opens with what was evaluated, so a reader who never saw the spec knows what the numbers mean. Four optional fields carry it:

description: "Which model triages a denied Medicare SNF claim correctly? 30 synthetic claims …"
workflow:
  steps:
    - id: triage
      description: "Decide why a claim was denied and what to do next."
      test_set:
        id: claim-denials
        labels_file: checks/manifest.json   # { "cd-001": { "tags": ["hard", "span-dates"], "split": "holdout" } }
evaluators:
  required_checks:
    triage-check:
      kind: command
      description: "Root cause, next action and deadline match the answer key."

description heads the report as What it evaluates; a step's description stands beside the step's id wherever it appears (a workflow's step cards, a step's report), so name steps for what they do and say it there; a check's description replaces its version hash under How success is decided; with labels_file, the report adds Results by input tag: the required check's passes per candidate for each tag and split. All four are frozen into the run's manifest.json. agenteval report on a run made before they existed reads them from the task's current spec and says so on the page.

Workflows

A task can have several steps, each evaluated on its own inputs. Add input_from: <earlier step id> to chain them: the run then also executes the whole chain twice — once with control_candidate on every step, once with each step's recommended candidate — and reports whether the combination beats the single model. The chain-offline sample shows this.

In the chain, a step's input is the previous step's output, rendered as text. For agents whose next stage needs the project itself, add handoff: workdir to the chained step: its work dir then starts with the files the previous step left, and its input text stays the root input's.

An input can also be a starting state: with fixture_dir: fixture under test_set, fixture/<input>/ is copied into a command candidate's work dir before it starts (its contents are part of the trial's identity, so editing a fixture is a new trial). Several steps can then share input names, one fixture each; references/<step>/<input> takes precedence over references/<input>. A command candidate is told where it is through EVAL_STEP, EVAL_INPUT and EVAL_CHAINED (1 when its work dir was handed over), so one command can serve every step; a check finds the fixture's path as fixture in meta.json.

A whole chain can be far costlier than any one step. workflow.e2e bounds it:

workflow:
  control_candidate: opus
  e2e:
    inputs: [demo-todo]        # run the chain on these root inputs only
    final_checks: [app-gates]  # grade the chain's end product with this check

e2e.steps.<step> (max_cost_per_trial_usd, maximum_completion_time_seconds) raises a step's limits in the arms only, where it may do far more than on its own inputs. final_checks replaces the last step's own check in the chain only: a step evaluated on its own inputs (one task, say) and a chain's delivery (every task) usually need different checks. e2e.judge (rubric_file, and optionally show and dimensions) compares the two arms' deliveries case by case, pairwise; like a step's judge it informs, and the gates decide.

Steps differ in what to judge and how hard they are to run. A step can carry its own judge block (rubric_file, methods, show, dimensions; the judge model stays evaluators.judge.model) and its own concurrency. --concurrency on the command line overrides both.

A long run that dies part way is resumed with agenteval run <task> --yes --reuse: every trial is journaled the moment it finishes (trials.partial.jsonl), and --reuse takes finished trials from there as well as from completed runs, so only the unfinished ones run again.

How much to trust a result

Every verdict carries a confidence:

• firm — at least 10 inputs and execution.minimum_meaningful_difference declared (the smallest gap you consider real), and not judge-preference.
• directional — otherwise: enough to look at the evidence, not to decide. Most small runs are directional, and judge-preference always is; the report lists why.
• needs review — no candidate passed the gates, so nothing was recommended.

Raise trials_per_case to see how stable a candidate is; add inputs to see how general.

Reading a run

Every run prints the directory it wrote, tasks/<task>/runs/<runId>/. Open report.html there, or find the run in agenteval dash. Read it top down:

1. The verdict — the recommended candidate, the operating mode that picked it, and its confidence (firm / directional / needs review; see How much to trust a result).
2. Selection — each eligibility gate in order: who it removed, with the number that removed them, and who was left. Gated means removed by a gate.
3. Candidates — per candidate: trials that succeeded, checks passed, pairwise record, cost per success, median time.
4. Judge — what the judge preferred, per dimension, with its reasons. It is reported beside the recommendation, not inside it: unless the mode is judge-preference, the judge does not choose, and the report says when it disagrees with the pick.
5. Evidence — every trial's answer exactly as the judge saw it, and the check's evidence and reason.
6. Not observed — what this run could not measure, recorded as unknown rather than guessed (GAPS.md).

Words you will meet:

• success / failure / undetermined (a trial) — succeeded means every required check passed; with no check it is undetermined. A candidate that exits non-zero, times out or returns nothing failed to complete (malformed, timeout, refusal, provider_error), which is also a failure.
• evaluator error — a check or the judge itself broke; never counted against the candidate.
• control candidate — control_candidate: the baseline. In a single step it breaks ties; in a chained workflow it is the single-model arm.
• pairwise, both orders — the judge compares A with B, then B with A. If the two disagree on a dimension, that dimension is a tie. "Output A/B" in the judge's text is the pair's first/second candidate, as the report labels them.
• cost source: provider-reported is what OpenRouter billed. estimated is priced from tokens, or self-reported by a command through reports_usage. none means not observed: a command candidate that reports nothing. Its cost per success is undefined, never $0.

A multi-step run has one such directory per step, plus workflow.json and, when chained, e2e-validation.json at the root. agenteval report <run> rebuilds report.html from these files (by run id or path); no number changes.

Dashboard

agenteval dash serves the workspace at http://127.0.0.1:4173 (--port to change it; --workspace DIR to serve another folder, e.g. --workspace datasets or --workspace examples in a clone). The home page lists tasks, their latest verdicts and total spend; a task's page shows its runs, its test plan read back from spec.yaml, and its files; a run's page opens each step's report. From a task's page you can also start a run: it first shows the plan and the budget, and runs only after you confirm those numbers. Stop the dashboard with Ctrl+C.

To show it to someone who doesn't run agenteval, export it:

agenteval dash --workspace datasets --export ~/Desktop/agenteval-dashboard

The folder opens by double-clicking index.html, with no server and no install, and looks and navigates like the live dashboard: every task, run, report and evidence table as it is at export time. It is read-only: no runs, no reviews, and a banner says when it was exported. Zip the folder to send it; the four datasets come to about 12 MB.

Controlling cost

• Run plan first. run without --yes is the same preview, so a typo cannot bill you.
• budget_usd is enforced: once spend reaches it, the run stops starting new work, records what it skipped, and exits 3.
• methods: [] plus command-only candidates make a task free.
• agenteval models <task> --catalog-only catches a mistyped model id for free.
• On macOS, run billed tasks under caffeinate -i. If the machine sleeps, the run detects the gaps, says so in GAPS.md, and exits 3.

Safety

• A command (agent-cli) candidate runs on your machine as you — your files, network and credentials. To run it in a Docker container instead, add image: to its cli: block. Only the work dir is writable; a script the task ships (agents/...) is mounted read-only, and checks/ is not mounted. The network is off unless you set network:, and only the variables you list in env: are passed in. Every trial records whether it was isolated. The docker-sandbox sample shows both ways side by side.
• Every trial, and its check, runs in a temp dir outside the workspace, so a project with no config of its own never picks up the workspace's (a vite.config.ts, a tsconfig.json). What it left is copied under the run as evidence, without node_modules or .git.
• While a run is in flight the task's references/ is unreadable, so a command candidate cannot read the answers; a check that grades against them finds a private copy at $EVAL_REFERENCES_DIR. This stops an agent that goes looking, not one set on breaking out: image: is the real boundary.
• A command runs as its own process group. A timeout stops the whole tree (the agent and whatever it started), as does Ctrl+C on agenteval. A command that prints a lot is not stopped: the last 2 MB of its output are kept. Files over 2 MB, or binary, are not read back as its answer; the trace names them.
• Check scripts and tsc always run on your machine. Declare only checks you would run yourself.
• An agent's own spending is invisible to agenteval unless its command reports it (reports_usage); unreported, it shows as unobserved. What is reported counts against budget_usd, end-to-end arms included: past the budget no new trial starts. One already running finishes, so cap each trial too (max_cost_per_trial_usd).
• The dashboard listens on 127.0.0.1 only and has no authentication.

Troubleshooting

Development

Working on agenteval itself — architecture, tests, the evidence format — is covered in CONTRIBUTING.md.

License

MIT © 57blocks