agent-eval-forge

---

Why

You change a prompt, swap a model, or add a new tool. The demo looks great. Three days later, a customer hits a regression the demo never covered.

Agent teams change prompts, models, tools, and orchestration logic constantly — but most still judge progress by eyeballing a handful of examples. That works for demos. It fails for production. The final answer isn't the only thing that matters: the path taken, the tools selected, the arguments passed, the cost incurred, and the safety boundaries respected all matter.

EvalForge turns release decisions from anecdotes into evidence.

"Did the agent actually get better, or did it just change?"

---

What It Is

A release-discipline product for tool-using AI agents. Local-first, CI-second. Strong default rubrics out of the box, all overridable. Built for one job: deciding whether an agent change is safe to ship.

Product philosophy

• Safety > Correctness > Efficiency — safety regressions fail by default
• Explicit golden baselines — compare against what you accepted, not what happened to run last
• Local-first, CI-second — catch issues before merge, enforce in CI
• Strong default rubrics — credible pass/fail behavior out of the box
• Framework-agnostic by adapter — not claimed, proven with tested integrations

---

What It Is Not

---

Quickstart

# Install
pip install agent-eval-forge

# With framework adapters and judge backends
pip install agent-eval-forge[langgraph,pydanticai,judge]

# Or install all adapter families
pip install agent-eval-forge[all]

# Run a scenario pack against your agent
evalforge run \
  --pack scenarios/core-launch.yaml \
  --agent python:my_package.my_module:run \
  --baseline v1.0.0 \
  --judge openai:gpt-4o-mini

# Override the default function name (default: "run")
evalforge run \
  --pack scenarios/core-launch.yaml \
  --agent python:my_package.my_module \
  --agent-function handle_request \
  --baseline v1.0.0 \
  --judge openai:gpt-4o-mini

# Save a baseline once you're happy
evalforge baseline save --name v1.0.0 --run run-20260728-001

# Gate in CI
evalforge run --pack scenarios/core-launch.yaml --agent python:my_agent.py --ci

See the User Guide for a complete walkthrough — first eval in 5 minutes, all commands, and 12 real-world gotchas.

---

Launch Pack (v0.1)

20 scenarios across 10 families, plus 8 security scenarios:

All scenarios ship in scenarios/core-launch.yaml and scenarios/security-launch.yaml.

---

Framework Adapters

Install extras: pip install evalforge[<family>] or pip install evalforge[all] for all adapter families.

---

Scoring

Two layers, cheapest first:

1. Deterministic scorers (always run, free, reproducible) — tool correctness, argument precision, schema validity, budget adherence, safety gates, grounding checks. 17 scorers built-in.

2. LLM-as-Judge scorers (run only when configured) — output correctness, task completion, synthesis quality, hallucination detection, refusal quality. 11 judge metrics built-in. Backends: OpenAI, Anthropic, Ollama, MLX (Apple Silicon).

Evaluation hierarchy: Safety failures trump everything. Correctness regressions warn by default. Efficiency regressions are informational.

See the Scoring Guide for the full metric catalog, custom scorers, and failure taxonomy.

---

Security

⚠️ Running without --sandbox exposes your API keys and environment variables to the agent process. Always use --sandbox in CI. For untrusted scenario packs, also use --trust external which restricts adapters to sandboxed subprocess only.

The security model includes:

• Sandbox mode — env stripping, filesystem isolation, optional Docker network isolation
• Trust policies — trust field on packs, adapter/tool matrix enforced at validate/run
• Audit trail — per-run append-only log of policy decisions
• API key sanitization — deep-redaction in artifacts and logs

See Security Review (v0.2) for the current model.

---

Field Testing

EvalForge was validated against 19 real-world open-source agents (11 LangGraph + 8 PydanticAI) in v0.1.0. v0.2.0 adds 7 new adapter families (CrewAI, OpenAI Agents SDK, smolagents, AutoGen, LlamaIndex, Claude Agent SDK, Google ADK) for a total of 12 supported frameworks.

---

Project Status

See the WBS (v0.2) for the full milestone plan with GitHub issue tracking.

---

Documentation

---

Community

• Report a bug
• Request a feature
• Submit a scenario pack

---

License

MIT