aieval-py

aieval

Deterministic-first runtime validation and reliability toolkit for LLM outputs and agent actions.

Catch predictable failures (schema violations, missing required keys, JSON format errors, invalid tool call args, policy constraints) in sub-millisecond execution before spending latency and money on semantic LLM-as-a-judge scoring.

---

Key Highlights

• ⚡ Deterministic First: Cheap, fast assertions run before calling model judges.
• 📦 Zero Bloat: Minimal external footprint (pydantic>=2.0.0 as schema engine, pure Python standard library for the rest).
• 🔄 Dual Sync & Async: Native support for synchronous scripts (evaluate()) and asynchronous pipelines (await aevaluate()).
• 🤖 Agent Tool Call Pre-flight: Validates function names, required arguments, type safety, and stringified JSON (OpenAI tool call format).
• 🎚️ Severity Routing: Flexible threshold routing (fail_on="error" vs strict fail_on="warning").
• 🧪 CLI & Golden Fixtures: Run validation from the command line on JSON test fixtures (aieval run <fixture.json>).
• 🎯 Full TypeScript Parity: Shares the exact failure taxonomy, code conventions, and golden test formats with @hamza1331/aieval.

---

Installation

pip install aieval-py

Behavior changes in 0.2.0

This release tightens validation behavior. Tool-call arguments not declared by required_args or the supplied schema are rejected by default; set allow_unexpected_args=True only when extra arguments are intentionally allowed. Custom checks must return None, Failure, CheckResult, or a sequence of supported failure results; unsupported values now raise EvaluationConfigError. Terminal failures also remain blocking regardless of the fail_on severity threshold. Review these changes before upgrading if your application relies on permissive argument handling or custom return values.

---

Quickstart

1. Synchronous Output Evaluation

from aieval import evaluate, schema_check, required_fields
from pydantic import BaseModel, Field


class AnalysisReport(BaseModel):
    summary: str
    confidence_score: float = Field(ge=0.0, le=1.0)
    tags: list[str]


llm_output = '{"summary": "All systems nominal", "confidence_score": 0.95, "tags": ["ops", "prod"]}'

result = evaluate(
    output=llm_output,
    checks=[
        schema_check(AnalysisReport),
        required_fields(["summary", "confidence_score"]),
    ],
)

if result.passed:
    print(f"Passed in {result.summary.duration_ms}ms!")
else:
    for failure in result.failures:
        print(f"[{failure.code}] {failure.message} (path: {failure.path})")

2. Pre-flight Agent Tool Call Validation

Validate tool calls before execution to avoid runtime exceptions and agent failure loops:

from aieval import evaluate, tool_call_check
from pydantic import BaseModel


class SendEmailArgs(BaseModel):
    recipient: str
    subject: str
    body: str


# Works with both native dicts and OpenAI stringified JSON arguments:
raw_tool_call = {
    "name": "send_email",
    "arguments": '{"recipient": "team@example.com", "subject": "Update"}',
}

result = evaluate(
    output=raw_tool_call,
    checks=[
        tool_call_check("send_email", schema=SendEmailArgs, required_args=["body"]),
    ],
)

print(result.passed)  # False - missing required argument 'body'

3. Asynchronous Pipeline (aevaluate)

import asyncio
from aieval import aevaluate, valid_json, regex_match


async def main():
    result = await aevaluate(
        output="Order #12345 confirmed.",
        checks=[
            regex_match(r"Order #\d+"),
        ],
    )
    print("Passed:", result.passed)


asyncio.run(main())

---

CLI Usage

Run Golden Test Fixtures

aieval run path/to/fixture.json

When a fixture includes an expected block, the command compares the evaluation result with that expectation and exits successfully only when they match. This allows both expected passes and expected failures to be regression fixtures.

Or output raw JSON:

aieval run path/to/fixture.json --json

Check Arbitrary Files

aieval check output.json --specs checks.json

---

Failure Code Taxonomy

---

Examples

Runnable walkthrough scripts are available in the examples/ directory:

• 01_basic_llm_output.py: Schema and field constraint validation on LLM output.
• 02_agent_tool_guardrail.py: Agent tool call pre-flight validation (OpenAI stringified arguments & multi-tool selection).
• 03_async_and_severity_routing.py: Asynchronous concurrent pipeline with severity routing.
• 04_semantic_judge_adapter.py: Integrating advisory LLM-as-a-judge checks via SemanticJudge.

---

License

MIT