agenteval-toolkit

AgentEval

Scan, evaluate, certify. AgentEval is an Apache-2.0 command-line toolkit with three jobs:

• it scans self-hosted AI infrastructure, A2A Agent Cards and MCP configurations against policy-as-code rule packs,
• it evaluates AI agents with standard exam papers, and
• it issues signed certificates when a system passes both.

English · 简体中文

Status: alpha (0.2.0). The rule packs are starter mappings, and every one is marked verified: false. They are a starting point for your own legal and compliance review. They are not legal advice.

---

Contents

• Why
• Features
• Install
• Quickstart
• Commands and exit codes
• Rule packs
• Writing rules
• Evaluating agents
• Agent protocols: A2A and MCP
• Certification
• Configuration
• Security model
• Project layout
• Roadmap
• Contributing
• License

Why

Teams that run their own models on Kubernetes have to answer two separate questions:

• Is the deployment configured the way our policies require? (auth, TLS, provenance, PII handling, …)
• Does the agent built on it do its job safely? (task completion, accuracy, prompt-injection resistance, …)

AgentEval answers both with versioned, reviewable files, rules and exams in YAML, and turns the combined result into a certificate that anyone can verify offline.

Features

Install

Requires Python 3.11+.

pip install "agenteval-toolkit[kubernetes]"   # drop [kubernetes] if you only scan snapshots
agenteval version

mkdir my-ai-governance && cd my-ai-governance
agenteval init          # copies the starter rules/, exams/, certification/ and examples/ here

The PyPI package is named agenteval-toolkit. The command is agenteval and the import name is agenteval. Optional extras: pdf (certificate PDFs through Playwright/Chromium) and dev (pytest and build).

The CLI looks for rules/, exams/ and certification/ in the current directory, and falls back to the starter copies bundled with the package. Pass --rules or --exams, or set the environment variables, to use other locations.

From source:

git clone https://github.com/pony360/AgentEval.git && cd AgentEval
pip install -e ".[dev,kubernetes]"

Quickstart

All of these commands run offline against the bundled sample cluster and demo agents.

# 1. Validate the rule packs
agenteval rules validate
#   ✓ 23 rules validated

# 2. Scan a cluster snapshot (exits 1: there are high/critical findings)
agenteval scan --snapshot examples/snapshots/sample-cluster.json -f html -o scan.html

# 3. Evaluate an agent. On a pass, also write pre-deployment-testing evidence for its asset
agenteval eval run -p cs-ecommerce-v1 -a examples/agents/demo-good.yaml -f json -o eval.json \
  --attest-asset deployment/ml-prod/llama-chat --attest-output att-eval.yaml
#   → 96.7 (grade A) · cases passed 10/10 · critical failures 0

# 4. Re-scan with the human attestations and the evaluation evidence
agenteval scan --snapshot examples/snapshots/sample-cluster.json \
  --attestations examples/attestations.yaml \
  --attestations examples/attestations-certification.yaml \
  --attestations att-eval.yaml -o scan.json --fail-on none

# 5. Certify the registered system, then verify the certificate
agenteval cert keygen --public-out issuer.pem
agenteval cert issue -s northwind-cs --scan scan.json --eval eval.json -o certificates/
agenteval cert verify certificates/AEV-*.json --public-key issuer.pem
#   Signature valid · State: active · DEMO certificate (heuristic judge)

To scan a live cluster, run agenteval scan --source kubernetes [--context ctx] [-n namespace]. It needs read-only access to Deployments, StatefulSets, Services, Ingresses and NetworkPolicies. For air-gapped clusters, use agenteval snapshot to export a snapshot and scan it elsewhere.

Every command accepts --lang zh for Chinese output.

Commands and exit codes

Rule packs

Rule text is in English, and Simplified Chinese translations are in rules/*/i18n/zh.yaml.

Writing rules

# yaml-language-server: $schema=../../agenteval/schemas/rule.schema.json
rules:
  - id: FED-SEC-002
    title: Externally exposed inference endpoints must use TLS
    description: Services exposed through LoadBalancer, NodePort or Ingress must encrypt traffic.
    severity: high
    type: automated
    category: data-protection
    applies_to:
      asset_types: [inference_service]
      when: {field: exposure.public, op: eq, value: true}
    check: {field: tls.enabled, op: eq, value: true}
    remediation: Add a tls section to the Ingress with a trusted certificate (e.g. cert-manager).
    sources:
      - framework: NIST SP 800-53 Rev.5
        reference: SC-8 Transmission Confidentiality and Integrity
        verified: false

The full reference is in docs/rule-schema.md. It covers every field, the condition operators, the fact catalog, attestations and translations. The schemas are in agenteval/schemas/.

Evaluating agents

An exam paper defines weighted dimensions, multi-turn cases, assertions (some of them critical) and rubric items. The bundled cs-ecommerce-v1 paper tests an e-commerce support agent in 10 cases, from order lookup and returns to PII handling and prompt injection.

agenteval eval run -p cs-ecommerce-v1 -a my-agent.yaml --judge anthropic --judge-model <model> \
  --judge-key-env ANTHROPIC_API_KEY -o report.html

See docs/exams.md for the paper format, scoring, adapters and judges.

Agent protocols: A2A and MCP

# Evaluate an A2A agent (v1.0 or v0.3): each exam case runs as one A2A conversation
agenteval eval run -p cs-ecommerce-v1 -a examples/agents/a2a.example.yaml

# Check an Agent Card, and the MCP servers your tools launch
agenteval a2a card https://agents.example.com
agenteval scan --a2a-card ./agent-card.json --mcp-config ~/.cursor/mcp.json -f html -o protocols.html

# Use AgentEval from Claude, an IDE or any MCP client
agenteval mcp serve --root .

See docs/a2a.md and docs/mcp.md.

Certification

cert issue signs a certificate only if all of the following hold:

• the security score is > 90 and the quality score is > 90,
• there are no open high or critical findings,
• no attestations are pending,
• there are no critical evaluation failures and no session errors, and
• the evidence is at most 30 days old.

Evaluations scored by the offline heuristic judge get DEMO certificates only. Certificates are valid for 6 months. See docs/certification.md for the gates, keys, the verification URL and status hooks, and how to revoke certificates.

Configuration

Security model

• Read-only. The scanner only lists and reads Kubernetes objects. It never changes them.
• No secret values. Plaintext-secret detection records environment variable names only.
• No arbitrary code. Rule conditions are a fixed DSL, and YAML is loaded with safe_load.
• Fail-closed. Missing facts never make a check pass.
• Secrets come from the environment. Agent and judge API keys are read from the variables you name.
• Private key. The signing key is created with mode 0600 and is git-ignored.

To report a vulnerability, see SECURITY.md.

Project layout

agenteval/            Python package (CLI, engine, collectors, evaluation, certification, schemas)
rules/                Rule packs (national / regional / industry / baseline)
exams/                Exam papers
certification/        Certification policy and registered systems
examples/             Sample snapshot, attestations, agent configs, A2A cards and MCP configs
docs/                 Reference documentation
tests/                Test suite (pytest)

Roadmap

• More collectors (cloud-managed endpoints, model registries, API gateways, live MCP server introspection)
• A2A streaming (SendStreamingMessage) and gRPC / HTTP+JSON bindings; Agent Card signature verification
• An MCP mock-tool server so agents that consume MCP can be scored on their tool calls
• More rule packs (EU AI Act, ISO/IEC 42001, HIPAA), contributed with reviewer sign-off
• More exam papers (IT helpdesk, sales assistant, RAG question answering)
• A hosted verification and status service reference implementation

Contributing

Contributions are welcome, especially rules, exam papers and translations. Read CONTRIBUTING.md first. Commits must be signed off under the DCO (git commit -s). This project follows the Code of Conduct.

pip install -e ".[dev]"
pytest

License

Licensed under the Apache License, Version 2.0. See NOTICE.

Framework and regulation names, including NIST, PCI DSS and California statutes, are used only to identify the source material a rule refers to. Their use does not imply endorsement by, or affiliation with, their owners.