agent-cover

🛡️ Agent Cover

Agent Cover is an observability and testing framework designed for LLMOps. Unlike traditional tools that measure executed lines of code, Agent Cover measures the Logical Coverage of your Agent.

It verifies that your test suite actually exercises the agent's capabilities: its Prompts, its Tools, and its Business Decision branches.

📚 Read the Documentation

---

📉 The Gap in LLM Testing

Traditional tools (like coverage.py) measure Python execution. However, in LLM-based applications, "line coverage" creates a false sense of security.

The Scenario: Your agent has a tool refund_user and a prompt instruction to "Ask for confirmation".

The Problem: Your test runs the code, but the LLM decides to skip the tool or ignore the prompt. Standard coverage says 100%. Agent Cover says 0% Tool Coverage.

🚀 How It Works

Agent Cover operates on a hybrid "Static + Runtime" model:

1. 🔍 Inventory Scan (Static): Before tests run, it scans your codebase to build a map of "Logical Assets":

   • Prompts: LangChain objects, Jinja2 templates, and raw global strings.
   • Tools: Functions decorated with @tool or inheriting from BaseTool.
   • Decisions: Automatically derives coverage rules from your Pydantic models (Enums, Literals, Booleans).

2. ⚡ Instrumentation (Runtime): It hooks into the execution lifecycle (LangChain, LlamaIndex, OpenAI, PromptFlow) to verify if those assets were actually used by the LLM.

---

📦 Installation

pip install agent-cover

For development and testing support:

pip install "agent-cover[dev]"

---

⚡ Quick Start

1. Configuration (Optional but Recommended)

Create an agent-cover.yaml to define your Business Logic requirements.

# agent-cover.yaml
decisions:
  - id: intent_classification
    description: "Ensure the agent triggers all support paths"
    target_field: intent
    expected_values: ["REFUND", "TECH_SUPPORT", "SALES"]

2. Run with Pytest

Simply add the --agent-cov flag to your existing test run.

pytest --agent-cov --agent-cov-html=coverage_report

3. View the Report

Open coverage_report/index.html. You will see:

• Prompt Coverage: Which templates were formatted and sent to the LLM.
• Tool Coverage: Which tools the LLM actually decided to call.
• Decision Coverage: Which logical branches (e.g., REFUND vs SALES) were observed in the output.

---

🔌 Supported Frameworks

Agent Cover dynamically patches these libraries to intercept logical events.

---

🛠 Advanced Usage: Microsoft PromptFlow

PromptFlow executes nodes in separate worker processes, which breaks standard coverage tools. Agent Cover includes a dedicated CLI wrapper to handle multi-process aggregation.

Instead of running pf run, wrap your command:

# Injects instrumentation into all child worker processes
agent-cover run --source-dir ./src -- pf run create --flow ./my_flow --data data.jsonl

This will automatically aggregate coverage data from all parallel workers into a single report.

---

🤝 Contributing

Contributions are welcome! Please ensure all pull requests pass the CI quality gates.

1. Clone the repository.
2. Install development dependencies: pip install -e ".[dev]"
3. Install pre-commit hooks: pre-commit install
4. Run the test suite: pytest

📄 License

This project is licensed under the MIT License.