0.1.0

Agent Diagnostician

When an LLM agent fails, the run log usually tells you that it failed — not why.

Agent Diagnostician reads a JSON trace (task, steps, tool calls, outputs) and returns a diagnosis: failure type, subtype, confidence, evidence, and a suggested fix direction. It does not depend on LangChain, LangGraph, or any other agent framework.

Python 3.11+.

from agent_diagnostician import Classifier
from agent_diagnostician.tracer import load_fixture
from agent_diagnostician.reporter import Reporter

classifier = Classifier()  # no API key; uses rules + embeddings
trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")
Reporter.print(classifier.diagnose(trace))

Pass an LLM judge when you want the third detection tier (ambiguous tool use, goal misinterpretation, hallucination, and similar cases). Without a key, those paths stay conservative rather than calling an API.

Install

pip install agent-diagnostician

From a clone of this repo:

pip install -e .
pip install -e ".[llm-openai]"      # optional
pip install -e ".[llm-anthropic]"   # optional

The first diagnose() download of the embedding model (all-MiniLM-L6-v2 via Hugging Face) is about 90MB and needs network access. After that, embeddings run locally.

Core install currently includes Pydantic, sentence-transformers, scikit-learn, and the Gemini SDK (default live provider). OpenAI and Anthropic are extras.

Quick start

Build a trace in code:

from agent_diagnostician import Classifier
from agent_diagnostician.models.trace import AgentTrace, Step
from agent_diagnostician.reporter import Reporter

trace = AgentTrace(
    run_id="run_001",
    task="Refund order ORD-123 for $49.99",
    status="failed",
    total_steps=1,
    final_output=None,
    steps=[
        Step(
            step_index=0,
            tool_name="issue_refund",
            tool_input={"order_id": "ORD-999", "amount": 49.99},
            tool_output={"error": "Order not found"},
        ),
    ],
)

classifier = Classifier()
result = classifier.diagnose(trace)
Reporter.print(result)
print(result.failure_type.value, result.subtype, result.confidence_score)

Or load one of the synthetic fixtures in test cases/:

from agent_diagnostician.tracer import load_fixture

trace = load_fixture("test cases/context_loss/context_loss__grounding__dropped_value.json")

More examples: examples/basic_usage.py.

Optional: live LLM judge

Ambiguous cases can call Gemini, OpenAI, or Anthropic. Classifier() never does this on its own.

export LLM_PROVIDER=gemini          # gemini | openai | anthropic | mock
export LLM_API_KEY=your-key         # or GEMINI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY
export LLM_MODEL=gemini-3.5-flash-lite

from agent_diagnostician import Classifier
from agent_diagnostician.analysis.llm import create_llm_judge_from_env

classifier = Classifier(llm_judge=create_llm_judge_from_env())

Smoke test: python scripts/configure_llm.py --test. Rate limits and mock vs live behavior: docs/TROUBLESHOOTING.md.

Run a subset of detectors:

from agent_diagnostician.models.enums import FailureType

classifier = Classifier(
    enabled_detectors=[FailureType.TOKEN_EXHAUSTION, FailureType.INFINITE_LOOP],
)

What it detects

The classifier runs the enabled detectors and returns the single strongest diagnosis (with a fixed priority order on ties).

Most detectors try cheap signals first, then embeddings, then an LLM only if needed:

1. Rules — schemas, constraints, error text
2. Embeddings — similarity (tool ranking, thought vs output, grounding)
3. LLM judge — structured prompts when the first two are inconclusive

Token exhaustion and infinite loop stay on rules/embeddings (no LLM). Hallucination uses the judge on each step. Thresholds live in agent_diagnostician/config.py. Pipeline detail: docs/ARCHITECTURE.md.

Evaluation

53 / 60 (88.3%) on synthetic JSON traces in test cases/ (Gemini gemini-3.5-flash-lite where a live judge was required). 36 fixtures resolved without an API call; 24 used live LLM.

These traces are synthetic and included in the repo so you can re-run the numbers. Known misses (7 fixtures): goal-failure aggregator edges, one classifier tiebreaker, and four tool-use fallback / negative-control cases. Per-fixture detail: docs/evaluation/SUMMARY.md.

python scripts/run_phased_fixture_evaluation.py \
  --initial-used 0 \
  --first-batch 3 \
  --batch-size 15 \
  --output docs/evaluation

Methodology: docs/evaluation/README.md.

Trace format

Minimum JSON:

{
  "run_id": "run_001",
  "task": "User task text",
  "status": "failed",
  "total_steps": 2,
  "final_output": null,
  "steps": [
    {
      "step_index": 0,
      "tool_name": "search",
      "tool_input": {"query": "..."},
      "tool_output": {"results": []},
      "thought": "optional reasoning",
      "error_message": "optional"
    }
  ]
}

Optional: available_tools, constraints, constraint_list, token counts. Schema: agent_diagnostician/models/README.md. Adapters: docs/INTEGRATIONS.md.

Docs

License

MIT — see LICENSE.