agentdrift

AgentDrift

Detect when your AI agents get dumber.

Open-source quality drift monitoring for AI agents. Track agent outputs over time, detect quality degradation, and get alerts before your agents go off the rails.

Live Demo | API Docs | Roadmap | Contributing

The Problem

AI agents degrade over time. Context pollution, error accumulation, prompt drift, and model updates silently erode output quality. By the time you notice, your users already have.

Quick Start

pip install agentdrift

from agentdrift import track_agent

@track_agent("my-agent")
def my_agent(query: str) -> str:
    return llm.call(query)

Check drift status:

agentdrift status

┌──────────────────────────────────────┐
│         Agent Drift Status           │
├──────────┬──────────┬───────┬───────┤
│ Agent    │ Status   │ Score │ Change│
├──────────┼──────────┼───────┼───────┤
│ my-agent │ STABLE   │ 0.847 │ +2.1% │
│ qa-bot   │ DRIFTING │ 0.612 │-18.3% │
└──────────┴──────────┴───────┴───────┘

How It Works

Agent Call → Log Trace → Score Quality → Detect Drift → Alert
                                              ↓
                                    Dashboard / OTel Export

1. Track — Log every agent input/output with the SDK
2. Score — LLM-as-Judge rates output quality (0.0 - 1.0)
3. Compare — Recent scores vs baseline period
4. Alert — Notify when quality drops below threshold

Key Features

• 1-line integration — @track_agent decorator or manual tracker.log()
• Web dashboard — Real-time quality trends, trace trees, golden tests management
• Golden tests — Regression testing from your best agent outputs
• Alert webhooks — Slack, Discord, Email on drift detection
• Multi-agent tracing — Parent/child trace relationships
• OpenTelemetry export — Jaeger, Zipkin, any OTLP backend
• REST API — Language-agnostic trace ingestion (POST /api/traces)
• TypeScript SDK — @agentdrift/sdk for Node.js agents
• PostgreSQL — Production-ready storage (pip install agentdrift[postgres])
• Docker + Helm — Kubernetes deployment ready

Usage

Web Dashboard

pip install agentdrift[dashboard]
agentdrift serve        # API server on :8888
cd web && npm run dev   # Dashboard on :3000

Manual Logging

from agentdrift import AgentTracker

tracker = AgentTracker("my-agent")
tracker.log(input=query, output=result, latency_ms=1200)

Multi-Agent Tracing

orchestrator = AgentTracker("orchestrator")
worker = AgentTracker("worker")

parent = orchestrator.log(input="plan task", output="delegating")
worker.log(input="subtask", output="result", parent_trace_id=parent.trace_id)

HTTP API

curl -X POST http://localhost:8888/api/traces \
  -H "Content-Type: application/json" \
  -d '{"agent_name": "my-agent", "input": "hello", "output": "world"}'

TypeScript SDK

import { AgentDriftClient, AgentTracker } from "@agentdrift/sdk";

const client = new AgentDriftClient({ baseUrl: "http://localhost:8888" });
const tracker = new AgentTracker("my-agent", client);
await tracker.log({ input: "hello", output: "world" });

Golden Tests

agentdrift test init my-agent --count 10 --min-score 0.8
agentdrift test run my-agent --module my_app:my_agent --threshold 0.7

Configuration

agentdrift init --config   # Generate agentdrift.toml

See examples/ for more integration patterns.

Deployment

# Docker
docker compose up

# Kubernetes
helm install agentdrift chart/agentdrift/

# Vercel (dashboard only)
cd web && vercel

Documentation

License

MIT