agent-self-edit

AgentSelfEdit

A self-improving agent prompt optimizer. Proposes edits to its own system prompt from execution feedback, A/B tests them against a held-out task set, and promotes only statistically-proven winners under deterministic guardrails.

Changelog · v0.2.0 Release Notes · v0.1.0 Field Test Report · v0.2.0 Field Test Report

Why

Most production agents are prompt-tuned once by hand, then frozen at ship time. Every recurring failure is silently absorbed. Two common approaches fall short:

• Reflection appends prose to context but never changes the prompt itself. The same failure repeats.
• Unconstrained LLM-judged edits poison the baseline within a few iterations.

AgentSelfEdit is a sidecar that observes execution traces and proposes prompt edits through a deterministic, evidence-driven loop.

Quick Start

pip install agent-self-edit
agent-self-edit init
agent-self-edit run --once

How It Works

Agent executes task → Execution trace stored (SQLite)
                           ↓
                 Feedback Analyzer (LLM)
                 reviews traces, proposes concrete edits,
                 each with a written hypothesis
                           ↓
            ──── A/B Test Engine ────
             candidate vs current prompt
             held-out task set, bootstrap CI,
             effect size, permutation p-value
            ───────────────────────────
                           ↓
               Promotion Gate (deterministic)
               1. Sample floor     4. Frozen sections
               2. Effect size      5. Edit-distance limit
               3. Confidence p-val 6. Drift detection
                           ↓
                 ┌──────────┼──────────┐
                 ▼          ▼          ▼
             Promoted   Near-miss   Rejected

1. Analyze — An LLM reviews execution traces and identifies failures.
2. Propose — Concrete, minimal prompt edits with stated hypotheses.
3. Test — Each candidate is A/B tested with confidence intervals and effect-size thresholds.
4. Promote or Archive — Winners become the new baseline; losers are archived with reasoning.
5. Guard — Frozen sections, edit-distance limits, and drift detection prevent degradation.

Core Components

The Promotion Gate

Six fail-fast checks before any edit is promoted:

1. Sample floor — minimum A/B trials completed
2. Effect size — minimum improvement threshold
3. Confidence — p < 0.05 (default)
4. Frozen core sections — user-annotated protected regions
5. Edit-distance limit — max lines changed per cycle
6. Drift detection — semantic similarity to original prompt

Field Test Results

v0.1.0

15-iteration loop against Qwen3.5-4B-4bit (local). Gate correctly rejected every edit.

v0.2.0

Validation across 4 domains, adversarial edits, real-trace ingestion, Docker. Strongest analyzer: Mistral Small 24B (OpenRouter). Directional gains but no promotable edit — framework is sound, analyzer quality is the bottleneck.

Full v0.2.0 field test report

Trigger Modes

• Batch — analyze after N tasks (default: 50)
• Time-based — analyze every N hours
• Manual — analyze on demand

Where It Helps

Agents that repeat a task type and see execution feedback: classification, code review, data extraction, content moderation, sales outreach, documentation generation.

Status

Roadmap

License

MIT — see LICENSE