ai-agentreflect

agentreflect

Generate AGENTS.md rules from AI coding agent failure logs. Closes the feedback loop in the agent quality trilogy.

Measure (coderace) → Generate (agentmd) → Guard (agentlint) → Learn (agentreflect)

What it does

Every developer using Claude Code or Codex has this problem: their agent makes a mistake, they fix it manually, update AGENTS.md, and hope it doesn't happen again. agentreflect automates the "update AGENTS.md" step.

Feed it failure logs → get targeted rule suggestions → apply them to your AGENTS.md.

In notes mode, agentreflect now prefers AGENTS-style operational rules over generic QA advice. If a note says the agent missed an exact routing target, skipped downstream verification after an irreversible action, or relied on fuzzy recall instead of exact facts already present in files, the suggestions will mirror that operating rule directly.

Install

pip install ai-agentreflect

For LLM-enhanced mode:

pip install 'ai-agentreflect[llm]'

Usage

From pytest output

# Capture failures
pytest --tb=short 2>&1 | tee failures.txt

# Generate suggestions
agentreflect generate --from-pytest failures.txt

From git log

agentreflect generate --from-git

Analyzes fix:, bug:, revert: commits and agent-related mistake commits.

From plain text notes

agentreflect generate --from-notes "agent forgot to check for None before accessing .value"
agentreflect generate --from-notes "fuzzy retrieval missed explicit facts already present in files; add an exact-match layer for handles and tool routing before semantic lookup"

CI Integration

agentreflect ships a GitHub Action and a CI mode flag to integrate rule generation into your pull request workflow.

GitHub Action quick-start

# .github/workflows/agentreflect.yml
name: agentreflect
on: [pull_request]

jobs:
  learn:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: mikiships/agentreflect@v0.3.0
        with:
          mode: ci
          from-git: 'true'
          threshold: '10'   # fail PR if >= 10 new rules detected

CI mode (machine-readable output)

# Write JSON to stdout, summary to stderr
agentreflect generate --from-pytest failures.txt --ci

# Fail with exit 1 if >= 5 rules generated (useful for PR gates)
agentreflect generate --from-git --ci --fail-on 5

CI mode JSON output:

{
  "ci": true,
  "rules_count": 3,
  "rules_generated": 3,
  "source": "git log",
  "suggestions": [
    { "rule": "...", "rationale": "...", "confidence": 0.85, "category": "..." }
  ]
}

Full quartet pipeline

See .github/workflows/examples/agent-quality-pipeline.yml for a complete MEASURE → GENERATE → GUARD → LEARN pipeline running all four agent quality tools in sequence:

1. agentlint — lint AGENTS.md for structural issues
2. agentmd — check context file freshness
3. coderace — review code changes
4. agentreflect — extract new rules from failures

Output formats

Markdown (default)

## agentreflect suggestions (2026-03-11)

### From: pytest failures (failures.txt)
- [ ] Always check for None before attribute access: use `if obj is not None` or `hasattr(obj, 'attr')`
- [ ] When catching AttributeError, log the object type with `type(obj).__name__`

_Source: 3 failures analyzed, 2 suggestions generated_

Diff format

agentreflect generate --from-pytest failures.txt --format diff

Outputs a unified diff ready to apply to AGENTS.md.

Apply directly

agentreflect generate --from-notes "agent used wrong variable" --apply AGENTS.md
# Asks for confirmation

agentreflect generate --from-pytest failures.txt --apply AGENTS.md --yes
# Applies without confirmation

LLM-enhanced mode

export ANTHROPIC_API_KEY=your_key_here
agentreflect generate --from-pytest failures.txt --llm

Uses claude-3-5-haiku-latest for contextual, specific suggestions tailored to your actual failures. Cost: ~$0.001 per analysis.

Basic pattern mode works without any API key.

Integration with the trilogy

License

MIT