actor-critic-loop

spike-claude-code-actor-critics

Spike — this is a learning experiment, not production software. The goal is to explore the actor-critic pattern, understand how it can refine output quality, and build practical intuition for using it effectively.

Overview

This CLI tool orchestrates an actor and multiple specialised critics by invoking the Claude Code executable as a subprocess. Each role has its own dedicated Claude configuration. Communication uses structured output.

1. The actor generates output (e.g. code, text, a plan)
2. Critics review that output — each focuses on a specific quality dimension
3. The actor refines based on critic feedback
4. The loop repeats until a stopping condition is met

The right orchestration strategy for multiple critics and the best stopping conditions (fixed rounds, critic green-light, consensus, etc.) are open questions this spike aims to answer.

Installation

Install from PyPI:

pip install actor-critic-loop
# or with uv
uv add actor-critic-loop

This installs the actor-critic CLI command. You still need at least one supported coding agent CLI (see below).

Prerequisites

• Python 3.13+
• uv — installation guide
• At least one supported coding agent CLI:
  • Claude Code (default) — installed and authenticated (claude login)
  • Codex — npm i -g @openai/codex
  • OpenCode — https://opencode.ai
  • pi-coding-agent — npm i -g @mariozechner/pi-coding-agent

Setup

git clone https://github.com/ondrasek/spike-claude-code-actor-critics.git
cd spike-claude-code-actor-critics
uv sync

Usage

Configure actor and critics

uv run actor-critic-loop init ./my-task \
  --actor actor_critic.actors.basic \
  --critic correctness=actor_critic.critics.basic \
  --critic style=actor_critic.critics.basic \
  --orchestrator actor_critic.orchestrators.sequential \
  --stop-condition actor_critic.stop_conditions.fixed_rounds

This scaffolds a workspace at ./my-task with dedicated folders for the actor and each critic, containing their Claude Code configurations (CLAUDE.md, settings, skills, subagents). Edit actor-critic.yaml and role configs after init to tune behaviour.

Scaffold from a template

Use --template to clone an existing workspace (e.g. one of the examples) as a starting point:

# Clone an example as your starting point
uv run actor-critic-loop init ./my-task --template ./examples/hello-world/template

# Override specific values while using the template structure
uv run actor-critic-loop init ./my-task --template ./examples/code-review/template --max-rounds 5

# Quick scaffold with defaults (no template needed)
uv run actor-critic-loop init ./my-task

When using --template, the --actor and --critic flags are not allowed (the template defines those). You can still override --orchestrator, --stop-condition, and --max-rounds.

Wizard mode

Use wizard to scaffold a workspace and immediately launch Claude Code for interactive setup:

uv run actor-critic-loop wizard ./my-task

The wizard creates the workspace structure, installs the /customize-workspace skill, and opens Claude Code. Claude guides you through configuring the actor, designing critics, writing prompts, and installing plugins — all interactively. If the workspace already exists, the wizard skips scaffolding and launches Claude Code directly.

Run the actor-critic loop

Place source material in the workspace's input/ directory, then run:

# Basic run (reads everything from actor-critic.yaml)
uv run actor-critic-loop run --workspace ./my-task

# With additional prompt (appended to input/prompt.md)
uv run actor-critic-loop run --workspace ./my-task --prompt "Focus on clarity"

# With verbose output (INFO logging)
uv run actor-critic-loop run --workspace ./my-task --verbose

All configuration — actor, critics, orchestrator, stop condition, and max rounds — lives in actor-critic.yaml at the workspace root.

CLI flags

Use --verbose when debugging issues. Claude Code is always invoked with streaming mode (--output-format stream-json --verbose), which provides real-time diagnostics.

What verbose mode shows:

• Session initialization (model, tools available)
• Tool calls as they happen (Read, Edit, Bash, etc.)
• Final result with token usage and cost

All I/O uses workspace directory conventions — no external path flags:

Exit codes

CLI output

After a run, the CLI displays:

1. Final Output — the actor's last stdout
2. Artifacts — files created/modified by the actor (if any)
3. Round Summary — per-round table showing timing and critic verdicts (PASS/FAIL/PASS*)
4. Critic Verdicts (Final Round) — per-critic verdict with explanation (reconciled verdicts show [reconciled: reason])
5. Result — rounds completed, final verdict (colored), stop reason, total time

PASS* (yellow) indicates a verdict that was reconciled via re-evaluation. This occurs when the orchestrator detects oscillation (a critic cycling between PASS and FAIL) or goalpost-shifting (consecutive FAILs with different issues each round). The suspect critic is re-invoked with its own verdict history and asked to resolve the contradiction.

Note: The exact CLI interface will evolve as the spike progresses.

Multi-Backend Configuration

The actor-critic loop supports multiple coding agent CLIs as backends. Each role (actor, critic) can use a different backend:

# actor-critic.yaml
actor:
  impl: actor_critic.actors.basic
  backend: codex          # Use Codex for the actor

critics:
  - name: quality
    impl: actor_critic.critics.basic
    backend: claude       # Use Claude for the critic

Supported backends:

Override all backends from the CLI: --backend codex

Features not supported by a backend degrade gracefully with logged warnings.

Key Design Decisions

• Claude Code as subprocess, not SDK — keeps the tool decoupled from API changes; each role runs in a full Claude Code environment with its own config, skills, and subagents
• Structured output — implementations control how Claude Code output is formatted and parsed (e.g. --output-format json, stream-json, --json-schema, or plain text)
• Separate configuration per role — each actor/critic gets its own CLAUDE.md, settings, skills, and subagents so roles can be tuned independently without interference
• Self-explanatory configuration — every value in actor-critic.yaml should be understandable without reading the source code; use fully qualified names over short aliases
• Plugins over MCP servers — prefer official Anthropic plugins, custom skills, and rules over MCP servers for simplicity and reliability
• No future-proofing — don't add parameters, code, or abstractions for features planned in later phases; add them when needed

Plugins, Skills, and Rules

Role workspaces can be enhanced with Claude Code capabilities in priority order:

1. Rules — auto-loaded guidance in .claude/rules/ (always use output.md to constrain writes)
2. Skills — custom slash commands in .claude/skills/ for specific workflows
3. Plugins — official Anthropic plugins for code intelligence and review
4. MCP servers — avoid when possible; use only for external service integration

Installing Plugins

# Install plugins to actor workspace (project scope)
cd my-task/actor
claude plugin install pyright-lsp --scope project
claude plugin install code-simplifier --scope project

# Install plugins to critic workspace
cd my-task/critics/correctness
claude plugin install code-review --scope project
claude plugin install security-guidance --scope project

Recommended Plugins

Skills and Rules

Skills provide custom slash commands:

.claude/skills/
└── generate/
    └── SKILL.md    # Instructions for /generate command

Rules provide auto-loaded guidance:

.claude/rules/
├── output.md       # Required: constrain writes to ./output/
└── style.md        # Optional: code style guidelines

See plans/PLUGINS-SKILLS-RULES.md for complete documentation.

Invocation Model

Each actor and critic is invoked as a Claude Code subprocess in non-interactive mode (claude -p). The orchestrator runs Claude Code from the role's workspace directory so it picks up the role's CLAUDE.md, settings, skills, and subagents. Roles access shared directories via --add-dir.

Actor invocation:

cd ./my-task/actor && claude -p "the prompt and critic feedback" \
  --add-dir ../input \
  --add-dir ../critics/correctness/output \
  --add-dir ../critics/style/output

Critic invocation:

cd ./my-task/critics/correctness && claude -p "<rendered prompt>" \
  --add-dir ../../input --add-dir ../../actor/output

Critic prompts are Jinja2 templates stored in critics/<name>/input/prompt.md. At runtime, the template is rendered with context about the actor's work:

Templates use XML sections following Anthropic prompt engineering guidelines (data/context first, instructions in middle, reminder at end):

<actor_task>
The original requirements given to the actor are in `{{ actor_prompt_file }}`.
</actor_task>

<actor_output>
The actor produced the following files:
{% for file in actor_output_files %}
- `{{ file }}`
{% endfor %}
</actor_output>

<review_instructions>
[critic-specific evaluation criteria]
</review_instructions>

<reminder>
- Ground your review in specific evidence.
- Your response MUST be valid JSON with `review`, `passed`, and optionally `issues`.
</reminder>

Every critic must provide its own template — there is no default fallback.

The actor writes its working output to <workspace>/actor/output/. Critics review it there via --add-dir and may produce optional artifacts in their own output/ directories. The actor can access critic artifacts via --add-dir to each critic's output/. After the final round, the orchestrator copies the actor's output to <workspace>/output/.

Each role is constrained to write only to its own output/ subdirectory via permissions in settings.json and a rule in .claude/rules/output.md.

Output format, schema enforcement, and response parsing are implementation-specific — each actor/critic implementation decides which Claude Code flags to use.

Session strategy

Whether a role resumes its Claude Code session across rounds or starts fresh is configurable. Both approaches have trade-offs:

The right strategy depends on the role and the task — this is one of the things the spike aims to explore empirically. Session strategy is configured per role in actor-critic.yaml.

Oscillation prevention

The most significant failure mode in actor-critic loops is critic oscillation — where fixing one critic's feedback breaks another's requirements, or a single critic cycles between PASS and FAIL across rounds. The orchestrator includes automatic oscillation detection and reconciliation:

Two triggers:

How reconciliation works:

1. After each round, the orchestrator checks failing critics for suspicious patterns
2. Suspect critics are re-invoked with their own verdict history and a reconciliation prompt
3. The critic reviews its prior verdicts and determines whether the current FAIL is genuine or self-contradictory
4. If the critic reconciles to PASS, the verdict is updated and marked as PASS* (reconciled)

Configuration (in actor-critic.yaml):

stop_condition:
  config:
    max_rounds: 5
    max_consecutive_fails: 4   # default: 3
    oscillation_reversals: 3   # default: 2

Reconciliation adds zero overhead on clean runs — detection is a pure function over round summaries. Only triggered critics incur an extra invocation.

Project Structure

├── src/
│   └── actor_critic/
│       ├── __init__.py
│       ├── cli.py
│       ├── types.py
│       ├── claude/                # Shared Claude Code invocation
│       │   ├── __init__.py
│       │   ├── invoke.py          # Core invoke() function
│       │   ├── events.py          # NDJSON event parsing
│       │   └── types.py           # ClaudeResult, ClaudeEvent
│       ├── config.py              # YAML config loading
│       ├── resolver.py            # Dynamic module import
│       ├── scaffold.py            # Workspace scaffolding (init command)
│       ├── scaffold_templates/    # Built-in templates for init
│       ├── backends/              # Coding agent CLI backends
│       │   ├── base.py            # BackendResult, BackendEvent
│       │   ├── resolve.py         # Name → module resolver
│       │   ├── claude.py          # Claude Code (wraps claude/invoke.py)
│       │   ├── codex.py           # OpenAI Codex CLI
│       │   ├── opencode.py        # OpenCode CLI
│       │   └── pi.py              # pi-coding-agent CLI
│       ├── actors/
│       │   ├── __init__.py
│       │   ├── basic.py
│       │   └── team.py            # Agent teams (parallel work)
│       ├── critics/
│       │   ├── __init__.py
│       │   └── basic.py
│       ├── orchestrators/
│       │   ├── __init__.py
│       │   ├── sequential.py
│       │   ├── oscillation.py     # Oscillation detection + reconciliation
│       │   └── feedback.py        # Critic feedback serialization
│       └── stop_conditions/
│           ├── __init__.py
│           ├── all_pass.py        # Stop when all critics pass
│           ├── any_pass.py        # Stop when any critic passes
│           ├── fixed_rounds.py    # Stop at max rounds
│           └── quorum.py          # Stop when N critics pass
├── tests/
│   ├── system/
│   ├── integration/
│   └── unit/
├── pyproject.toml
└── README.md

Note: The structure above represents the planned/target layout for this spike and not all directories or files may exist yet in the repository.

Example workspace

A workspace is created by actor-critic-loop init and contains the full Claude Code configuration for each role:

my-task/
├── actor-critic.yaml
├── input/                 # Source material (user-provided before run)
├── output/                # Final output (written by orchestrator)
├── snapshots/               # Created at runtime — actor output history
│   └── actor/
│       ├── round-1/         # Copy of actor/output/ after round 1
│       └── round-2/         # Copy of actor/output/ after round 2
├── actor/
│   ├── output/            # Actor's working output (intermediate, per-round)
│   ├── CLAUDE.md
│   └── .claude/
│       ├── settings.json  # Permissions: writes to ./output/ only
│       ├── rules/
│       │   └── output.md  # "Write all output files to ./output/"
│       ├── agents/
│       │   └── researcher.md
│       ├── commands/
│       │   └── generate.md
│       └── skills/
│           └── code-generation/
│               └── SKILL.md
└── critics/
    ├── correctness/
    │   ├── input/
    │   │   └── prompt.md  # Jinja2 template — critic review prompt
    │   ├── output/        # Critic's optional output artifacts
    │   ├── CLAUDE.md
    │   └── .claude/
    │       ├── settings.json
    │       ├── rules/
    │       │   └── output.md
    │       ├── agents/
    │       │   └── test-verifier.md
    │       └── skills/
    │           └── verify-logic/
    │               └── SKILL.md
    └── style/
        ├── input/
        │   └── prompt.md  # Jinja2 template — critic review prompt
        ├── output/
        ├── CLAUDE.md
        └── .claude/
            ├── settings.json
            ├── rules/
            │   └── output.md
            └── skills/
                └── check-style/
                    └── SKILL.md

actor-critic.yaml

Top-level orchestration configuration that ties the workspace together:

actor:
  path: ./actor
  impl: actor_critic.actors.basic

critics:
  - name: correctness
    path: ./critics/correctness
    impl: actor_critic.critics.basic
  - name: style
    path: ./critics/style
    impl: actor_critic.critics.basic

orchestrator:
  impl: actor_critic.orchestrators.sequential

stop_condition:
  impl: actor_critic.stop_conditions.all_pass
  config:
    max_rounds: 3
    max_consecutive_fails: 3    # optional: trigger reconciliation after N consecutive FAILs
    oscillation_reversals: 2    # optional: trigger reconciliation after N verdict reversals

Testing

Tests are run with pytest and organised into:

• CLI / system tests — end-to-end verification of the console interface
• Integration tests — interaction between actor and critic components
• Unit tests — where they add clear value

# Run all tests with coverage
uv run pytest --cov

Quality Gate Hooks

This project uses Claude Code hooks to enforce code quality standards. When Claude completes work, a quality gate automatically runs and feeds any failures back to Claude for fixing, creating an iterative loop until all checks pass.

Iterative feedback loop

┌─────────────────────────────────────────────────────────────────┐
│                    ITERATIVE FEEDBACK LOOP                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Claude completes work                                         │
│         │                                                       │
│         ▼                                                       │
│   Stop event fires ──► Quality gate hook runs                   │
│         │                                                       │
│         ├── Any check fails ──► Exit 2 ──► Claude fixes ──┐     │
│         │                                                 │     │
│         ▼                                                 │     │
│   All checks pass ──► Auto-commit hook runs               │     │
│         │                                                 │     │
│         ├── Pre-commit fails ──► Exit 2 ──► Claude fixes ─┤     │
│         │                                                 │     │
│         ▼                                                 │     │
│   Commit + push ──► Session ends cleanly                  │     │
│                                                           │     │
│         ◄─────────────── Stop event fires again ──────────┘     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

The hooks are configured in .claude/settings.json and are mandatory for all developers. Exit code 2 feeds stderr back to Claude, prompting automatic fixes.

Quality checks

Environment variable overrides

Developers can temporarily adjust checks without modifying shared configuration:

# Skip slow checks during rapid iteration
SKIP_TYPE_CHECK=1 SKIP_SECURITY_SCAN=1 claude

# Lower coverage threshold temporarily
COVERAGE_THRESHOLD=60 claude

Running checks manually

# Run the full quality gate
./.claude/hooks/quality-gate.sh

# Run individual checks
uv run pytest -x --tb=short
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/
uv run pyright src/
uv run bandit -r src/ -ll
uv run vulture src/ --min-confidence 80

Hook execution order

PreToolUse (before every Write/Edit/MultiEdit):

1. tdd-gate.sh — blocks implementation writes without a failing test (60s timeout)

PostToolUse (after every Write/Edit/MultiEdit):

1. per-edit-fix.sh — ruff fix + format + codespell (60s timeout)
2. post-edit-tests.sh — runs pytest for immediate feedback (120s timeout)

Stop (when Claude finishes):

1. Code review agent — inspects changed files (120s timeout)
2. quality-gate.sh — full test suite + lint + types (10 min timeout)
3. auto-commit.sh — commit + push if clean (4 min timeout)

If any hook returns exit 2, Claude receives stderr and fixes issues. The loop continues until all checks pass.

TDD workflow with hooks

1. Write failing test → Claude writes test file (tdd-gate.sh allows test files)
2. Post-edit tests run → post-edit-tests.sh reports the new test fails (expected)
3. Write implementation → tdd-gate.sh allows (failing test exists)
4. Post-edit tests run → post-edit-tests.sh reports pass/fail immediately
5. Claude finishes → Code review agent inspects changes
6. Quality gate → Full suite + lint + types + coverage
7. All pass → Auto-commit, session complete

TDD hook environment variables

# Disable TDD gate during non-code tasks
SKIP_TDD_GATE=1 claude

# Disable post-edit tests during rapid iteration
SKIP_POST_EDIT_TESTS=1 claude

# Disable both for non-TDD workflows
SKIP_TDD_GATE=1 SKIP_POST_EDIT_TESTS=1 claude

Future

See docs/FUTURE.md.