agentic-code-reviewer

Agentic Code Reviewer

Multi-Agent LLM Code Review with Layered Deterministic Verification

A production-grade, agentic code reviewer. It analyzes real software changes the way a senior engineer would - planning, targeted repository context, staged analysis across 23 evidence-backed categories, layered deterministic verification (regex, AST, symbol, taint), and a final aggregated review - then measures whether that agentic approach actually beats single-pass review with a leakage-controlled benchmark. Ships as the acr CLI with a full-screen Textual terminal dashboard, works with six LLM providers, and needs no .env file: everything is configured from the CLI or the TUI.

---

Table of Contents

• The Problem
• What This Does
• Key Results
• The 23 Review Categories
• Verification & Evidence
• Architecture
• Repository Structure
• Installation
• Quick Start
• Usage (CLI)
• Interactive Terminal Dashboard
• Providers
• Quality Gates (CI)
• Repository Policy (.reviewer.yaml)
• Machine-Readable Output
• Benchmarking & Evaluation
• Development
• Configuration Reference
• Troubleshooting
• Limitations
• Related Work
• Citation
• License

---

The Problem

Code review is a verification problem disguised as a language problem. A single LLM call against a diff produces fluent prose with no guarantee any of it is true - findings get invented, lines get cited that were never changed, and there is no mechanism that catches a hallucinated defect before it reaches the developer. Manual review avoids this failure mode but doesn't scale: reading, cross-referencing, and judging every changed line by hand is exactly the high-volume, well-defined cognitive labor automation should be doing.

The core requirement is not "generate findings." It is "generate findings that have been checked."

Agentic Code Reviewer answers this by never letting a single model both write and grade its own work - every claim is re-verified against the actual repository, the real diff, and the changed lines before it ships.

---

What This Does

1. 23 review categories - 22 LLM analysis agents plus a deterministic dead-code checker that needs no LLM call
2. Multi-step agentic workflow - planner, change understanding, parallel category analysis, evidence verification, cross-category synthesis, and aggregation into a final review
3. Layered deterministic verification - every LLM finding is checked against regex, AST, symbol, and taint layers and assigned an evidence status: confirmed, strongly_supported, insufficient_evidence, or rejected
4. Lightweight taint analysis - user-controlled sources are tracked to SQL, shell, filesystem, HTTP, deserialization, and log sinks, with sanitizer and parameterized-query awareness
5. Shared repository snapshot - one file / AST / symbol / reference / route / env-var / test index is built per review and queried by every stage, so reviewers never re-read or re-parse files
6. Cross-category correlation - overlapping findings are clustered into canonical findings with primary_category and related_categories; cross-change interactions are detected as materially new defects
7. Deliberate context, never "dump the repo" - symbol-aware chunking and semantic retrieval (OpenAI / Gemini / Ollama / OpenAI-compatible embeddings, with a deterministic local TF-IDF fallback) under bounded context budgets
8. Large diffs handled intelligently - token-aware decomposition into file groups with per-group parallel analysis and merged findings (no arbitrary truncation)
9. Interactive terminal dashboard - a full-screen Textual TUI with a launcher, live pipeline view, severity-coloured findings browser, per-category grouping, and command palette
10. CI-ready quality gates - --fail-on <severity> exits non-zero when the gate trips; per-repo policy files drive gates for every contributor
11. Machine-readable output - JSON and SARIF 2.1.0 (GitHub Advanced Security / GitLab code scanning / VS Code), with human status on stderr so stdout stays parseable
12. Six LLM providers - OpenAI, Anthropic, Google Gemini, any OpenAI-compatible endpoint (vLLM, LM Studio, Groq, OpenRouter, DeepSeek), Ollama locally, and a deterministic mock for tests
13. Reproducible benchmark harness - single-pass vs context vs RAG vs agentic systems compared under identical conditions, with leakage-controlled datasets
14. Persistent knowledgebase - curated rules/conventions, auto-captured verified findings (repo memory), and indexed docs injected into analysis prompts via $KNOWLEDGE$; deterministic token-overlap retrieval with zero extra token cost
15. Token-aware by design - category-focused diff views per agent, compact plan/summary digests, Anthropic prompt caching on the shared system block, and per-agent output caps keep cost low on every run
16. Secure by design - repository content is treated as untrusted data (prompt-injection defenses), reviewed application code is never executed, and secrets are never logged

---

Key Results

432 tests passing, ruff clean, mypy clean across 100 source files, and a fully offline test suite - provider calls are transport-mocked and workflow/UI tests run against deterministic mock clients.

An honest benchmark: metrics are empty until experiments actually run - nothing is fabricated. Run acr benchmark (keyless with the mock provider) to reproduce the full harness yourself.

---

The 23 Review Categories

The reviewer judges a change across 23 evidence-backed categories. All share the same hard rules: concrete evidence tied to a changed line, severity critical → info, confidence in [0, 1], and style preferences are never reported.

---

Verification & Evidence

Findings do not flow from the LLM straight to the report. Each one is resolved against the repository first:

LLM finding
    ↓
Evidence resolver
    ↓
Regex verification
    ↓
AST verification
    ↓
Symbol/reference verification
    ↓
Semantic verification (taint)
    ↓
Evidence status

Only confirmed and strongly_supported findings normally reach the user (configurable via VERIFICATION_STRICTNESS). Deterministic confirmation is never fabricated - if the evidence cannot be established, the finding does not ship.

---

Architecture

Code Change (GitHub PR / commit / local git)
        ↓
Diff ingestion → Planner → Change Understanding → Context Retrieval (RAG)
        ↓
  22 analysis agents in parallel        Dead Code (deterministic, no LLM)
  correctness, security, testing,          ↓
  regression, performance, ...       Evidence verification (regex → AST → symbol → taint)
  authorization, reliability,              ↓
  architecture, compatibility,        Cross-category correlation + interaction analysis
  configuration, resource lifecycle       ↓
                                    Aggregation → Final Review (Markdown / JSON / SARIF)

Every stage is a typed component (pydantic in/out) with its own timeout, retry policy, structured error recording, and observability. LLM failures degrade gracefully - a single failed reviewer never destroys the review; the workflow continues and records the stage error. The workflow streams typed events (orchestration/events.py) so the TUI renders live progress without blocking the review thread.

---

Repository Structure

agentic-code-reviewer/
|
+-- src/agentic_code_reviewer/
|   +-- agents/            planner, change_analyzer, 22 analysis agents + a
|   |                      deterministic dead-code checker, verifier, aggregator
|   +-- orchestration/     workflow state machine, review state, typed event stream
|   +-- analysis/          diff parsing, repository snapshot, AST rules, taint
|   |                      engine, severity model, category rules, aggregation
|   +-- retrieval/         symbol-aware chunking, embeddings, vector index
|   +-- knowledge/         persistent knowledgebase: models, JSONL store,
|   |                      token-overlap retrieval, auto-capture, prompt renderer
|   +-- github/            GitHubClient, models, adapters
|   +-- llm/               provider clients (OpenAI, Anthropic, Gemini, Ollama,
|   |                      OpenAI-compatible, mock), retries, structured output
|   +-- evaluation/        benchmark, baselines, metrics, experiments, runner
|   +-- cli/               typer CLI: review commands, init, completion, exporters,
|   |                      interactive triage, knowledge management
|   +-- ui/                Textual TUI: apps, screens, widgets, theme
|   +-- config/  models/  observability/  local/  prompts/
|
+-- tests/                 unit + integration + failure-path tests (fully offline)
+-- benchmarks/            datasets (fixture + importer output), metadata, results
+-- configs/               example experiment YAML
+-- docs/                  architecture, research, security, screenshots
+-- pyproject.toml
+-- LICENSE

---

Installation

# From PyPI
pip install agentic-code-reviewer

# From source (development / full extras)
git clone https://github.com/royxforge/agentic-code-reviewer.git
cd agentic-code-reviewer
pip install -e ".[all]"

Requirements: Python 3.11+ · pydantic 2.12+ · typer 0.15+ · rich 13.9+ · textual 8.0+ (provider SDKs are optional extras)

Optional extras: .[openai] for the OpenAI SDK, .[anthropic] for the Anthropic SDK, .[dev] for testing/linting tooling, .[all] for everything. Gemini and Ollama speak plain HTTP - no SDK required.

---

Quick Start

# 1. One-time provider setup (stored in your user config, no .env file)
acr get-started

# 2. Review local uncommitted changes
acr review-local . --output review.md

# 3. Watch it run live in the terminal dashboard
acr tui review-local .

One word, like claude: install the global launcher once (adds acr to your PATH), then from any git project just type:

powershell -ExecutionPolicy Bypass -File scripts/install-cli.ps1

acr                 # launcher menu on the current directory
acr path/to/project # launcher menu on any other git project
acr --help          # all commands

On the very first run (no provider configured yet) a welcome screen auto-opens and walks you through provider setup - or lets you explore instantly with the keyless mock provider. If the working tree has no uncommitted changes, a review automatically falls back to the last commit. The old reviewer command is kept as a backwards-compatible alias.

No key handy? The mock provider runs a deterministic test double (never a real model) - the keyless smoke path for the whole pipeline:

LLM_PROVIDER=mock acr benchmark --dataset benchmarks/datasets/fixture_small.json \
    --systems single-pass,agentic --limit 2

---

Usage (CLI)

Common options:

• --output review.md - write the review as Markdown
• --format markdown|json|sarif - output format (default markdown)
• --fail-on <severity> - quality gate: exit 1 when a finding at or above this severity exists
• --interactive / -i - triage findings after the review: apply (via git apply), dismiss, copy, skip, or quit
• --apply - non-interactively apply every suggested fix to the working tree
• --publish - post the review on GitHub (requires a write-scoped GITHUB_TOKEN)
• --base <ref> - diff against a different base ref for review-local (default HEAD)
• --requirement "..." - stated intent to review against; feeds the planner and the requirement-alignment check
• --provider <name> - override LLM_PROVIDER for a single run (benchmark commands)

acr review --repo owner/repository --pr 123
acr review-commit --repo owner/repository --commit abc123
acr review-local . --output review.md
acr review --repo owner/repository --pr 123 --publish
acr review-local . --requirement "Search must be case-insensitive"

---

Interactive Terminal Dashboard

Typing acr opens a full-screen Textual launcher; the live pipeline, findings browser, and everything else live inside the same session. Logs are redirected to a temp file while it runs, so the screen is never corrupted.

Real screenshots (regenerate anytime with python scripts/screenshot_tui.py):

• Boot splash - the brand moment when acr launches: block-letter logo, product name, version, and an animated loading line; auto-dismisses after ~1.8s or skips on any key
• Launcher - the Claude-Code-style home screen: review this directory, review another path, providers & API keys, history, help, or quit (↑/↓ or j/k + enter, or the number keys)
• Pipeline - animated stage tracker (Planning → Analysis → Synthesis), live severity counters, usage ticker (tokens, cost, LLM calls, elapsed)
• Results - severity-pill summary, findings table, detail card with evidence, impact, and recommendation
• Categories (c from results) - every finding grouped by check type across all 23 categories, with live severity chips and zero-count categories kept visible
• Benchmark - system chips, live progress bar, per-run log, final comparison table
• Providers - status chips per provider (active / configured / environment), masked key editing, ctrl+s saves to your user config
• History - recent reviews with severity counts, model, and cost
• Help - in-app reference: launcher keys, per-screen keybindings, CLI commands, provider notes
• Command palette (ctrl+p) - review, providers, history, help, exports (Markdown / JSON / SARIF), clear history, open config folder

---

Providers

Set LLM_PROVIDER (in the environment or via acr get-started) and the matching credentials. Both openai-compatible and openai_compatible spellings are accepted.

The Gemini client talks to the v1beta REST API over plain HTTP with responseMimeType: application/json for structured output. Any endpoint speaking the OpenAI chat-completions protocol works as openai-compatible. EMBEDDING_PROVIDER=auto picks the embedding model that matches your LLM provider and falls back to a deterministic local TF-IDF embedder - RAG works fully offline.

---

Quality Gates (CI)

Every review command accepts --fail-on <severity>: when the review contains a finding at or above that severity, the command exits with code 1 - the standard hook for CI/CD. Human status always goes to stderr; stdout stays clean for the machine format.

# Fail the pipeline on any critical or high finding
acr review-local . --fail-on high

# Machine-readable report + gate, in one shot (CI-friendly)
acr review-local . --format json --fail-on high > review.json

# .github/workflows/code-review.yml (GitHub Actions)
- name: Agentic code review
  run: acr review-local . --fail-on high --format sarif \
       --output ${{ github.workspace }}/review.sarif

--fail-on beats the repo policy; without the flag, the policy in .reviewer.yaml applies.

---

Repository Policy (.reviewer.yaml)

Run acr init to scaffold a policy file, or write one by hand. It is discovered upward from the reviewed path, so the file checked into your repo controls every contributor's run. Unknown keys are rejected loudly (a typo'd policy must never silently do nothing).

# .reviewer.yaml
quality_gate:
  critical: 0      # fail when any critical finding exists
  high: 5          # fail when more than 5 high findings exist
  medium: 20
  low: null        # never fail on low / info
  info: null

ignore_patterns:            # findings on these paths are dropped
  - "**/test_*.py"
  - "**/tests/**"
  - "docs/**"
  - "*.lock"

severity_weights:
  critical: 1.0
  high: 0.8
  medium: 0.5
  low: 0.2
  info: 0.0

# Optional model overrides (any Settings field, kebab or underscore)
# max_tokens: 8192
# temperature: 0.2
# providers:
#   llm_provider: openai
#   openai_model: gpt-4o-mini

---

Knowledgebase

acr can remember across reviews. The knowledgebase is a persistent store (JSONL under your user config dir, config_dir()/knowledge/) of three kinds of entries, all injected into analysis prompts via the $KNOWLEDGE$ block:

acr knowledge add --repo owner/repo --category security \
    --title "Parameterized SQL required" \
    --content "Never interpolate user input into SQL strings; use ? placeholders."

acr knowledge list --repo owner/repo
acr knowledge search "sql injection" --repo owner/repo
acr knowledge index-docs docs/ --repo owner/repo

Retrieval is deterministic token-overlap scoring - no LLM call and no embedding endpoint - so the knowledgebase adds zero token cost and works fully offline. The next review of the same repository sees the entries most relevant to the current diff and can reference past decisions instead of re-deriving them.

Learning loop: the knowledgebase is not just one-way memory. Dismiss a finding in interactive triage (-i) and a dismissed entry is recorded for that repository - future reviews suppress findings that match it (same category + strong token overlap, or same file + category). The reviewer learns "we already decided this class of issue is acceptable here" from your decisions.

Token Usage

Every run is token-aware by design, and several levers cut cost on the 22 parallel analysis calls:

• Category-focused diffs: each analysis agent only receives the hunks its deterministic patterns care about (diff_text_for_category), falling back to the full diff when nothing matches. The full diff is still what the evidence verifier checks against.
• Compact digests: prompts receive only the plan/change-summary fields agents actually read, not the full model dumps.
• Anthropic prompt caching: the identical system block is marked cache_control (opt-out: USE_PROMPT_CACHING=false), so parallel agents pay one full read + cache reads.
• Per-agent output caps: ANALYSIS_MAX_TOKENS=4096 (vs the global MAX_TOKENS=8192).
• Bounded knowledge: KNOWLEDGE_MAX_CHARS caps the injected $KNOWLEDGE$ block.
• Aggregator cap: AGGREGATOR_MAX_FINDINGS (default 60) limits the findings serialized into the aggregator prompt.
• LLM response cache: with LLM_CACHE_ENABLED=true, structured responses are stored on disk keyed by prompt content + model + params, so repeat reviews of identical diffs (CI) skip the provider entirely.

Watch the live usage ticker (tokens, cost, LLM calls) in the TUI pipeline screen, or read token_usage / estimated_cost_usd from --format json output.

Machine-Readable Output

• --format json - canonical machine view of the review (findings, metrics, model, cost, token usage) for dashboards and downstream tooling
• --format sarif - SARIF 2.1.0, so findings plug into GitHub Advanced Security, GitLab code scanning, VS Code, and CI dashboards
• --format markdown (default) - the human-readable report

With --format json|sarif the human summary is printed to stderr, keeping stdout strictly parseable. Interactive triage (--interactive / --apply) walks findings after the review and can apply suggested fixes to the working tree via git apply (never stages or commits).

---

Benchmarking & Evaluation

Four systems are compared under identical conditions:

• single-pass - one LLM call over the whole diff
• context - single-pass with targeted repository context
• rag - single-pass with semantic retrieval over the repository
• agentic - the full multi-step workflow (planner → agents → verifier → aggregator)

# Run all four systems over the fixture dataset (keyless: --provider mock)
acr benchmark --dataset benchmarks/datasets/fixture_small.json \
    --systems single-pass,context,rag,agentic

# Reproducible experiment from a YAML config
cp configs/experiment.yaml.example configs/experiment.yaml
acr evaluate --config configs/experiment.yaml

Experiment config keys: name, dataset, systems (list or comma string), limit (or null), settings (any Settings field, e.g. ablation switches like workflow_use_retrieval: false), notes. Results - metrics, prompts, per-entry artifacts - land in an immutable, timestamped directory under experiments/.

Fetch real historical-bug data (SWE-bench; requires pip install datasets):

python scripts/fetch_swebench.py --output benchmarks/datasets/swebench_review.jsonl --limit 20

---

Development

pip install -e ".[dev]"           # or ".[all]" for providers + dev tools

pytest                            # 432 tests: unit + integration + failure-path
ruff check src/ tests/            # lint
mypy src/agentic_code_reviewer            # type checking

The test suite is fully offline: provider calls are transport-mocked, and the workflow/UI tests run against the deterministic mock and AdaptiveMock clients. The TUI screenshots are regenerated with python scripts/screenshot_tui.py.

---

Configuration Reference

All settings are read from environment variables and map to Settings fields in src/agentic_code_reviewer/config/settings.py. Providers and API keys are normally set through acr get-started or the TUI Providers screen, which persist them to your user config.

---

Troubleshooting

---

Limitations

• LLM-dependent quality: review quality depends on the configured model and prompt versions; results are nondeterministic by nature.
• GitHub write access is opt-in and requires a token with the appropriate scope; read-only review works without publishing.
• Benchmark data is populated via the importer; the committed fixture dataset is synthetic (CI/smoke testing only).
• Execution-based verification is not enabled: repository code is never executed; verification is evidence-based (diff/pattern/symbol/taint checks).

---

Related Work

• Multi-Agent Research System - The same "never let one model grade its own work" principle applied to literature synthesis: its Critic verifies claims against sources the way Agentic Code Reviewer's evidence resolver verifies findings against the repository.
• RAG Evaluation Framework - LLM-judged faithfulness scoring and hallucination-rate metrics; Agentic Code Reviewer's layered verification is the code-review analogue of that evaluation gate.
• Production Drift Detection - Both systems replace expensive human judgement with continuous automated measurement: drift detection monitors model behavior in production, Agentic Code Reviewer monitors change quality at review time.
• Unsupervised Confidence Estimation - Confidence calibration without ground truth; the evidence-status and severity model here is the review-time analogue of that project's label-free confidence signal.

---

Citation

@software{roy2026agenticcodereviewer,
  author = {Roy, Sourav},
  title  = {Agentic Code Reviewer: Multi-Agent LLM Code Review with Layered Deterministic Verification},
  year   = {2026},
  url    = {https://github.com/royxforge/agentic-code-reviewer}
}

See CITATION.cff for the machine-readable citation metadata.

---

License

MIT - see LICENSE

---

_{Built by Sourav Roy · Artificial Intelligence Engineer · Accure Inc.}