agentmeasure

AgentMeasure Healthcheck

A local check-up report for your coding agent runs. Point it at the logs your agent runtime already writes. It answers three questions with evidence — which records are duplicated, where did the agent pay retries for one logical operation, and where did the same tool fail over and over — and writes a terminal summary plus a local HTML report.

Same honesty rules as the conformance pack: every verdict is OK / FINDING / UNPROVABLE, and UNPROVABLE is a first-class result — when the logs cannot decide, that is disclosed, never zeroed.

Status verified 2026-10-15: CLI v0.8.0 is available on PyPI and GitHub; the separately versioned Provider SDK is v0.2.0 on npm. 511 Healthcheck tests passed locally. See release notes.

Install

# Current verified release from PyPI (v0.8.0, verified 2026-10-15).
pip install agentmeasure            # or: pipx install agentmeasure
agentmeasure demo

# Pinned Git install as fallback.
pipx install "git+https://github.com/roy-tong/AgentMeasure@v0.8.0#subdirectory=healthcheck"

# Or run from a repository checkout (use a release tag for reproducibility).
python3 healthcheck/agentmeasure demo

Historical note: the 0.4.0 wheel (2026-09) had a known first-command crash. Since v0.5.0 the release pipeline builds and smoke-tests the packaged wheel in a fresh environment before every push; PyPI is again the recommended path.

Python 3.9+ standard library only; zero runtime dependencies; no network code (a test enforces both).

The 0.7.0 wheel's conformance command is broken because a bundled dependency was omitted. Upgrade to 0.7.2 or later for conformance; Healthcheck itself was unaffected. Installed conformance quickstart.

Installation and error recovery includes Windows and virtual environments. The report path is printed after a successful check. Keep full JSON/HTML local; only share an inspected summary.

Quick start

agentmeasure demo      # synthetic session — see it work, no data needed
agentmeasure check     # your agent logs, last 7 days (Codex or Claude Code)
agentmeasure check --all           # every local session
agentmeasure check --runtime claude   # force the Claude Code adapter
agentmeasure check --dir ~/.codex/sessions/2026/09/05
agentmeasure selftest  # adapters + checks on bundled fixtures
agentmeasure validate export.json  # check an export against its schema
agentmeasure share run.json        # PREVIEW the sanitized summary (writes nothing)
agentmeasure share run.json --out summary.md   # export after review
agentmeasure --version

(Without installing, prefix the commands with python3 healthcheck/agentmeasure.)

Options for check / demo: --html PATH (default agentmeasure-report.html), --json PATH (full machine-readable export), --share PATH (sanitized summary, .md or .json), --days N, --save-snapshot [PATH], --no-history. demo accepts the same output paths and --no-history.

Input filters for check and compare:

--since 2026-09-01 --until 2026-09-05   # date range (overrides --days)
--project myrepo                        # keep sessions whose project matches
                                        # (project = cwd basename, case-insensitive)

Outcome-billing verification: prepare → recount → dispute → recovery

For a hands-on first run, I walk through two reproducible synthetic examples in Check an Intercom Fin bill locally in 10 minutes: a CSV recount and a separate confirmed-versus-assumed settlement statement.

The buyer-side bill check for per-resolution vendors (Intercom Fin, Zendesk AI Agents, …). Rules live in the vendor-rules registry (single source of truth, each rule cited and confidence-graded). Four commands, one pipeline:

agentmeasure prepare --export native-export.csv --vendor zendesk --inspect
agentmeasure prepare --export native-export.csv --vendor zendesk --out prepared.csv
#   native Intercom/Zendesk export -> canonical skeleton. Judgement columns
#   stay EMPTY for human review; native signal columns travel as hint_*
#   cells; prov_* cells record where every value came from. No verdicts.

agentmeasure recount --export prepared.csv --vendor zendesk --inspect
agentmeasure recount --export prepared.csv --vendor zendesk --json recount.json
agentmeasure recount --export prepared.csv --vendor zendesk --contract contract.json
#   Tier 1: the vendor's own published rules, line by line. Findings in BOTH
#   directions; absent evidence is UNPROVABLE, never guessed. --contract adds
#   the buyer's outcome standard (e.g. reopen_window_hours: 72) as a separate
#   outcome lane — renewal leverage, never netted into the claim.

agentmeasure dispute --export prepared.csv --vendor zendesk --contract contract.json
#   the negotiable pack: cover letter citing the vendor's own rule per
#   finding + appendix + (optionally) Tier 2 under the AMS-1 standard.

agentmeasure recovery --pack pack/dispute-pack.json --confirmations confirmations.csv
#   what the vendor actually conceded: credited/paid per disputed line, with
#   confirmation dates and evidence refs. Realized value is counted
#   separately from the claim — an argument won is not cash.

agentmeasure recount --list-vendors shows the registry with each rule's source and confidence (P = primary/official, S = secondary, U = unverified). The browser-local recount at the website runs the same Tier 1 with no upload; a cross-language parity test keeps the two from drifting.

Before a client review, confirm the contract version and outcome subtype. Current public Intercom terms include Procedure outcomes and handoffs. Uncovered terms must remain unresolved rather than be automatically marked unbillable. CRM enrichment also has two reproduced input-boundary issues (missing or aliased columns, and multi-record ordering) awaiting fixes; verify enriched values before using them as client evidence.

The whole story in one artifact

agentmeasure crosscheck --export prepared.csv --vendor zendesk --ledger billing.csv
#   your billing ledger (计费流水) against the export: charges the export
#   doesn't flag, billed flags the ledger doesn't confirm, amounts off the
#   published price. Disagreements are named, never resolved.

agentmeasure verify --export prepared.csv --vendor zendesk --contract contract.json \
    --ledger billing.csv --confirmations confirmations.csv --out vledger
#   the Verified Ledger: Tier 1 + outcome lane + ledger cross-check +
#   realized recovery in ONE document (json + md), each lane labelled
#   claim / leverage / realized and never summed across.

agentmeasure dashboard --pack vledger/verified-ledger.json --out dashboard.html
#   offline single-file HTML: cards, tables, recovery bar. No JS, no
#   network, printable — renders, never recomputes.

agentmeasure rules-diff last-month-registry.json
#   the monthly ritual: diff last period's rules snapshot against the
#   packaged registry. Exit 1 = a material change (trigger, price, reopen
#   rule) — re-verify the period under the new rule and disclose it.

agentmeasure narrative --pack pack/dispute-pack.json --audience vendor \
    --emit-prompt prompt.txt
#   step 1 of two: the drafting prompt, built from the pack. Run YOUR OWN
#   model on it, then:
agentmeasure narrative --pack pack/dispute-pack.json --check-draft draft.txt --out letter.md
#   step 2: every number in the draft is checked against the pack; any
#   invented or rounded number refuses the draft and the rule-based
#   template ships. The model never touches the numbers. No model? The
#   template is the default.

agentmeasure delivery --log delivery.jsonl --log-event \
    --engagement ACME --vendor zendesk --period-index 1 --phase mapping --minutes 45
agentmeasure delivery --log delivery.jsonl
#   appendix-E metrics from logged phases: first look ≤10h, third period
#   ≤1h, repeat-period reuse ≥70%. Use the mapping fingerprint `prepare`
#   prints to track which engagements share the reusable mapping.

Outcome pricing beyond per-resolution support (qualified lead, completed case) is defined — schema first, per the BP — at schemas/outcome-unit.schema.json with worked examples. The outcome engine now ships too: agentmeasure outcomes --units units.jsonl verifies them fail-closed, and the schema covers action and usage-aggregate units — an engineering extension beyond the current support-billing sales scope.

agentmeasure crm-join --export prepared.csv --crm crm.csv --out joined.csv
agentmeasure outcomes --units outcome-units.jsonl --json outcome-ledger.json
agentmeasure period-compare last-month/verified-ledger.json vledger/verified-ledger.json
agentmeasure template --export --vendor intercom --fingerprint <fp> --contract contract.json
agentmeasure certify --pack vledger/verified-ledger.json --digest-out digest.txt
agentmeasure incrementality --events treatment-holdout.csv

Agent commerce: existing engineering reference

Commercial development of this line is paused in the October buyer-review plan. The following documents existing commands, not a current sales offer.

Traditional attribution starts at the click; agent commerce has none. The merchant-side measurement receipt (F2.1-F2.9):

agentmeasure commerce-ledger \
    --events agent-events.jsonl --orders merchant-orders.csv \
    --payments merchant-payments.csv --policy-version 1.0.0
#   metric + value + evidence table; retries reconcile to 1 Operation /
#   2 Attempts / 1 verified order; net GMV is strong; incremental is
#   UNPROVABLE (no holdout, no claim); materiality prints the discrepancy
#   vs the naive dashboard in currency AND percent.

live / replay / synthetic never mix inside one aggregation (error, not a blend); metric names carry their strength prefix and are linted against the evidence that backs them; every input is sha256-anchored. The operating layer on top:

agentmeasure pnl-trees --orders orders.csv --campaigns campaigns.csv   # Organic / Paid, ROAS in Paid only
agentmeasure cm-ledger --orders orders.csv --costs costs.csv --period 2026-09
agentmeasure demand-audit --measurements m.csv --thresholds t.json     # Launch / Watch / Not ready
agentmeasure intent-taxonomy --file clusters.json                      # Operating Cell health
agentmeasure decision-audit --policy policy.json --executions log.jsonl
agentmeasure benchmark-export --rows brand-rows.csv --min-brands 5     # aggregates only, ever

Schema: schemas/commerce-profile.schema.json; canonical retry vector pinned in CI at conformance/vectors/commerce-retry-baseline/.

certify grades the artifact on the AMS-1 adoption ladder (Stage 0..3) and emits the deterministic digest; hand the digest to your own signing tool — this package never runs one. incrementality produces statistics about a difference, never a verdict: the BP's line between rule judgement and causal inference stays visible in the output itself.

Sharing results: preview first, export second

The recommended flow never writes a shareable file until you have read it:

agentmeasure check --json run.json          # the report export embeds the summary
agentmeasure share run.json                 # terminal PREVIEW — nothing is written
agentmeasure share run.json --out summary.md  # export only after you reviewed it

share re-checks the summary against its whitelist before showing or writing anything, so a tampered or future-format export is refused rather than leaked. check --share PATH remains available as the one-step explicit alternative.

Snapshots & compare — did the change help?

The re-run preview: save a snapshot, change something (a prompt, a Skill, a dependency), run the agent again, compare.

python3 healthcheck/agentmeasure check --save-snapshot before.json
# …make your change and run the agent…
python3 healthcheck/agentmeasure check --save-snapshot after.json
python3 healthcheck/agentmeasure compare before.json after.json
# or skip the second snapshot: compare runs a fresh check as side B
python3 healthcheck/agentmeasure compare --all before.json

compare prints metric deltas (failed executions, retry chains, …), check verdict transitions (HC-02 OK → FINDING), and honest caveats: token totals are differenced only when both sides are provable, and version / window / mode mismatches are disclosed because they change what a delta is allowed to mean. Snapshots are local artifacts (they contain project names and session short-ids); the snapshot schema is versioned and validated on load.

Versioned export schemas — how other tools integrate (R8)

Everything a third party needs is a JSON document with a version on it:

Reference schemas live in schemas/. Compatibility policy: within a schema version, fields are only added — never renamed, retyped, or removed; a consumer that meets an unknown newer version must stop and say so. agentmeasure validate FILE… checks any export against its schema, and examples/track-weekly.py is a complete external consumer: it reads snapshots through the schema only, imports nothing from the package, and refuses future schema versions. MCP-style servers can wrap the same exports later; the exports, not internal modules, are the integration surface.

What it checks

Plus a coverage overview: sessions, turns, executions by outcome, token consumption (with subset discipline), compactions, sub-agent activity, corrupt lines, and a per-project breakdown (project = cwd basename, canonical deduplicated counts). Every finding carries evidence — file, line number, exit codes — and a concrete next step.

Honest-number rules (the part most tools get wrong)

• Executions are counted from exactly one stream. Codex Desktop can write a tool call as both a response_item record and an item_completed event. Summing both — or grepping the file — double-counts when both are present. HC-01 discloses the observed ratio instead of hiding it.
• Token subsets are never added into totals. cached_input ⊆ input, reasoning_output ⊆ output in Codex logs. The report shows subsets alongside, never inside, totals.
• Token totals use the last cumulative snapshot per session, not sums of per-event deltas (which double-count after context compaction). The aggregate is marked UNPROVABLE unless every logical session has one valid snapshot; missing or malformed snapshots are reported separately.
• Corrupt or truncated input lowers every count. All numbers are lower bounds where lines failed to parse or a file hit the safety cap; the limitation is reported, never dropped.
• UNPROVABLE, not zero. Sessions whose format lacks outcome events make HC-02/HC-03 UNPROVABLE for that part — reported as such.

Statistics definitions (for support and accounting)

Privacy

• No network. The package contains no network code; a test enforces this by scanning every module for network imports.
• Raw data stays local. Prompts, message text, and reasoning content are never read into the model (only envelope types, ids, exit codes, durations, and token counters).
• Two artifacts, two levels. The HTML report is personal (local paths and evidence metadata — raw commands are represented by hashes, and the banner says so). The --share summary is built from a fixed whitelist of aggregate counts: no prompts, paths, commands, repo names, or session ids can appear in it, and a planted-string test enforces this.
• Run history (~/.agentmeasure/history.jsonl) stays on the machine and is deletable.

Verified runtime support

New cli versions are handled defensively: unknown event types are accounted (unknown type counts), never silently dropped.

Exit codes

Development

python3 -m unittest discover -s healthcheck/tests   # 511 tests verified 2026-10-15
python3 healthcheck/agentmeasure selftest           # fixtures + redaction + compare + schemas
bash healthcheck/scripts/smoke_install.sh           # clean-venv install smoke (packaged artifact)

The install smoke is also a CI job (Python 3.9/3.11): it builds the package, installs it into a fresh venv, and runs selftest, demo, validate, and the external example from the installed artifact — the repository checkout is not used.

Python 3.9+ standard library only. Zero dependencies, by policy: if any step asks you to register, connect to a network, or pay — that is a bug.

Feedback

• Open an issue — include the --json export (it contains local paths; strip them first if the log location is sensitive).
• Email

Identify an input before checking it

agentmeasure inspect-input sample.jsonl --json input-routing.json

This reads every nonempty JSONL row and identifies canonical provider observations, FMT-002, or Codex rollout shapes. Mixed, invalid and unknown inputs exit 2 with next steps. Exit 0 means a format was recognized, not that the schema or measurement passed. The report includes no source rows. Canonical observations use the repository validator and provider onboarding path; do not invent fields to force them into FMT-002.

Compare Claude response snapshots (0.8.0)

agentmeasure response-audit session.jsonl fork.jsonl --json responses.json compares explicit inputs without rewriting logs or claiming invoice totals. See scope and next steps.