afterstate

afterstate

Deterministic external-state invariant testing for AI agents.

Your agent said "done." What did it actually do to your database, your payments, your inventory? afterstate doesn't grade the agent's answers or its tool-call trajectory — it attacks the systems your agent touches with duplicate requests, client retries, conflicting idempotency keys and duplicate webhooks, then verifies hard state invariants: money moved exactly once, a decline left no trace, a compensation actually restored the balance.

No LLM judge anywhere. Same inputs, same verdict, every run.

[PASS] retry-after-timeout-grants-once
[FAIL] duplicate-request-grants-once
       one_effect_per_key: duplicate effects in credit_ledger: dup-k1×2
       balance_delta: delta is 500 (before=0, after=500), expected 250

Why

Eval tools compare messages and tool-call trajectories. But the failures that cost real money happen one layer down, in the state your agent leaves behind:

• a timeout retry charges the customer twice
• a declined payment still writes a ledger row
• a refund exceeds the original charge
• a webhook is emitted twice and the consumer double-fulfills

Every one of these can look like a 200 OK with a perfectly reasonable transcript. You only catch them by diffing external state under fault injection — which is what afterstate does, in CI, deterministically.

Install

Requires Python 3.11+.

pip install afterstate                # once v0.1.0 lands on PyPI
pip install git+https://github.com/Createyouracccount/afterstate  # until then

The demo service and its contracts live in the repo, not the wheel — clone for the full experience.

Ninety-second demo, one command

The repo ships a deliberately realistic demo: a SaaS credit-issuance service whose buggy mode contains four production-grade bug classes (ignored idempotency keys, declines with side effects, overdrawable balances, double-emitted webhooks).

git clone https://github.com/Createyouracccount/afterstate && cd afterstate
./demo.sh                     # or: docker compose run --rm demo

demo.sh deploys the buggy service, attacks it with 7 money-safety contracts (6 fail, each with the exact duplicated ledger rows or wrong deltas), then deploys the fixed service and shows the same contracts go green. A daily sandbox-loop workflow re-runs this red→green proof in CI and publishes the dated reports — the demo cannot silently rot. See a sample failure report.

Your own system

afterstate init               # scaffolds contracts/starter.yaml
afterstate run contracts/starter.yaml

Point target at a sandbox deployment of your service and its database, list the tables your money flows through, pick faults and invariants. Guides:

• Writing your first contract
• Full contract reference

How it works

A contract is a YAML file describing one scenario:

name: retry-after-timeout-grants-once
target:
  base_url: http://127.0.0.1:8123
  db: sqlite:///credits.db          # or postgresql://...
setup:
  - http: { method: POST, path: /admin/reset }
capture:
  tables: [credit_ledger, balances]
invoke:
  - http:
      method: POST
      path: /credits/grant
      json: { user_id: u1, amount: 100 }
      headers: { Idempotency-Key: retry-k1 }
    faults: [retry_after_failure]    # timeout, then client retry
    expect_status: [200]
assert:
  - one_effect_per_key: { table: credit_ledger, key: idempotency_key }
  - balance_delta:
      query: "SELECT COALESCE(SUM(amount),0) FROM balances WHERE user_id='u1'"
      delta: 100

The runner executes setup → snapshot → invoke (with faults) → snapshot → assert → cleanup and reports per-invariant verdicts.

Fault injection

Request faults go under invoke[].faults:

Webhook delivery faults go under the contract's webhooks.faults block (which also runs the inbox that records what your service emits — see the contract reference):

Built-in invariants

CI

afterstate run exits nonzero on any failure and emits JUnit XML:

# .github/workflows/afterstate.yml
name: afterstate
on: [push]
jobs:
  invariants:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install afterstate
      - run: ./scripts/start-sandbox.sh   # your sandbox SUT
      - run: afterstate run 'contracts/*.yaml' --junit afterstate.xml --report report.md
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: afterstate-report, path: report.md }

Scope (v0.1)

• HTTP systems under test; SQLite and PostgreSQL state capture; webhook inbox.
• afterstate acts as the client and injects faults client-side — which is deterministic and covers the client-visible half of timeouts, resets and redeliveries. A transparent proxy that intercepts a live agent's own traffic is the v0.2 extension point.
• Not an agent framework, not an LLM evaluator, no dashboard. It is a testing engine you point at a sandbox.

License

MIT