0.11.2

Agent Lexicon

A deterministic terminology layer for AI agents
One canonical vocabulary across agents, branches, and tool calls.

Proof · Quickstart · Concepts · How it works

When many agents work a long coding session, each one quietly invents its own names. One branch writes accessToken, another authToken, a third bearer_token — all the same concept. By merge time the service speaks five dialects of itself. Agent Lexicon gives every agent a single canonical vocabulary to read from, resolves the words they actually use back to that canon, and flags terminology that drifted before it lands in main.

It is dependency-free, runs locally, and is deterministic by design: the same input always produces the same output, and every decision carries a reason you can audit.

As a command-line tool, install it with pipx so agent-lexicon and the short alex alias are available in every project:

pipx install agent-lexicon
pipx install "agent-lexicon[completion]"   # with shell tab-completion

To use it as a library inside a project, install it with pip into that project's environment instead:

pip install agent-lexicon

Requires Python 3.10+. Apache 2.0. Zero runtime dependencies.

---

Proof

Benchmark: less terminology drift

Paired synthetic benchmark: the same coding tasks run with and without an agent-lexicon context brief.

Setup: Claude Sonnet 4.6, temperature 0, 10 tasks × 3 repeats × 2 conditions = 60 runs. Scored by an independent regex scorer over raw model output.

This is an early synthetic benchmark, not a universal claim. Harness, tasks, scorer, and raw results: SkeinRank/agent-lexicon-benchmark.

Live output

Real output from the shipped example lexicon (examples/customer_limits/lexicon.yaml). Two terms share the surface word limit — billing.credit_limit and api.rate_limit. This is exactly where an agent drifts and calls the wrong tool.

An ambiguous word stops the agent instead of guessing:

$ agent-lexicon resolve examples/customer_limits/lexicon.yaml "please raise the limit"
Status: ambiguous
Action: ask_clarification
Message: Found 2 possible canonical terms.
Lexicon snapshot: sha256:98b7c5324a20c58926ea8e3413f87851c6d8c354e93197a69c56f1e142ea962e
Candidates:
- api.rate_limit (rate limit) scopes=api matches='limit'
- billing.credit_limit (credit limit) scopes=billing matches='limit'

The same word, scoped, resolves cleanly:

$ agent-lexicon resolve examples/customer_limits/lexicon.yaml "please raise the limit" --scope billing
Status: resolved
Action: use_terms
Message: Resolved to billing.credit_limit.
Lexicon snapshot: sha256:98b7c5324a20c58926ea8e3413f87851c6d8c354e93197a69c56f1e142ea962e
Candidates:
- billing.credit_limit (credit limit) scopes=billing matches='limit'

A wrong tool call is blocked before it runs:

$ agent-lexicon guard examples/customer_limits/lexicon.yaml "raise the credit limit" --tool api.update_rate_limit
Status: blocked
Action: block
Allowed: no
Reason: Requested tool is not allowed for the resolved terminology.
Resolution: resolved
Lexicon snapshot: sha256:98b7c5324a20c58926ea8e3413f87851c6d8c354e93197a69c56f1e142ea962e
Matched terms:
- billing.credit_limit
Allowed tools:
- billing.update_credit_limit

No model was called. No embedding was computed. Run it again and you get the same answer, byte for byte.

---

Why this exists

The longer and wider an agent session runs, the more the shared vocabulary drifts. This is not a hallucination problem — the agents are not inventing facts. They are naming the same concept inconsistently, in branches that never see each other until merge. The result is a codebase where one idea lives under several names, and nobody decided that on purpose.

Existing tools do not close this gap:

• Knowledge graphs model how concepts relate, but require pre-built structure and do not gate a tool call at runtime on raw text.
• LLM or embedding similarity can guess that two names mean the same thing, but the guess is non-deterministic and cannot be reproduced or audited a year later.
• Linters catch inconsistent identifiers in code, but have no notion of a canonical term and do not work on prose, comments, or tool-call text.

Agent Lexicon is a different layer: it takes raw text in, normalizes it, resolves it against a reviewed canonical vocabulary, and returns a structured, deterministic decision. Optional semantics sit on top — as a suggestion to a human, never as the thing that decides.

---

How it works

Three pieces, each doing one job.

Resolve — Given a span of text, find the canonical terms and aliases inside it. Matching uses a dependency-free Aho-Corasick trie, so it is fast and works on prose, comments, and code-style identifiers (accessToken, access_token, ACCESS_TOKEN all resolve to the same term). Input is Unicode-normalized first, so invisible separators, full-width characters, and bidi-control tricks cannot slip a different term past the matcher.

Guard — Given resolved terminology and a tool the agent wants to call, decide whether that call is allowed. Ambiguous terminology returns ask_clarification. A tool that is not permitted for the resolved term returns block. Bidi-control characters in the triggering text are surfaced as a high-risk finding and block by default.

Drift detection at merge — Read the added lines between two git refs and classify every identifier: already known, a likely alias of an existing term, or a genuinely new term that nobody reviewed. The dangerous class — a coined name with no canonical neighbour — is what surfaces by default.

$ agent-lexicon check-merge --root . --base main --head feature-branch --include 'src/**'
Git merge terminology check: 1 files, 6 added lines
Range: main...feature-branch
Lexicon: lexicon/lexicon.yaml
Lexicon snapshot: sha256:98b7c5324a20c58926ea8e3413f87851c6d8c354e93197a69c56f1e142ea962e
Summary: known=2, deprecated=0, likely_alias=0, likely_new_term=3, unresolved_unknown=0, hidden_unresolved=1
Known terminology:
- auth.py:2 'authToken' -> auth.access_token (access token) scopes=auth
New terminology candidates:
- auth.py:3 'credentialBlob' unknown; possible new term
- auth.py:4 'sessionKey' unknown; possible new term
- auth.py:5 'quuxHandle' unknown; possible new term
Hidden unresolved identifiers: 1. Use --include-unresolved-unknowns to inspect low-signal identifiers.

Add --fail-on-review to make this a blocking CI check. It returns a non-zero exit code for deprecated terminology and unreviewed drift.

---

Three ways to use it

Command line — the full local loop, no code required. Every command is also available under the short alias alex, so alex resolve … works the same as agent-lexicon resolve ….

agent-lexicon init                      # create lexicon/, workspace, policy, and scan config
agent-lexicon scan                      # discover candidate terms from configured paths
agent-lexicon scan README.md docs src   # or override paths explicitly
agent-lexicon review                    # open the local web inbox to accept/reject

agent-lexicon publish --update-lexicon  # publish accepted decisions and update lexicon.yaml
agent-lexicon resolve <lexicon> "text"  # resolve terminology in any text
agent-lexicon guard   <lexicon> "text" --tool <name>   # gate a tool call
agent-lexicon context <lexicon>         # print the canonical vocabulary brief for an agent
agent-lexicon lint-diff                 # lint the working-tree diff for terminology drift
agent-lexicon check-merge --base main --head <branch>  # detect drift at merge
agent-lexicon check-merge --base main --head <branch> --semantic-check  # CI-style pass/fail

In an agent workflow

Two of these commands are built for wrapping an AI coding agent:

Before a task — hand the agent the project's canonical vocabulary so it starts with the right language:

agent-lexicon context lexicon/lexicon.yaml
# Use these canonical terms:
# - ContextSpace
# - RuntimeSnapshot
#
# Avoid:
# - WorkspaceScope (use "ContextSpace" instead)

During a task — check a working diff for terminology drift before it is committed, with layered severity:

git diff | agent-lexicon lint-diff
# Terminology lint: 2 files, 18 added lines
#
# Deprecated terms (fail):
# - src/session.py:14 WorkspaceScope -> use "ContextSpace"
#
# Possible typos / near-misses (warn):
# - docs/api.md:7 ContextSapce -> did you mean "ContextSpace"?
#
# New project terms (info):
# - src/memory.py:22 TaskMemoryProfile
# (exit code 1: a deprecated term was used)

Level 1 (declared deprecated terms) fails the check. Level 2 (lexical near-misses) warns, or fails under --strict. Level 3 (unknown project terms) is reported for awareness only. An optional --semantic flag adds probabilistic suggestions but never changes the exit code — enforcement stays deterministic.

At merge / PR — a deterministic terminology gate alongside your other CI checks:

agent-lexicon check-merge --base main --head HEAD --semantic-check
# Terminology check: 3 files, 42 added lines
# Semantic conflicts detected (1):
# - customer cap vs credit limit (use "credit limit")
# (exit code 1, so CI fails)

Both are deterministic: they flag terms the lexicon already declares (deprecated aliases and near-misses to canonical terms), never guesses. lint-diff detects a piped unified diff automatically; --stdin remains available when an explicit input mode is preferred.

Python library — call the same logic inline.

from agent_lexicon import load_lexicon, resolve_text, guard_tool_call

lexicon = load_lexicon("lexicon/lexicon.yaml")

decision = resolve_text(lexicon, "please raise the limit", scopes=["billing"])
print(decision.status)        # ResolutionStatus.RESOLVED
print(decision.action)        # ResolutionAction.USE_TERMS

guard = guard_tool_call(
    lexicon,
    "raise the credit limit",
    tool_name="api.update_rate_limit",
)
print(guard.status)           # ToolGuardStatus.BLOCKED

MCP server — expose the lexicon to any MCP-compatible agent over stdio.

agent-lexicon mcp serve --root . --lexicon lexicon/lexicon.yaml

The server exposes six tools: resolve_term, check_language, guard_tool_call, find_evidence, submit_proposal, and get_snapshot. List their full definitions with agent-lexicon mcp tools.

Repository scan config

agent-lexicon init creates .agent-lexicon/config.yaml so common repository scans do not need long CLI commands. By default, agent-lexicon scan starts from documentation and common source roots, applies language-aware include globs for popular stacks, and respects the repository .gitignore.

scan:
  paths:
    - README.md
    - docs
    - src
    - app
    - packages
    - lib
    - services
  include:
    - "docs/**/*.md"
    - "docs/**/*.txt"
    - "**/*.py"
    - "**/*.ts"
    - "**/*.tsx"
    - "**/*.go"
    - "**/*.rs"
    - "**/*.java"
    - "**/*.kt"
    - "**/*.cs"
    - "**/*.sql"
    - "**/*.yaml"
  exclude:
    - ".venv/**"
    - "node_modules/**"
    - "dist/**"
    - "**/generated/**"
  respect_gitignore: true
  max_file_bytes: 1000000

.gitignore is treated as the first line of repository-specific ignore behavior. Use scan.exclude for Agent Lexicon-specific rules such as generated fixtures that are still tracked in Git.

CLI flags still win when you need a one-off run:

agent-lexicon scan docs src --include "src/**/*.py" --exclude "src/generated/**"
agent-lexicon scan --no-gitignore
agent-lexicon check-merge --base main --head HEAD --exclude "docs/generated/**"

---

GitHub Actions workflow

The repository includes a terminology review workflow for pull requests. It validates the tracked lexicon and runs merge-time drift detection against the PR diff:

name: Agent Lexicon Terminology Review

on:
  pull_request:
    branches: [main]

jobs:
  terminology-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install --upgrade poetry==2.1.1
      - run: poetry install --with dev
      - run: poetry run agent-lexicon validate lexicon/lexicon.yaml --lint --strict-lint
      - run: poetry run agent-lexicon check-merge --root . --base origin/${{ github.base_ref }} --head HEAD

The checked-in workflow is review-first by default: it prints terminology drift without blocking every PR. Set base_ref, head_ref, and fail_on_review=true for an on-demand blocking run, or add --fail-on-review when deprecated terminology and unreviewed drift should become required merge checks.

---

The dictionary is code

The canonical vocabulary lives in a git-tracked YAML file. A term has a canonical form, aliases, the scopes it belongs to, and optionally the tools that are allowed to act on it.

version: 1
scopes:
  - id: billing
    label: Billing
  - id: api
    label: API
terms:
  - id: billing.credit_limit
    canonical: credit limit
    scopes: [billing]
    tools: [billing.update_credit_limit]
    aliases:
      - surface: customer cap
      - surface: account limit
  - id: api.rate_limit
    canonical: rate limit
    scopes: [api]
    tools: [api.update_rate_limit]
    aliases:
      - surface: requests per minute

Because it is just a file in the repo, the vocabulary versions, diffs, and reviews the same way your code does. Runtime decisions also carry a content-addressed snapshot reference (sha256:<digest>), so the same text can be replayed later against the exact same vocabulary content. A built-in linter warns when a surface is broad enough to over-trigger or to affect a guard decision:

$ agent-lexicon lint lexicon/lexicon.yaml
Lexicon lint: warnings (1 warning)
[warning] tool_broad_surface: tool-routed term uses a broad surface that can
  affect guard decisions (term=data.primary_key; surface='PK'). Hint: Use
  explicit tool-facing aliases and avoid bare words on terms with tools.

Review and publish decisions are kept as workspace provenance records and can be exported as JSONL for audit or handoff:

$ agent-lexicon workspace export-decision-log --root . --action review_decision_saved
{"action":"review_decision_saved","actor":"local","rule_id":"human_review",...}

---

Optional semantics, kept honest

When the deterministic heuristics are confident, they decide alone. When a new identifier lands in a gray zone — close to an existing term but not a clear match — an optional semantic reranker can suggest the most likely canonical neighbour, so a reviewer sees "authToken might be your access token" instead of an unsorted pile of unknowns.

pip install "agent-lexicon[oov]"        # tokenizer-backed out-of-vocabulary scoring
pip install "agent-lexicon[semantic]"   # semantic near-miss reranking

This is deliberately a suggestion to a human, marked as non-deterministic, never an autonomous decision. The semantic layer never commits a term on its own. The thing that decides stays deterministic and auditable; the thing that suggests is allowed to be smart. That boundary is the point — it is what keeps every committed decision reproducible.

---

Design guarantees

These hold on the deterministic runtime and local review paths:

• Deterministic. The same text against the same immutable lexicon snapshot always produces the same decision. No model, no embedding, no randomness on the resolve and guard paths.
• Reproducible. Runtime and merge reports include a content-addressed lexicon_snapshot_ref (sha256:<digest>), so a decision can be replayed later against the exact same vocabulary content.
• Auditable. Every runtime decision reports its reason — which surface matched, at which span, in which scope, and why a tool was allowed or blocked. Local review and publish decisions are also written to an append-only provenance log with actor, action, rule, result, and lexicon snapshot metadata.
• Dependency-free core. The resolver and matcher have zero runtime dependencies and run entirely in memory. Optional extras are opt-in and never touch the hot path.
• Safe by construction. Local writes are atomic (a reader sees a complete file or none), and the workspace database is configured for concurrent access without torn reads.
• Storage boundary. The local workspace is SQLite-backed by default, but workflow code depends on a small WorkspaceStore boundary so future shared storage can be added without changing the deterministic runtime.

---

Documentation

• Quickstart — local setup, scan, review, publish, and runtime usage.
• Concepts — terms, aliases, scopes, resolution, guard decisions, and merge-time drift detection.
• Python API reference — load_lexicon, resolve_text, guard_tool_call, and the decision and enum types they return.
• MCP server reference — the six MCP tools, their arguments, and their return values.

Contributing, security, and community:

• Contributing — development setup and how to send a change.
• Security policy — how to report a vulnerability privately.
• Code of conduct — community standards.

---

Status

Agent Lexicon is an early, actively developed project (0.7.x). The core — resolve, guard, near-miss, dictionary-as-code, and merge-time drift detection — is well tested (326 passing tests) and used through the CLI, the Python API, and the local MCP server. Scaling it across many processes or a networked deployment is on the roadmap, not yet proven in production.

If terminology consistency across long, multi-agent sessions is a real cost for you — especially in regulated domains where decisions must be reproducible and auditable — this is built for exactly that.

---

License

Apache 2.0. Free for commercial use.