0.2.2

AdminLineageAI

AdminLineageAI makes crosswalks between administrative locations such as districts (ADM2), subdistricts (ADM3), states (ADM1), and countries (ADM0) across datasets that may come from completely different sources and different periods. It uses AI to compare likely matches, reason over spelling variants and language-specific forms, administrative split/merges/renames and produce a usable crosswalk plus review artifacts.

Matching administrative units by hand is labour-intensive work. Through this package, we hope to reduce the manual work of matching administrative units between datasets while still keeping a clear review trail and reproducibility.

The package generates candidate matches between two datasets, asks Gemini to choose among them, and writes a crosswalk plus review artifacts. It outputs a final evolution key plus review files as CSV and Parquet.

Published package: adminlineage on PyPI

Possible use cases

Below are few possible scenarios where this package can be of assistance. Moreover, we would love to hear about other user experiences and use cases for this package.

• For instance, one has scheme dataset from a government scheme and need to match it against a standard administrative list such as a census table. The scheme source may write Paschimi Singhbhum while another uses West Singhbhum. Plain fuzzy matching will miss cases like this unless you manually standardize prefixes and suffixes first. While AI can do matching for this because it has context that paschim in Hindi means west. The same kind of issue shows up across many widely spoken languages.
• Handling administrative churn. Districts and other units are regularly split, merged, renamed, or grouped differently, and there is often no up-to-date public evolution list for newly created units, the package does a wide google search and find possible predessor or sucessor for each administrative unit in the primary dataset
• Creating entirely new evolution crosswalks that do not exist between two time period at an administrative level.

Important Features

• The default setting of the package is set to have best results with minimal token cost. Please feel free to change them according to your needs.
• To keep the token costs minimal, we do exact string match plus pruning of matching candidates on the primary side before first stage.
• Hierarchical matching with exact_match. If your data are nested, you can match names within exact scopes such as country, state, or district. For example, you can choose to match only district names within a states or subdistricts with a district. This works well, but the exact-match column string need to line up exactly across both datasets.
• Replay and reproducibility. Academic pipelines often need to be rerun many times. With replay enabled, repeated semantic requests can reuse prior completed LLM work instead of calling the API again. The seed parameter helps keep request identity deterministic and makes reruns easier to reproduce.

The supported live workflow in AdminLineageAI is:

• Compatible with any gemini-3+ model
• Google Search grounding enabled
• strict JSON output from the model
• user-controlled batching with automatic split fallback on failed multi-row requests
• an optional bounded second-stage rescue pass for unmatched rows when string_exact_match_prune is set to from or to

The bounded second stage works like this:

• first pass still does the normal grounded shortlist adjudication
• if string_exact_match_prune="from", the rescue pass revisits rows with merge="only_in_from"
• if string_exact_match_prune="to", it revisits rows with merge="only_in_to"
• it runs one grounded research call to look for a predecessor or successor name
• if that research comes back as unknown with no lineage hint, the row is left alone and the rescue pass stops there
• otherwise it searches the full opposite table, rebuilds a short global shortlist, and runs one final strict JSON decision call without additional search grounding
• the second stage is sequential, one-pass, resumable, and writes second_stage_results.jsonl

How To Use

You do not need the CLI to use AdminLineageAI. The simplest path is the Python API.

1. Install the published package.

pip install adminlineage

Install the optional parquet dependency if you want parquet output support:

pip install "adminlineage[io]"

2. Set a Gemini API key in GEMINI_API_KEY, or use another environment variable name and pass it explicitly.

GEMINI_API_KEY=your_api_key_here

The package can load a nearby .env file when it looks for the key.

3. Choose the name column on each side, and add optional exact-match columns, IDs, or extra context columns if you have them.

4. Run the matcher.

import pandas as pd
import adminlineage

df_from = pd.read_csv("from_units.csv")
df_to = pd.read_csv("to_units.csv")

crosswalk_df, metadata = adminlineage.build_evolution_key(
    df_from,
    df_to,
    country="India",
    year_from=1951,
    year_to=2001,
    map_col_from="district",
    map_col_to="district",
    exact_match=["state"],
    id_col_from="unit_id",
    id_col_to="unit_id",
    relationship="auto",
    string_exact_match_prune="from",
    evidence=False,
    reason=False,
    model="gemini-3.1-flash-lite-preview",
    gemini_api_key_env="GEMINI_API_KEY",
    replay_enabled=True,
    seed=42,
)

print(crosswalk_df[["from_name", "to_name", "merge", "score"]].head())
print(metadata["artifacts"])

6. Review the outputs. By default, AdminLineageAI writes artifacts under outputs/<country>_<year_from>_<year_to>_<map_col_from>. The main ones are evolution_key.csv, review_queue.csv, and run_metadata.json.

Common Options

• exact_match: Restricts matching to rows that agree exactly on one or more scope columns such as country, state, or district.
• string_exact_match_prune: Controls how aggressively exact string hits are removed from later AI work. Use this to control token spend.
• relationship: Declares the kind of relationship you expect, or leave it as auto.
• max_candidates: Limits how many candidate rows are shown to the model for each source row. The default is 6.
• evidence: Adds a short factual summary column.
• reason: Adds a longer explanation column.
• replay_enabled: Reuses prior completed LLM work when the semantic request matches.
• seed: Keeps request identity deterministic for more reproducible reruns.
• output_dir: Changes where run artifacts are written.

Matching Flow Example

This example follows a nested district-level match inside India > Uttar Pradesh from 2011 to 2025. Here string_exact_match_prune='to' (this set to as primary side and from as secondary side where all candidates stay global).

flowchart TD
    A["From table (2011)<br/>India / Uttar Pradesh / Agra<br/>India / Uttar Pradesh / Kanpur Dehat<br/>India / Uttar Pradesh / Faizabad<br/>India / Uttar Pradesh / Allahabad"]
    B["To table (2025)<br/>India / Uttar Pradesh / Agra<br/>India / Uttar Pradesh / Kanpur Rural<br/>India / Uttar Pradesh / Ayodhya<br/>India / Uttar Pradesh / Prayagraj"]
    C["Nested settings<br/>map_col='district'<br/>exact_match=['state']<br/>string_exact_match_prune='to'<br/>this set 'to' as primary side<br/>and 'from' as secondary side<br/>where all candidates stay global"]
    D["Validate inputs and normalize names"]
    E["Exact string match pruning before LLM"]
    F["Agra -> Agra<br/>no LLM used here<br/>just exact string match"]
    H["AI matches remaining rows on primary side<br/>(Kanpur Rural, Ayodhya, Prayagraj)<br/>using grounded Gemini search<br/>"]
    I["AI matches Kanpur Dehat -> Kanpur Rural<br/>because it has context that 'dehat' means 'rural' in Hindi"]
    J{"Do Ayodhya or Prayagraj stay unmatched<br/>after first stage?"}
    L["Do intensive Gemini search of potential predecessor / successor of Ayodhya / Prayagraj<br/>if they were renamed, merged, split, or transferred"]
    M["If Gemini finds a potential predecessor / successor for that district<br/>match it with the global district list from the secondary side"]
    N["Write final evolution key<br/>Agra -> Agra<br/>Kanpur Dehat -> Kanpur Rural<br/>Faizabad -> Ayodhya<br/>Allahabad -> Prayagraj"]
    O["Write artifacts<br/>evolution_key.csv<br/>review_queue.csv<br/>run_metadata.json<br/>replay bundle"]

    subgraph G["First stage"]
        H
        I
    end

    subgraph P["Second stage"]
        L
        M
    end

    A --> C
    B --> C
    C --> D
    D --> E
    E --> F
    E --> H
    H --> I
    I --> J
    J -- "No" --> N
    J -- "Yes" --> L
    L --> M
    A --> N
    B --> N
    M --> N
    N --> O

Hand Check Against Scheme Ground Truth

This is a quick hand check against a human-made evolution key for a government scheme implemented nationally in India. The scheme side is 2025 districts, mapped back to their predecessor 2011 districts.

The comparison is oriented from the scheme side: for each district_2025 in the hand key, does the evolution key recover the expected district_2011 predecessor? Names were normalized before comparison. Spelling and transliteration-only differences were treated as aligns. A row counts as a match only when the evolution key has a non-blank from_name.

• aligns means the evolution key points to the same 2011 district name
• disagrees means the evolution key points to a different 2011 district
• no match means the evolution key does not provide any non-blank from_name

Takeaway: most scheme districts map back to the same 2011 predecessor as the hand key, a few disagree, and a small number have no match. Treat this as a sanity check, not a full audit.

Optional CLI Workflow

The CLI is useful when you want a saved YAML config for repeatable runs, but it is optional.

adminlineage preview --config examples/config/example.yml
adminlineage validate --config examples/config/example.yml
adminlineage run --config examples/config/example.yml
adminlineage export --input outputs/india_1951_2001_subdistrict/evolution_key.csv --format jsonl

The package includes these example assets:

• examples/config/example.yml
• examples/loaders/sample_loader.py
• examples/adminlineage_gemini_3_1_flash_lite.ipynb

Python API

Public objects available from import adminlineage:

• build_evolution_key
• preview_plan
• validate_inputs
• export_crosswalk
• get_output_schema_definition
• OUTPUT_SCHEMA_VERSION
• __version__

build_evolution_key

Build the evolution key and write run artifacts.

Required arguments:

Optional arguments:

Return value:

• tuple[pd.DataFrame, dict]
• first item: the crosswalk DataFrame
• second item: run metadata with counts, warnings, request details, and artifact paths

preview_plan

Preview grouping and candidate-generation behavior without calling Gemini.

adminlineage.preview_plan(
    df_from,
    df_to,
    *,
    country,
    year_from,
    year_to,
    map_col_from,
    map_col_to=None,
    exact_match=None,
    id_col_from=None,
    id_col_to=None,
    extra_context_cols=None,
    string_exact_match_prune="none",
    max_candidates=6,
)

Return value: a diagnostics dict describing validity, group sizes, exact-string hits, and candidate budgets.

validate_inputs

Validate the two input tables without running the pipeline.

adminlineage.validate_inputs(
    df_from,
    df_to,
    *,
    country,
    map_col_from,
    map_col_to=None,
    exact_match=None,
    id_col_from=None,
    id_col_to=None,
)

Return value: a diagnostics dict that reports whether the inputs are valid and what is missing or duplicated.

export_crosswalk

Convert a materialized crosswalk file into another format.

adminlineage.export_crosswalk(
    input_path="outputs/india_1951_2001_subdistrict/evolution_key.csv",
    output_format="jsonl",
    output_path=None,
)

Return value: the written output path.

Supported output formats:

• csv
• parquet
• jsonl

get_output_schema_definition

Return a machine-readable description of the materialized output schema.

schema = adminlineage.get_output_schema_definition(include_evidence=False)

Arguments:

Return value: a dict containing the schema version, ordered output columns, required columns, and enum values, including the merge indicator enum.

OUTPUT_SCHEMA_VERSION

String constant for the current materialized output schema version.

__version__

String constant for the package version.

Optional CLI Reference

Commands:

adminlineage run --config path/to/config.yml
adminlineage preview --config path/to/config.yml
adminlineage validate --config path/to/config.yml
adminlineage export --input path/to/evolution_key.csv --format {csv|parquet|jsonl} [--output path]

preview and validate do not call Gemini. run writes the full artifact set. export converts an existing materialized crosswalk file. If you are using the Python API directly, you can ignore this section.

CLI YAML Config Reference

Top-level sections:

• request
• data
• llm
• pipeline
• cache
• retry
• replay
• output

request

data

Loader contract for python_hook mode:

def load_data(config: dict) -> tuple[pd.DataFrame, pd.DataFrame]:
    ...

The included example hook is examples/loaders/sample_loader.py.

For file mode, data.from_path and data.to_path are resolved relative to the config file location, not your shell location.

llm

pipeline

cache

retry

replay

Relative replay store paths are resolved from the config file location. This section only matters if you are using the CLI workflow.

output

Minimal config shape:

request:
  country: India
  year_from: 1951
  year_to: 2001
  map_col_from: subdistrict
  map_col_to: subdistrict
  exact_match: [state, district]
  id_col_from: unit_id
  id_col_to: unit_id
  relationship: auto
  string_exact_match_prune: none
  evidence: false
  reason: false

data:
  mode: files
  from_path: ../data/from_units.csv
  to_path: ../data/to_units.csv

llm:
  provider: gemini
  model: gemini-3.1-flash-lite-preview
  gemini_api_key_env: GEMINI_API_KEY
  temperature: 0.75
  seed: 42
  enable_google_search: true
  request_timeout_seconds: 90

pipeline:
  batch_size: 5
  max_candidates: 6
  review_score_threshold: 0.6

cache:
  enabled: true
  backend: sqlite
  path: llm_cache.sqlite

retry:
  max_attempts: 6
  base_delay_seconds: 1.0
  max_delay_seconds: 20.0
  jitter_seconds: 0.2

replay:
  enabled: false
  store_dir: .adminlineage_replay

output:
  write_csv: true
  write_parquet: true

Outputs And Utilities

Main Artifacts

Crosswalk Columns

review_queue.csv is a filtered subset of the crosswalk for rows that were flagged for manual review. Target-only rows remain in the final evolution key with merge="only_in_to".

Operational Notes

• exact_match scopes the candidate search. If you set exact_match=["state", "district"], a row only compares against rows from the same (state, district) group. This is the main hierarchical matching mechanism in the package.
• Candidate generation happens before Gemini. max_candidates controls how many shortlist entries the model sees for each source row. The default is 6, but you can still raise it explicitly.
• Exact string handling happens before the model call. string_exact_match_prune controls whether already matched rows remain in later AI work.
• Live Gemini work is grounded with Google Search and returns strict JSON. The pipeline then materializes CSV and Parquet outputs itself.
• When string_exact_match_prune is from or to, the package can run one bounded second-stage rescue pass on unmatched primary-side rows. That pass does one grounded research call, and only does a second shortlist decision call if the research returned a usable lineage_hint.
• Replay is opt-in. When replay_enabled=True, rerunning the same semantic request reuses the prior completed LLM output instead of calling Gemini again.
• seed helps keep request identity deterministic and makes runs easier to reproduce.
• Cache is configured in CLI config. When enabled, the package uses a SQLite cache at cache.path.
• Retry behavior is configurable in CLI config. Transient Gemini failures are retried according to the retry section before a row is marked unresolved.
• export_crosswalk and adminlineage export convert an existing materialized crosswalk into csv, parquet, or jsonl.

A Few Practical Defaults

• model="gemini-3.1-flash-lite-preview"
• temperature=0.75
• enable_google_search=True
• evidence=False
• reason=False
• relationship="auto"
• string_exact_match_prune="none"

Those are the current defaults. Change them when you need replay, evidence, stricter scoping, or different review thresholds.

Reporting Issues

If you run into a bug, a broken match, or a confusing output, please open an issue on GitHub.

The most useful issue reports include:

• the package version, for example adminlineage.__version__
• whether you used the Python API, CLI, or one of the notebooks
• the model name and the main matching settings, especially exact_match, string_exact_match_prune, batch_size, max_candidates, and enable_google_search
• whether the run was fresh, resumed from an existing output directory, or reused replay artifacts
• a small sanitized input example that reproduces the problem
• the relevant rows from evolution_key.csv or review_queue.csv
• run_metadata.json, and when relevant links_raw.jsonl, grounding_notes.jsonl, and second_stage_results.jsonl
• the traceback or log excerpt if the run failed

Citation

If you use AdminLineageAI in published work, please cite the package and briefly report the workflow you used.

Suggested software citation:

Siddiqui, T. I., and Vetharenian Hari. (2026). AdminLineageAI (Version 0.2.1) [Python package]. https://pypi.org/project/adminlineage/

If the workflow matters for interpretation, report the key settings in your methods or appendix:

• country and time span
• administrative level and exact-match scope
• string_exact_match_prune mode
• Gemini model name
• whether Google Search grounding was enabled
• whether the bounded second-stage rescue pass was active
• whether outputs were manually reviewed or corrected