ai-data-platform

Ai-data-platform

Why it matters

Data is the prerequisite for every analytics dashboard, ML model, pipeline test, and stakeholder demo — yet most teams spend weeks waiting for it. Production exports require compliance approval. Hand-crafted test files miss relationships and break at scale. POCs stall while access is negotiated.

ai-data-platform is a local-first tool that turns a schema declaration or a small sample into FK-safe, realistic synthetic data in minutes — with automated quality scoring, semantic models, SQL analytics, and AI-driven workflows. No cloud account. No PII exposure. Works in retail, healthcare, finance, and any other domain without custom code.

Acceleration timeline

Works in any domain — zero customization

There is no if domain == "healthcare" in the codebase. Retail orders, patient admissions, bank transactions, factory work orders, and telecom CDRs are all just tables, keys, categories, amounts, and dates.

---

End-to-end — what you get

One project, one flow, five tangible deliverables:

  YOUR INPUT                    ADP PIPELINE                      YOUR OUTPUT
  ──────────                    ────────────                      ───────────

  spec.yaml          ──▶  apply-spec  ──▶  catalog + profiles
  or CSV/DB sample   ──▶  scan/profile ──▶  learned distributions
                              │
                              ▼
                         generate-data
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
         output/         quality.md      semantic model
    (parquet/csv/      (score +          (Cube.js YAML
     duckdb files)      evidence)         for BI tools)
              │
              ▼
         explore / ui / sql
    (query, dashboard KPIs,
     natural-language analytics)

A complete example: A product manager describes a retail dataset → agent applies a spec → 50k rows generated → quality score 100/100 → revenue-by-city query returns realistic KPIs → stakeholder demo ready in one session. No production data touched.

Full walkthrough: docs/GETTING-STARTED.md

---

Who it's for

Not sure where to start?

---

How it works

Three ways in. Same pipeline. Same validated output.

flowchart TD
  subgraph paths [Choose your path]
    Q1{Have sample data?}
    Q1 -->|No| PathA["Path A: spec.yaml\napply-spec → generate"]
    Q1 -->|Yes| PathB["Path B: learn from data\nconnect → scan → profile → generate"]
    Q1 -->|AI IDE| PathC["Path C: MCP\nCursor / Claude / Windsurf"]
  end

  PathA --> Pipeline
  PathB --> Pipeline
  PathC --> Pipeline

  subgraph Pipeline [Generation pipeline]
    Gen[FK-safe generator]
    QC[quality-check]
    Gen --> QC
  end

  QC --> Out[output/ files]

  Out --> Next[semantic-model · explore sql · ui · docs]

Five steps, plain language:

Full internals: docs/USER-FLOW.md · Architecture: docs/ARCHITECTURE.md

---

Get started

Install

pip install 'ai-data-platform[all]'      # everything (recommended)
pip install 'ai-data-platform[mcp]'       # MCP only (add to existing install)
pip install 'ai-data-platform[postgres]'  # PostgreSQL connector

Requires Python 3.11+.

Install via Homebrew (macOS)

brew tap yogi776/tap
brew install ai-data-platform

After installation adp is available everywhere in your terminal.

adp version        # verify
adp --help         # see all commands
adp mcp-server --help

Upgrade: brew upgrade ai-data-platform
Remove: brew uninstall ai-data-platform
Verify: which adp && adp version

The formula installs all optional extras ([all]), including MCP server support. Requires macOS (Apple Silicon or Intel) with Python 3.11+ via Homebrew's python@3.12. For full MCP client setup (Cursor, Claude Desktop, etc.), see the MCP Guide.

Troubleshooting: If adp is not found after install, start a new terminal session or run brew link --overwrite ai-data-platform. If the issue persists, add Homebrew's bin to your PATH:

echo 'export PATH="$(brew --prefix)/bin:$PATH"' >> ~/.zshrc && source ~/.zshrc

Path A — no data needed (~5 min)

You have a design doc or schema in mind. No sample files required.

mkdir demo && cd demo
adp init --name demo
adp apply-spec path/to/spec.yaml       # see healthcare-claims (GitHub) or benchmarks/fixtures/
adp generate-data --rows 50000 --output parquet
adp quality-check --report quality.md
# Optional: push to Snowflake / BigQuery / Postgres (pip install 'ai-data-platform[load]')
# adp load --destination snowflake_dev

You get: output/*.parquet + quality.md with score 100/100. See docs/LOAD.md for warehouse export.

Path B — learn from sample data (~10 min)

You have CSV, Parquet, DuckDB, PostgreSQL, or MySQL samples.

mkdir demo && cd demo
adp init --name demo
adp connect --name my-db --type csv --path ./data
adp scan && adp profile
adp generate-data --rows 50000 --output parquet
adp quality-check

You get: Data that mirrors your sample's distributions, scaled to 50k rows.

Path C — drive from Cursor / Claude (~3 min)

pip install 'ai-data-platform[mcp]'
cd my-project && adp init && adp setup-agent

MCP works with any client (Claude, Cursor, Windsurf, VS Code). Cursor users also get auto-installed agent skills via adp init.

Add .cursor/mcp.json next to your adp.yaml (or use adp init to auto-write):

{"mcpServers": {"adp": {"command": "adp", "args": ["mcp-server"]}}}

Open the project folder as your workspace, reload MCP, then ask:

"Apply the healthcare spec and generate 10k rows. Run a quality check and show me the first 10 rows."

You get: Agent reports quality score, shows sample rows, files land in output/.

Runnable walkthrough with expected output at every step: docs/GETTING-STARTED.md

All 16 use cases: docs/USE-CASES.md

---

Key capabilities

One backend (ADPClient), four interfaces: CLI · SDK · Web UI · MCP. Same logic everywhere.

from ai_data_platform import ADPClient

client = ADPClient(".")
client.apply_spec("spec.yaml")
result = client.generate_data(rows=50_000, output_format="parquet")
print(client.quality_check()["quality_score"])   # → 100.0

---

Examples

One walkthrough project — retail-ecommerce: sales performance analysis with 3 years of order history, generate parquet, adp load to Snowflake, and ready-made KPI SQL.

For spec-only generation (no sample data), see healthcare-claims or benchmarks/fixtures/seasonal-retail-spec.yaml.

---

Configuration

Data sources (adp.yaml):

sources:
  - name: csv_files
    type: csv
    path: ./data
  - name: postgres_prod
    type: postgres
    dsn: "postgresql+psycopg://user:${PGPASSWORD}@host:5432/shop"
    schema: public

Secrets use ${ENV_VAR} interpolation — plaintext passwords are rejected.

AI provider (for NL-to-SQL and propose_spec):

model_provider:
  provider: minimax          # minimax | openai | anthropic | gemini | local
  api_key_env: MINIMAX_API_KEY

provider: local runs fully offline without LLM calls.

Declarative spec (spec.yaml) — see docs/SPEC-REFERENCE.md:

version: 1
tables:
  - name: dim_customer
    columns:
      - {name: customer_id, type: uuid, primary_key: true}
      - {name: gender, type: string, values: {Male: 48, Female: 50, Other: 2}}
      - {name: age, type: int, min: 18, max: 85}
      - {name: signup_date, type: date, start: 2020-01-01, end: 2026-01-01}

---

Documentation

---

Benchmarks

Performance benchmarks are in benchmarks/bench_generation.py. Run them with Python 3.11+:

# Smoke test — 100K rows
python benchmarks/bench_generation.py --rows 100000

# Full benchmark — 10M rows per table (3 tables, 30M total rows)
python benchmarks/bench_generation.py \
  --rows 10000000 \
  --rows-per-table "fact_orders=10000000,fact_payments=10000000,fact_shipments=10000000"

Benchmark: Seasonal-Retail Spec · 10M rows/table · Parquet · 1 worker

Generated July 2026 on an 8-core machine.

Per-table breakdown:

Bottleneck identified

The execution planner's complexity analyzer correctly flags GIL-bound Python string samplers as the thread-scaling ceiling:

⚠ fact_orders: 2 per-row Python string sampler(s) (order_id, order_ts_season)
  are GIL-bound and cap thread scaling at scale
  (Phase 3: vectorize).

uuid and seasonal_date samplers are the affected types — they construct strings per-row in Python. Vectorizing these (Phase 3 roadmap) is the single highest-leverage optimization for multi-worker throughput.

Memory model accuracy

The static memory estimator (heuristic) predicted 3,020 MB while actual peak RSS was 2,004 MB — a −34% overestimate. The OVERHEAD=2.0 factor in memory_estimator.py is conservative; the engine's chunked execution is more memory-efficient than modeled. The estimator correctly flagged the large-table risk but the real system handles it better than the static bound.

Scaling ladder (observed)

Throughput scales from 121K → 171K rows/s as the system warms up (JIT, OS caches), confirming vectorized numeric generation is not the bottleneck at this scale.

Output format comparison

Seasonal-retail spec · 5M rows/table · 15M total rows · 1 worker · July 2026.

# Compare formats at 5M rows/table
python benchmarks/bench_generation.py \
  --rows 5000000 \
  --format csv \
  --rows-per-table "fact_orders=5000000,fact_payments=5000000,fact_shipments=5000000"

python benchmarks/bench_generation.py \
  --rows 5000000 \
  --format parquet \
  --rows-per-table "fact_orders=5000000,fact_payments=5000000,fact_shipments=5000000"

python benchmarks/bench_generation.py \
  --rows 5000000 \
  --format duckdb \
  --rows-per-table "fact_orders=5000000,fact_payments=5000000,fact_shipments=5000000"

Generation benchmark (5M rows/table)

* DuckDB writes a single generated.duckdb file; quality/seasonality checks currently expect CSV or Parquet on disk.

Micro-benchmark (1M rows · generate + read + aggregate)

Recommendations

---

Product & pricing

Commercial packaging and roadmap (draft):

• Market research — why buyers pay, positioning, target customers, revenue model
• Data marketplace — sell synthetic data directly; feasibility and launch plan
• Data store SKUs — first 5 product listings ($19–$299)
• Product roadmap — what to sell now, build next, and target customers
• Pricing — tier packaging, vertical SKUs, and services
• Outbound templates — email templates for first design partners
• Dataset license — commercial license template for data store

---

Development

git clone git@github.com:Yogi776/data-generation-sdk.git
cd data-generation-sdk
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,all]"

pytest && ruff check . && mypy src

---

License

Apache-2.0 — see LICENSE.