agentflow-runtime

AgentFlow

Event-native metrics layer: business metrics that move when events happen — measured 1.1 s p50 event-to-metric freshness on production defaults. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.

Why this exists

BI on a replica answers yesterday's questions. Support, ops, and merch workflows need current orders, metrics, and health signals at the moment of decision — not a stale warehouse snapshot, not a pile of one-off service adapters, and not a cache that quietly serves 30-second-old numbers.

AgentFlow's axis is event → live metric: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan (docs/freshness-benchmark.md). One serving boundary on top of that axis:

• streaming ingestion for operational events (validated, enriched, journaled)
• a semantic layer that exposes entities, metrics, lineage, and query endpoints
• typed, versioned contracts — each metric ships with its source events and a staleness budget
• Python and TypeScript clients that speak the same API surface

Consumers are whoever needs the number now: humans, dashboards, downstream services, and AI agents — agents are one consumer, not the product.

Highlights

• Measured event-to-metric freshness — an event entering the pipeline is reflected in GET /v1/metrics/* in 1.06 s p50 / 1.99 s p95 on production defaults (event-driven cache invalidation, no webhook registration), tunable to 238 ms p50; a plain TTL cache on the same pipeline sits at ~15 s. Reproducible via python scripts/benchmark_freshness.py → freshness benchmark
• Lineage as a contract — all six metrics declare their source events, serving table, and a 2.5 s p95 staleness budget in versioned contracts, exposed through /v1/catalog and /v1/contracts and pinned by tests against the actual write path
• Published release line through v1.6.0 on PyPI (agentflow-runtime, agentflow-client) and npm (@yuliaedomskikh/agentflow-client) via OIDC Trusted Publishers with SLSA provenance on every artifact
• Tested and gated — 1,500+ unit tests plus a broad Windows no-Docker suite; CI enforces 13 required status checks (lint, schema, unit, integration, helm, perf, terraform, bandit, safety, npm-audit, trivy, contract, build-smoke) through branch protection
• Dual SDK parity across Python and TypeScript — retries, circuit breakers, batching, pagination, contract pinning, idempotency keys, as_of historical reads — over sub-second entity lookups (p50 38–55 ms, p99 167 ms on local hardware)
• Security in the hot path — tenant isolation on every read surface, parameterized queries, sqlglot AST validation for NL-to-SQL, fail-closed auth, secret scrubbing, and a Bandit gate for new findings
• Production-shaped extras — two CDC paths (hardened Debezium/Kafka Connect + a ClickHouse per-branch fan-out), on-call runbooks, and a narrated demo of the DV2 multi-branch warehouse

Quick start

Upgrading from v1.0.x? See the v1.1 migration guide before installing.

Prerequisites:

• Python 3.11+
• make
• Docker Compose (make demo starts Redis and the ClickHouse serving store)

PowerShell 7+:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
. .\scripts\setup.ps1
make demo

macOS / Linux:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
source ./scripts/setup.sh
make demo

make demo starts Redis and ClickHouse, seeds demo data through the full pipeline (validated events land in the ClickHouse serving store), and serves the API on http://localhost:8000. Swagger UI is available at http://localhost:8000/docs.

Try it:

curl http://localhost:8000/v1/entity/order/ORD-20260404-1001

curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"Show me top 3 products"}'

Local demo runs without API-key enforcement unless you explicitly configure AGENTFLOW_API_KEYS_FILE.

Architecture

Event sources -> Kafka -> Flink -> Iceberg --------\
                                                    -> Semantic layer -> FastAPI -> Agent / SDK
Local demo   -> local_pipeline -> ClickHouse ------/
                       (DuckDB stays the local lake / test store)

Stack:

• Ingestion: Kafka producers, Debezium/Kafka Connect CDC, and a local synthetic pipeline
• Processing: Flink plus validation and enrichment stages
• Storage: Iceberg for production-shaped tables; ClickHouse is the serving store (ADR 0006 — ReplacingMergeTree upserts, final=1 reads), DuckDB the local-dev / test store
• Serving: FastAPI, contract registry, lineage, search, and operational endpoints
• Orchestration: Dagster
• IaC: Terraform, Helm, Docker Compose, and a Fly.io demo config

See docs/architecture.md for the detailed design, trade-offs, and deployment topologies.

CDC source capture is standardized on Debezium/Kafka Connect; downstream consumers use the canonical AgentFlow CDC contract defined in ADR 0005.

What's inside

Documentation

Core

• Architecture — system context, data flow, failure modes
• API Reference — endpoint-by-endpoint curl / Python / TypeScript examples
• Operational Runbook + On-Call Runbooks — local stack, CDC capture, and production-incident playbooks
• Security Audit — threat model, controls, and evidence
• Glossary — interview-ready explanations of the core technical terms
• Interactive Technical Walkthrough — MkDocs Material guide (Mermaid architecture, SDK, deployment, observability)

Deep dives

• DV2.0 Multi-Branch Extension — Data Vault 2.0 model for mid-market e-com (5 locations / 3 jurisdictions): schema, end-to-end flow, demo evidence
• CDC Deployment Plan — Debezium/Kafka Connect rollout
• Competitive Analysis · Release Readiness · Cost Analysis
• Fly.io Demo Deploy — minimal hosted demo
• Contributing · Changelog

Development

# verified release slice
python -m pytest tests/unit tests/integration tests/sdk -q

# benchmark and regression gate
python scripts/run_benchmark.py
python scripts/check_performance.py --baseline docs/benchmark-baseline.json --current .artifacts/load/results.json --max-regress 20

# benchmark trend: [.github/perf-history.json](.github/perf-history.json) is appended on every main push;
# render the history locally with `make perf-plot` (writes docs/perf/history.html).

# contracts and security
python scripts/generate_contracts.py --check
bandit -r src sdk --ini .bandit --severity-level medium -f json -o .tmp/bandit-current.json
python scripts/bandit_diff.py .bandit-baseline.json .tmp/bandit-current.json

Status

v1.6.0 is the current release line — PyPI agentflow-runtime / agentflow-client and npm @yuliaedomskikh/agentflow-client, all published via OIDC Trusted Publishers with SLSA provenance attestations. CI on main is green across all 13 required checks.

The v1.1.0 → v1.6.0 arc landed in six increments on top of a security audit-closure sprint:

• v1.1.0 — audit closure: tenant isolation across every read surface, SQL guard centralized on sqlglot, entity allowlist enforcement, fail-closed auth, secret rotation, Helm hardening, OpenAPI drift gate, and the required status checks.
• v1.2.0 — DV2 multi-branch warehouse: 38 Data Vault 2.0 tables (8 hubs / 8 links / 22+ satellites), an Argo Workflows dv2-refresh template, a dbt project (3 mart models + 12 tests), and per-branch CDC fan-out via ClickHouse MaterializedPostgreSQL.
• v1.3.0 — helm/kafka-connect hardening matched to helm/agentflow (NetworkPolicy + PDB + securityContext), live Helm validation across both charts, and the narrated DV2 demo (terminal + web-UI + dbt docs).
• v1.4.0 — maintenance: on-call runbooks, SECURITY.md, issue/PR templates, contract/DORA CI hardening, repo hygiene, and a dependency wave (mypy, Terraform AWS provider, TypeScript, GitHub Actions, Vitest). No runtime API changes from v1.3.0.
• v1.5.0 — security & correctness hardening: argon2id key hashing with an O(1) peppered lookup index (M-C4), an NL→SQL guard bypass fix (typed read_csv / read_parquet scan functions now denied in projection position), sqlglot control-byte and mutation-target repairs, and a strict-mypy expansion across the orchestration and freshness slices. No public API changes.
• v1.6.0 — the architecture-fixing release: ClickHouse becomes the shipped serving engine (pipeline sink, ReplacingMergeTree row versions, backend-routed event scan, a dedicated CI E2E lane against a real ClickHouse), PII protection moves from the removed app-level string-parse gate to engine-enforced vault governance (fail-closed column grants, per-jurisdiction officer roles, row policies, SQL SECURITY DEFINER views — every live adversarial probe green), plus the vendored NL→SQL generation engine (LangGraph, routed through GraceKelly), the DV2 raw vault on PostgreSQL with LISTEN/NOTIFY freshness, the MinIO-backed PyIceberg catalog, and the OpenSSF Scorecard channel (5.8 → 7.0).

The tagged line and main are in sync as of v1.6.0. See the changelog for full detail.

Scope

This is a reference data-engineering project. The streaming, warehouse, and deployment artifacts (Flink, Iceberg, Helm, Terraform, k8s) are exercised against a local pipeline and a kind cluster in CI rather than a managed cloud. Wiring it to a live production source needs inputs that live outside the repo — CDC source onboarding (runbook ready in docs/operations/cdc-production-onboarding.md), a public benchmark on production-grade hardware, and an external pen-test attestation.

Screenshots

Capture notes and publish-time checks are listed in docs/publication-checklist.md.

License

MIT. See LICENSE.

Credits

Built as a data-engineering reference project. Initial release cycle 2026-04-10 → 2026-04-20, with post-audit hardening and the DV2 extension landing through v1.4.0. Architecture decisions are recorded as ADRs in docs/decisions/.