acf-framework

ACF Framework

A graph-based framework for measuring Artificial General Intelligence.

The AGI Certification Framework (ACF) provides a rigorous, multi-dimensional scoring system for evaluating AI system capabilities against human professional standards. All knowledge — dimensions, measures, hypotheses, collected data — lives in an RDF graph, queryable via SPARQL.

Quick Start

# Install from PyPI:
pip install acf-framework

# Explore the framework
acf dimensions              # List all 12 ACF dimensions
acf measures                # List all 75 measures
acf levels                  # Show certification levels (ACF-1 through ACF-6)

# Evaluate your system
acf validate path/to/data/  # Validate data files against schemas
acf score path/to/data/     # Generate ACF profile with certification level

# Query the knowledge graph directly
acf query "SELECT ?id ?name WHERE { ?m acf:mapsTo <#Depth> ; acf:id ?id ; rdfs:label ?name . }"

The Twelve Dimensions

The ACF evaluates AI systems across 12 dimensions (independently measurable; see the limitations section for the H122.1 finding on cross-dimensional correlation):

Why Neurosymbolic? Relevance to NeSy Evaluation

ACF was designed to surface capabilities that LLM-only benchmarks (MMLU, HellaSwag, GSM8K, BIG-Bench, HELM) structurally cannot measure. Four dimensions are NeSy-native:

These four dimensions are jointly reported on the same task instances, which is the primary contribution of ACF as a benchmark (see Paper 122 for related work positioning vs HELM, PrOntoQA, FOLIO, CLEVR-Hans, and R-Tuning).

Certification Levels

Graph-First Architecture

ACF uses yurtle-rdflib to load all knowledge files into a live RDF graph at startup. This means:

• All 75 measures carry a dimension-mapping triple linking them to ACF dimensions (acf:mapsTo; the proposed tranche uses acf:mapsToDimension)
• All 16 hypotheses are queryable with their targets and validation methodology
• Collected data (JSON files) is ingested as RDF triples for unified querying
• Ad-hoc analysis via SPARQL — no custom code needed for common queries

# Which dimensions have fewer than 4 measures? (coverage gaps)
SELECT ?dim ?name (COUNT(?m) AS ?count) WHERE {
  ?dim a acf:Dimension ; rdfs:label ?name .
  OPTIONAL { ?m acf:mapsTo ?dim . }
} GROUP BY ?dim ?name HAVING (COUNT(?m) < 4)

Data Collection

ACF defines three record types for collecting evaluation data:

# Print a blank template
acf template experiment-run

# Validate your data
acf validate my-data/experiment-results.json

CLI Reference

acf dimensions [name]                  # List/show dimension details
acf measures [--dimension X]           # List measures, filter by dimension
acf levels                             # Show certification levels
acf validate <data-file|data-dir>      # Validate data against schemas
acf score <data-dir>                   # Score system → ACF profile
acf compare <profile1> <profile2>      # Compare two ACF profiles
acf template <record-type>             # Print blank data template
acf query "<sparql>"                   # Run SPARQL over knowledge + data
acf info                               # Show framework version and stats

75 Measures

The framework includes 75 measures across 13 categories:

• Accuracy (M-001–M-010, M-070): Routing, hallucination, provenance, calibration, register mislabel
• Latency (M-011–M-018): Processing paths, queries, end-to-end response
• Efficiency (M-019–M-021): Compute acceleration, call reduction
• Quantity (M-022–M-025): Knowledge graph growth, topic coverage
• Quality (M-026–M-029): Decision quality, task autonomy, error patterns
• Metacognitive (M-030–M-035): Self-assessment, automated decisions, reflection
• Y-Layer Population (M-036–M-043): Knowledge items per layer (Y0-Y6)
• Y-Layer Growth (M-044–M-046): Enrichment rate, confidence, diversity
• Y-Layer Quality (M-047–M-052): Fact correctness, inference coherence
• Y-Layer Structural (M-053–M-055): Provenance, connectivity
• Proposed / Action Capability (M-056–M-069): Domain count, Bloom profile, task completion, action capability (M-067–M-069, v1.1)
• Safety / Containment (M-071–M-072, M-075): Containment violation rate, manifest-grant enforcement, regression rollback (v1.2)
• Knowledge Transfer (M-073–M-074): Load/unload differential, unloaded hallucination rate (v1.2)

16 Hypotheses

The framework includes 16 testable hypotheses about AGI measurement:

• H122.1: Breadth-Depth Independence (r < 0.3)
• H122.2: Human Certification Equivalence (ACF-3+)
• H122.3: Service Orientation Correlation (r > 0.7)
• H122.4: Epistemic Honesty / Never-Launder — assertoric false_proofs=0 AND register mislabel_rate=0, full generative capability retained (supersedes the v1.0 Provenance Requirement, which survives as the corroborating accuracy-with-provenance sub-measure)
• H122.5: LLM Depth Ceiling (<60% on L5-L6)
• H122.6: Understanding Efficiency (UER >=10x)
• H122.7: Compositional Generalization Gap
• H122.8: Knowledge Transparency Enables Verification
• H122.9: GBA Distinguishes Expert Systems
• H122.10: Paradigm Selection Capability
• H122.11: Symbolic-to-Motor Transfer
• H122.12: Autonomous Gap Detection (70%+)
• H122.13: Crystallization Efficiency (99%+)
• H122.14: Self-Directed Learning
• H122.15: Capability Containment — a loaded module cannot exceed its signed manifest grants (Safety/Containment SC4; v1.2)
• H122.16: Capability Transfer — module-attributable load/unload differential; unloaded hallucinated-domain-answer rate = 0 (Knowledge Transfer KTR4; v1.2)

Installation

From PyPI

pip install acf-framework

From source (development)

git clone https://github.com/hankh95/acf-framework.git
cd acf-framework
pip install -e ".[dev]"

Dependencies

• Core: yurtle-rdflib, click, rich
• Development: pytest, ruff, mypy

Honest Limitations

The framework is open-source and usable today, but several caveats matter for anyone considering ACF as an evaluation reference:

• Zero third-party adoption to date. Every reported result comes from the author's own neurosymbolic platform. Cross-paradigm baselines (LLMs, classical expert systems) are in progress but not yet reproduced by independent evaluators.
• Reference scoring for FR-36 and CG-100 is graph-structural, not answer-grading. The reference scoring procedure measures structural properties of the system's reasoning trace (number and diversity of inference rules, chain depth, predicate diversity) and assigns pass/fail by sorting items in a stable order and passing the top N according to a rate computed from those metrics. This means the reference scorer is appropriate for graph-backed (neurosymbolic) systems that expose a rule-firing trace; LLM evaluation requires an adapted scoring path (LLM-judge or human-judge) which is outside the scope of the v1.1.1 reference scorer. See Paper 122 §5 and the per-battery methodology docs for the full procedure.
• Zorblaxia uses heuristic response classification on 10 items. Decline / Hedged / Confabulated classification matches 29 decline-signal patterns and 18 confabulation-signal patterns in the response text; the 80% decline-rate pass threshold is operational, not validated against human raters. Sample size and threshold calibration are explicit future work.
• H122.1 (Breadth-Depth orthogonality) was rejected. Across 7 evaluated systems, Breadth and Depth show a moderate inverse correlation (r = -0.434), not the expected near-independence (|r| < 0.3). The framework continues to treat the dimensions as independently measurable rather than statistically independent — see Paper 122 for the discussion of this finding.
• No human-rater validation of Bloom-level scoring. Bloom L1-L6 classification is automated. An inter-annotator agreement study is planned future work.
• Novel batteries (Zorblaxia) lack comparable published precedent. Zorblaxia is positioned as a contribution of this framework, not as a replication of an existing test.

Ecosystem

Part of the open-source stack behind Congruent Systems:

• nusy-reasoners — proof-carrying reasoning engines over Apache Arrow: derivations you can audit, abstention you can trust.
• nusy-kanban — Arrow-native, distributed kanban for multi-agent teams, with a built-in Hypothesis-Driven-Development research workflow.
• noesis-ship — pluggable multi-agent communication platform on NATS (EventBus, KV, object store).

License

MIT — see LICENSE.

Citation

If you use the ACF framework in your research, please cite:

@software{acf-framework,
  title = {ACF Framework: A Graph-Based Framework for Measuring AGI},
  author = {Head, Hank and {Congruent Systems LLC}},
  year = {2026},
  doi = {10.5281/zenodo.21651775},
  url = {https://doi.org/10.5281/zenodo.21651775},
  license = {MIT}
}