aether-observer

Aether KG Eval

Importable Python SDK for evaluating knowledge-graph builders against benchmark KG datasets.

The core customer flow is:

kg dataset -> kg builder -> built KG / build trace -> kg eval result

Install

python -m pip install aether-observer

For local development from this repository:

python -m pip install -e ".[dev]"

Python API

from aether_kg_eval import dataset_from_triples, evaluate_kg_builder

dataset = dataset_from_triples(
    [
        ("entity:1", "relation:works_at", "entity:2"),
        ("entity:3", "relation:reports_to", "entity:1"),
    ],
    name="customer-kg-smoke",
)

def builder(dataset):
    return dataset.triples

result = evaluate_kg_builder(dataset, builder)

print(result.overall_score)
print(result.intent_drift.status)
print(result.behavior_drift.status)
print(result.kg_quality.to_dict())

The v1 result reports:

• intent_drift: whether the builder preserved the intended KG construction task.
• behavior_drift: whether builder behavior or KG shape moved away from baseline.
• orchestration_drift: reserved as not_applicable for single-builder pipelines.
• kg_quality: triple, entity, relation, duplicate, invalid, and contradiction metrics.

Builder Contract

evaluate_kg_builder(dataset, builder) accepts builders that return one of:

• BuiltKG
• dict with triples and optional trace
• list[KGTriple]
• list[tuple[subject, relation, object]]

Trace entries can be supplied as existing EpisodeRecord objects or dicts matching the legacy JSONL episode shape. When trace is absent, KG quality is still scored and behavior drift falls back to KG quality.

Legacy Episode Format

Each JSONL record is an episode:

{"id":"e1","timestamp":"2026-05-15T00:00:00Z","content":"Alice joined Acme.","content_type":"text","source":"seed","agent_id":"agent-a","task_id":"build-kg","entity_types":["Person","Org"],"relation_types":["JOINS"],"tool_calls":["extract"],"expected_intent":"add employment fact","outcome":"success"}

Important optional fields:

• entity_types, relation_types, graph_nodes, graph_edges, embedding, contradictions, invalidations
• tool_calls, handoff, coordination_failure, boundary_violation, human_intervention
• expected_intent, observed_intent, reasoning_trace, outcome

The legacy kg_drift_builder modules remain available internally for rolling-window scoring and Graphiti/Kuzu experiments, but the sellable SDK surface is aether_kg_eval.

Optional Graphiti/Kuzu

The old Graphiti/Kuzu ingestion adapter is still present as an optional integration layer. It is not required to evaluate normal Python KG outputs.