agent-reliability

Agent Reliability

AI agents can finish successfully while still doing the wrong thing. Agent Reliability provides vendor-neutral Python primitives for measuring whether agents meet explicit reliability objectives.

It brings evaluations, SLOs, error budgets, burn rates, and measurement provenance to agent applications—and refuses to produce a misleading number when evaluation methodologies are incompatible.

Status: GA (1.2.1). Public APIs documented as stable in GA_CONTRACT.md follow Semantic Versioning. See compatibility.

Why this exists

Traces explain what an agent did. Reliability answers whether it consistently achieved a defined outcome. Agent Reliability connects one logical execution to an explicit evaluation method, then calculates exact local reliability against an SLO.

The OSS package works offline and without a hosted service. It does not automatically capture prompts, responses, tool arguments, credentials, or arbitrary application payloads. The base install has no runtime dependencies and sends nothing over the network.

30-second example

python -m pip install agent-reliability

This standalone example instruments four agent-like calls, evaluates task_success, records attributable results, and applies a 75% SLO:

from fractions import Fraction

from agent_reliability.domain import ObjectiveDirection, Slo, UnknownPolicy
from agent_reliability.evaluation import (
    EqualityEvaluator,
    EvaluationResult,
    EvaluatorIdentity,
)
from agent_reliability.reliability import (
    AggregationConflict,
    ReliabilityObservation,
    evaluate_reliability,
)
from agent_reliability.sdk import AgentReliability, EvaluatorRunner

sdk = AgentReliability()
runner = EvaluatorRunner()
evaluator = EqualityEvaluator(EvaluatorIdentity("expected-answer", "1"), "approved")
observations = []

for actual in ("approved", "approved", "needs-review", "approved"):
    with sdk.run(agent_id="approval-agent", name="Approval Agent", version="1") as run:
        result = runner.evaluate(evaluator, actual)
        if not isinstance(result, EvaluationResult):
            raise RuntimeError("evaluation did not produce an observation")
        run.record_evaluation(indicator="task_success", result=result)
        observations.append(
            ReliabilityObservation.from_evaluation(
                indicator="task_success", result=result
            )
        )

report = evaluate_reliability(
    indicator="task_success",
    observations=observations,
    slo=Slo("task-success", Fraction(3, 4), ObjectiveDirection.AT_LEAST),
    unknown_policy=UnknownPolicy.EXCLUDE,
)
if isinstance(report, AggregationConflict):
    raise RuntimeError("incompatible measurement methodologies")

print(f"Reliability: {float(report.ratio.pass_ratio):.2%}")
print(f"SLO status: {report.slo_evaluation.status.value.upper()}")

Output:

Reliability: 75.00%
SLO status: MET

This block runs in CI. The canonical example also shows the error budget. Follow the 5–10 minute quickstart for interpretation and next steps.

What it measures

• An indicator says what is measured, such as task_success.
• An evaluator says how it is judged and returns PASS, FAIL, or UNKNOWN.
• Provenance records evaluator name, behavior version, configuration, and determinism.
• An SLI is the observed ratio; an SLO is the desired target.
• The error budget is permitted unreliability; burn rate compares an observed bad-event rate with that allowance.

UNKNOWN means evaluation completed but was indeterminate. An EvaluationExecutionFailure means the evaluator or its timestamping failed; it is not an agent failure and creates no observation.

Reliability answers “how often is the agent behaving correctly?” Measurement health answers “do we have enough trustworthy evidence to make that claim?” They remain independent, and applications—not the SDK—decide how degraded evidence affects an action. See the measurement-health guide. Runnable fail-open, fail-closed, and bounded-degradation examples are in the application policy guide.

If evaluator v1 and v2 measured the same indicator, the engine returns an AggregationConflict instead of averaging them. A changed measurement method is not automatically comparable. See Core concepts.

Installation

Python 3.11–3.13 is supported. The distribution and import names differ:

pip install agent-reliability
import agent_reliability

The only optional runtime extra is the OpenTelemetry API bridge:

python -m pip install "agent-reliability[otel]"

Framework compatibility

Any Python agent can use the explicit sync or async context manager. Wrap one logical task execution, evaluate the relevant output, and retain observations for the window your application chooses. No framework adapter, monkey patch, API key, storage layer, or network service is required. See Integrations and the async example.

The local engine calculates one supplied collection at a time; it does not retain history or select rolling windows.

OpenTelemetry

OpenTelemetryRunContextBridge activates the agent span in an existing host trace. Your application owns the TracerProvider, sampling, processors, propagation, exporter, collector, and backend. Agent Reliability configures none of them and exports nothing by itself. See the OTel example and mapping reference.

Project status and scope

Agent Reliability is a stable, local-first Python SDK for measuring and testing the reliability of AI agents.

The SDK provides sync and async agent instrumentation, deterministic evaluators, evaluator provenance, reliability aggregation, SLO and error-budget semantics, measurement-health tracking, optional OpenTelemetry interoperability, human-readable local reports, machine-readable reliability results, and framework-neutral SLO assertions for tests and CI.

The SDK is designed to work standalone. It requires no hosted account, API key, or network service, and has zero mandatory runtime dependencies.

Current release: 1.2.1

The public API follows semantic versioning, with compatibility and installed-artifact verification included in the release process.

No remote ingestion, dashboard, LLM judge, persistence, auto-instrumentation, or framework-specific adapter is included. See the roadmap.

Documentation

• New developer: Quickstart → Concepts
• Integrator: Integration guide → Evaluate → Measure reliability locally → Test reliability
• Advanced user: Evaluator framework and local engine
• Architecture reader or contributor: documentation index

Development and contributing

See CONTRIBUTING.md for setup and quality gates. Security vulnerabilities belong in the private process in SECURITY.md, not a public issue.

License

Apache License 2.0. See LICENSE.