ai01-eval

ai01-eval

Benchmark your AI agent or RAG pipeline against the AI01 leaderboard.

Install

pip install ai01-eval

Quick start

from ai01_eval import AI01Eval

client = AI01Eval(api_key="your-api-key")

# 1. Browse datasets
datasets = client.datasets.list()

# 2. Download a dataset
dataset = client.datasets.get("general-single-topic-v1")

# 3. Run your pipeline (ground-truth references are looked up server-side)
results = []
for item in dataset:
    answer = your_agent.run(item["query"], item.get("context"))
    results.append({
        "id":     item["id"],
        "query":  item["query"],
        "answer": answer,
    })

# 4. Submit — metrics are computed server-side
run = client.submit(
    dataset="general-single-topic-v1",
    results=results,
    agent_name="My RAG Agent v1",
)
print(run.scores)
# {'exact_match': 0.71, 'f1': 0.84, 'faithfulness': 0.79}
print(run.report_url)
# https://ai01.dev/benchmark?run=a3f2b1c9

---

Configuration

API key

Log in at ai01.dev to get your API key, then pass it explicitly or set the AI01_API_KEY environment variable:

# Explicit
client = AI01Eval(api_key="your-api-key")

# Via environment variable
# export AI01_API_KEY=your-api-key
client = AI01Eval()

API Reference

AI01Eval

Main entry point for the package.

client = AI01Eval(api_key="...", base_url="https://api.ai01.dev")

The client exposes three sub-clients:

---

client.datasets

client.datasets.list()

Returns metadata for all available datasets.

datasets = client.datasets.list()
# [
#   {"id": "general-single-topic-v1", "name": "...", "num_queries": 120, "metrics": [...]},
#   ...
# ]

Returns: list[dict]

---

client.datasets.get(dataset_id)

Downloads a dataset by ID and returns a Dataset object you can iterate over.

dataset = client.datasets.get("general-single-topic-v1")
print(dataset.id)          # "general-single-topic-v1"
print(dataset.name)        # human-readable name
print(dataset.num_queries) # total number of items
print(dataset.metrics)     # ["exact_match", "f1", "faithfulness"]
print(len(dataset))        # number of items downloaded

for item in dataset:
    print(item["id"])       # unique item identifier
    print(item["query"])    # the question to answer
    print(item["context"])  # grounding document (RAG datasets only)

Note: Ground-truth references are not included in downloaded items — they are looked up server-side when you submit results.

Returns: Dataset

---

Dataset

An iterable container for dataset items.

---

client.submit(...)

Submits your results to the AI01 server. Metrics are computed server-side.

run = client.submit(
    dataset="general-single-topic-v1",
    results=results,
    agent_name="My RAG Agent v1",
    submitter="your-username",       # optional, defaults to "anonymous"
    experiment_name="RAG baseline",  # optional
    description="First run",         # optional
    duration_seconds=t["duration_seconds"],  # optional
)

Parameters:

Each dict in results must contain:

Returns: RunReport

---

RunReport

Returned by client.submit(...) and client.runs.get(run_id).

print(run.id)          # "a3f2b1c9"
print(run.scores)      # {'exact_match': 0.71, 'f1': 0.84}
print(run.report_url)  # "https://ai01.dev/benchmark?run=a3f2b1c9"

---

client.runs

client.runs.get(run_id)

Retrieves a past submission report by run ID.

run = client.runs.get("a3f2b1c9")
print(run.scores)

Returns: RunReport

---

Error handling

All errors raised by this library inherit from AI01Error:

from ai01_eval import AI01Error, AI01AuthError, AI01NotFoundError, AI01RateLimitError

try:
    dataset = client.datasets.get("unknown-dataset")
except AI01AuthError:
    print("Check your API key.")
except AI01NotFoundError:
    print("Dataset not found.")
except AI01RateLimitError:
    print("Slow down — rate limit hit.")
except AI01Error as exc:
    print(f"Unexpected error: {exc}")

---

Available datasets

---

Metrics

All metrics are computed server-side to ensure fairness:

---

License

MIT