0.2.1

AGI Client

A Python client library for interacting with the General Reasoning platform API.

The documentation is available here.

Installation

You can install the package using pip:

pip install agi

Making API Calls

Obtain an API key from the website. Then:

import agi

client = agi.Client(api_key=YOUR_API_KEY)

# Download reasoning traces and verifications as a .jsonl
client.data.get(task='math-word-problems', model='DeepSeek-R1')

Evaluating your reasoning model

The full evaluation guide is available here. A boilerplate example is shown below:

import agi
                
client = agi.Client("YOUR_API_KEY")
model = "USERNAME/MODEL_NAME"

# Retrieve test questions
data = client.evals.get(
    task='mathematical-brainteasers', 
    split='test'
)
question = data['questions']

# Submit model evaluations
for question in questions:
    reasoning_trace, answer = MyModel(
        system_prompt=question['system_prompt'],
        question=question['text']
    ) # Replace with your model logic
    client.evals.submit(
        id=question['id'],
        model=model,
        reasoning_trace=reasoning_trace,
        answer=answer
    )

Running our verification and grading models

Read more about our grader and verifier here

import agi 
 
client = agi.Client(api_key=YOUR_API_KEY)
 
response = client.verify.create(
  model="endeavour-preview",
  question="Are Dyson spheres feasible to build?",
  model_answer="Yes"
)

import agi 
 
client = agi.Client(api_key=YOUR_API_KEY)
 
response = client.grade.create(
  model="endeavour-preview",
  question="Are Dyson spheres feasible to build?",
  model_answer="Yes",
  ground_truth_answer="No",
)

License

This project is licensed under the MIT License - see the LICENSE file for details.