agentlab-py

AgentLab Python Client

A Python client library for the AgentLab evaluation platform using Connect RPC. This library provides a simple and intuitive interface for running AI agent evaluations, managing evaluators, and accessing evaluation results.

🚀 Quick Start

pip install agentlab-py

Set your API token as an environment variable:

export AGENTLAB_API_TOKEN=your-api-token-here

from agentlab import AgentLabClient, CreateEvaluationOptions

client = AgentLabClient()

evaluation = client.run_evaluation(CreateEvaluationOptions(
    agent_name='my-agent',
    agent_version='1.0.0',
    evaluator_names=['correctness-v1'],
    user_question='What is the capital of France?',
    agent_answer='The capital of France is Paris.',
    ground_truth='Paris is the capital of France',
    metadata={'confidence': 0.95}
))

print(f"Evaluation completed: {evaluation.name}")

Retrieving Results

evaluation_run = client.get_evaluation_run('evaluation-run-id')

result_data = client.get_evaluation_result('evaluation-run-id')
print(result_data['results'])  # Parsed evaluator outputs

for evaluator_name, result in evaluation_run.evaluator_results.items():
    print(f"{evaluator_name}: {result.output}")

Listing Evaluation Runs

runs_response = client.list_evaluation_runs('project-123')
for run in runs_response.evaluation_runs:
    print(f"Run: {run.name} - Question: {run.user_question}")

Managing Agent Prompts

from agentlab import CreateAgentVersionOptions

# Publish agent version with prompts (idempotent)
result = client.publish_agent_version(CreateAgentVersionOptions(
    agent_name='my-assistant',
    version='1.0.0',
    prompts={
        'system': 'You are a helpful AI assistant...',
        'guidelines': 'Always be polite and professional.'
    }
))

print(f"Published version: {result.create_time}")
for name, content in result.prompts.items():
    print(f"  {name}: {content[:50]}...")

Analyzing Agent Performance

from agentlab import AnalysisParameters

# Create analysis for the last 30 days
params = AnalysisParameters(min_evaluation_runs=5, time_range_days=30)
session = client.analyze_agent('my-agent', '1.0.0', params)

# Get results
session = client.get_analysis_session(session.id)
if session.status.value == "ANALYSIS_STATUS_COMPLETED":
    stats = session.analysis_data.statistical_summary
    print(f"Success rate: {stats.success_rate:.1%}")
    print(f"Average score: {stats.average_score:.3f}")

Working with Test Datasets

# List all test datasets
datasets = client.list_test_datasets()
print(f"Found {len(datasets.test_datasets)} dataset(s)")

# Get tests from a dataset
tests = client.get_tests('dataset-uuid')
for test in tests.tests:
    print(f"Q: {test.user_question}")
    print(f"A: {test.ground_truth}")

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Issues and Feature Requests

• 🐛 Report a bug
• 💡 Request a feature

🔗 Links

• AgentLab Platform
• JavaScript SDK

🏢 About VectorLabs

AgentLab is developed by VectorLabs, a company focused on advancing AI agent evaluation and development tools.

---

Made with ❤️ by the VectorLabs team