actbench

              __  __                    __  
  ____ ______/ /_/ /_  ___  ____  _____/ /_ 
 / __ `/ ___/ __/ __ \/ _ \/ __ \/ ___/ __ \
/ /_/ / /__/ /_/ /_/ /  __/ / / / /__/ / / /
\__,_/\___/\__/_.___/\___/_/ /_/\___/_/ /_/ 
                                  

Overview

actbench is a extensible framework designed to evaluate the performance and capabilities of web automation agents and LAM systems.

Installing actbench CLI

actbench requires Python 3.12 or higher. We recommend using pipx for a clean, isolated installation:

pipx install actbench

Usage

1. Setting API Keys

Before running benchmarks, you need to set API keys for the agents you want to use.

actbench set-key --agent raccoonai

You can list the supported agents and check which API keys are stored:

actbench agents list

2. Listing Available Tasks

actbench provides a built-in dataset of web automation tasks, crafted by merging and refining tasks from the webarena and webvoyager datasets.
Duplicate tasks have been stripped out, and the queries have been refreshed to align with the most recent information.
If you want to explore how the tasks have been modified, you can trace their IDs back to the original datasets for a side-by-side comparison.

To see all the tasks currently available, just run this command:

actbench tasks list

3. Running Benchmarks

The run command is the heart of actbench. It allows you to execute tasks against specified agents.

Basic Usage

actbench run --agent raccoonai --task 256 --task 424

This command runs tasks with IDs 256 and 424 using the raccoonai agent.

Running All Tasks

actbench run --agent raccoonai --all-tasks

This runs all available tasks using the raccoonai agent.

Running Random Tasks

actbench run --agent raccoonai --random 5

This runs a random sample of 5 tasks using the raccoonai agent.

Running with All Agents

actbench run --all-agents --all-tasks

This runs all tasks with all configured agents (for which API keys are stored).

Controlling Parallelism

actbench run --agent raccoonai --all-tasks --parallel 4

This runs all tasks using the raccoonai agent, executing up to 4 tasks concurrently.

Setting Rate Limiting

actbench run --agent raccoonai --all-tasks --rate-limit 0.5

This adds a 0.5-second delay between task submissions.

Disabling Scoring

actbench run --agent raccoonai --all-tasks --no-scoring

This disables the LLM powered scoring, and gives all tasks a score of -1.

Combined Options

You can combine these options for more complex benchmark configurations:

actbench run --agent raccoonai --agent anotheragent --task 1 --task 2 --random 3 --parallel 2 --rate-limit 0.2

This command runs tasks 1 and 2, plus 3 random tasks, using both raccoonai and anotheragent (assuming API keys are set), with a parallelism of 2 and a rate limit of 0.2 seconds.

4. Viewing Results

The results command group allows you to manage and view benchmark results.

Listing Results

actbench results list

You can filter results by agent or run ID:

actbench results list --agent raccoonai
actbench results list --run-id <run_id>

Exporting Results

You can export results to JSON or CSV files:

actbench results export --format json --output results.json
actbench results export --format csv --output results.csv --agent raccoonai

Here's a complete table detailing the actbench CLI commands, their flags (options) and explanations:

Extending actbench

Adding New Agents

1. Create a new client class: Create a new Python file in the actbench/clients/ directory (e.g., my_agent.py).
2. Implement the BaseClient interface: Your class should inherit from actbench.clients.BaseClient and implement the set_api_key() and run() methods.
3. Register your client: Add your client class to the _CLIENT_REGISTRY in actbench/clients/__init__.py.

Adding New Datasets

1. Create a new dataset class: Create a new Python file in the actbench/datasets/ directory (e.g., my_dataset.py).
2. Implement the BaseDataset interface: Your class should inherit from actbench.datasets.BaseDataset and implement the load_task_data(), get_all_task_ids(), and get_all_tasks() methods.
3. Provide your dataset file: Place your dataset file (e.g., my_dataset.jsonl) in the src/actbench/dataset/ directory.
4. Update _DATASET_INSTANCE: If you want to use this dataset by default, update the _DATASET_INSTANCE variable in src/actbench/datasets/__init__.py.

Adding New Evaluation Metrics

You can customize the evaluation process by modifying the Evaluator class in actbench/executor/evaluator.py or by creating a new evaluator and integrating it into the TaskExecutor.

Contributing

Contributions are welcome! Please follow these simple guidelines:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Write clear and concise code with appropriate comments.
4. Submit a pull request.