agentodyssey

AgentOdyssey is a lightweight interactive environment that supports both novel game generation, a unified agent interface, and multifaceted evaluation. It is designed to evaluate test-time continual learning agents across five key abilities: exploration, world knowledge acquisition, episodic memory, skill learning, and long-horizon planning. Its main features include:

• Open-Ended Long-Horizon Game Generation: Generate games with entirely new and rich entities, dynamics, and storylines from a single command.
• Unified Agent Interface: All LLM-based agents maximize prompt sharing via inherited classes to ensure fair comparison. Adding a new agent can be done by simplely implementing a few methods.
• Multifaceted Evaluation Metrics: Includes a range of metrics beyond game progress to probe specific failure modes of agents.

Table of Contents

Quickstart

1. Install

conda create -n agentodyssey python=3.12 && conda activate agentodyssey
git clone https://github.com/agentodyssey/agentodyssey.git && cd agentodyssey
pip install -e .

2. Set your API key (if using proprietary LLMs). For example, for OpenAI:

export OPENAI_API_KEY="your-key"

3. Run an evaluation

# Play the game yourself
python eval.py --game_name remnant --agent HumanAgent

# Evaluate an LLM agent (i.e. Long Context Agent)
python eval.py --game_name remnant --agent LongContextAgent --llm_provider openai --llm_name gpt-5

[!NOTE] See the full parameters for evaluation → Running Evaluations

PyPI Package

1. Install

pip install agentodyssey

2. Python API

AgentOdyssey provides a Python wrapper for seamless integration into your own evaluation pipelines:

from agentodyssey import AgentOdyssey

AgentOdyssey.run(game_name="remnant", agent="LongContextAgent", llm_provider="openai", llm_name="gpt-5")

3. CLI tool

# Play the game yourself
agentodyssey run --game-name remnant --agent HumanAgent

# Evaluate an LLM agent
agentodyssey run --game-name remnant --agent LongContextAgent --llm-provider openai --llm-name gpt-5

[!NOTE] The CLI uses hyphens (--game-name) while eval.py and the Python API use underscores (--game_name).

Game Generation

The generation pipeline creates a complete game world through three stages:

1. Entity generation populates the world with locations, objects and NPCs.
2. Rule generation adds new world dynamics including action rules that describe player-invoked actions (e.g., pick up, craft, etc) and step rules that describe automatic environment dynamics (e.g., day-night cycle).
3. Quest generation devises the main storyline that acts as goals for the agent to pursue.

# Generate a themed game and run it
agentodyssey generate "a pirate-themed island adventure" --game-name pirate
agentodyssey run --game-name pirate --agent LongContextAgent --llm-provider openai --llm-name gpt-5

Generate with full control over world size, quest structure, and the LLM used for generation:

agentodyssey generate "a haunted castle with undead enemies" \
    --game-name haunted \
    --num-places 4 \
    --num-objects 20 \
    --num-npcs 10 \
    --num-action-rules 2 \
    --num-step-rules 1 \
    --num-quest-chapters 2 \
    --quest-description "Defeat the Lich King and restore the castle" \
    --llm-provider openai \
    --llm-name gpt-5

[!NOTE] See the full parameters for game generation → Generating Games

Learn more about the game ontology → Game Ontology

Agent Paradigms

The environment already implements agents spanning 6 paradigms as shown below. All LLM-based agents use ReAct prompting.

Some agents can be augmented with three optional add-ons: reflection, summarization, and short-term memory (Please refer to the paper for details.)

[!NOTE] See detailed descriptions of each paradigm → Agent Paradigms

Learn how to implement your own agent → Custom Agents

Evaluation Metrics

AgentOdyssey evaluates agents along multifaceted axes:

Game Progress measures how far the agent advances through in-game objectives. The main reward tracks main quest stage completion, while the supplementary reward captures exploration, crafting, combat, and side quest progress.

Model Cost tracks the total input and output tokens consumed by each agent during a run.

Diagnostic Testing probes specific capabilities through targeted tests:

[!NOTE] See metric definitions and how to run diagnostic evaluation → Evaluation Metrics

Additional Dependencies

The base requirements.txt covers most functionality, but certain agents and providers need extra packages:

Install only what you need, e.g.:

pip install tiktoken umap-learn tenacity   # for RaptorRAGAgent

Contributing and Trouble Shooting

For common issues and solutions, please refer to our Troubleshooting Guide. We welcome contributions from everyone and appreciate your help making the project better. For bug reports, feature suggestions, code contributions, and general questions, please refer to our Contribution Guidelines.

Citation

✨ If you find AgentOdyssey useful, please cite our work:

@inproceedings{agentodyssey2026,
  title     = {AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents},
  author    = {Zhang, Zheyuan and Wen, Zehao and Zhang, Alvin and Wang, Andrew and Xie, Jianwen and Khashabi, Daniel and Shu, Tianmin},
  year      = {2026},
}