aftab

Overview

Aftab (Persian: آفتاب, meaning "sun" or "sun rays") is a benchmarking framework for evaluating CNN-based encoders in PQN across Atari games. It provides standardized training, evaluation, and reproducibility tools for deep reinforcement learning research.

See how the Aftab architecture compares to standard PQN baselines in these video demonstrations.

This research was done without receiving any funds; therefore, if you found our work useful, please consider sponsoring on GitHub 💛.

Encoder Experiments

Hadamax Experiments

References:

• Hadamax Encoding: Elevating Performance in Model-Free Atari

Q-Values Experiments

References:

• Stop Regressing
• Deep Exploration via Bootstrapped DQN
• Improving Regression Performance with Distributional Losses

Procgen (Overfitting Prevention) Experiments

Since there are no public benchmarks comparing human-normalized-scores of Procgen environments, we created PNS (Procgen Normalized Score) that is a minimal min-max normalization of scores across seeds.

Installation

Install via pip:

pip install aftab

Alternatively, you can clone the repository and install in editable mode.

git clone https://github.com/tahashieenavaz/aftab.git aftab_source
pip install -e aftab_source

We highly recommend using Micromamba for creating virtual environments with instructions detailed here.

Training Agents

Currently JAX API is under development and is planned to be finished by the end of 2026. Contributions are highly encouraged.

from aftab import Aftab
from aftab import aftab_environments

seeds = [1, 2, 3, 4]

for environment in aftab_environments:
    agent = Aftab(encoder="gamma", frames="pilot")
    for seed in seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

Custom Encoder Injection

You can define your own encoder as a PyTorch module and pass it to the agent:

import torch
from aftab import Aftab

class CustomImageEncoder(torch.nn.Module):
    pass

agent = Aftab(encoder=CustomImageEncoder)

Results

All experimental results are organized by experiment category. Each section contains:

• Tables: numerical results (HNS/PHS and raw scores)
• Charts: IQM normalized scores and training curves

Encoder Experiments

Tables

• Human Normalized Scores
• Scores

Charts

• IQM HNS
• Loss Evolution

---

Hadamax Experiments

Tables

• Human Normalized Scores
• Scores

Charts

• IQM HNS
• Loss Evolution

---

Q-Value Experiments

Tables

• Human Normalized Scores
• Scores

Charts

• IQM HNS
• Loss Evolution

---

Procgen Experiments

Tables

• Procgen Normalized Scores
• Scores
• PNS AUC by Seed
• PNS AUC by Game

Charts

• IQM PNS

Model Complexity

Base Variants

Note: The Eta variant has significantly more parameters than other variants, primarily due to the encoder producing a large number of features.

---

Hadamax Variants

Hyperparameters

The following tables reflect the defaults defined by Aftab. The experiment_name argument is required and has no default.

Training and Environment

Optimizer

Distributional and Bootstrapped (Ensemble) Q-Values

Procgen Overrides

For Procgen environments, Aftab automatically applies the two overrides above; other defaults remain unchanged.

Statistical Significance

Encoder Experiments

Hadamax Experiments

Q-Value Experiments

Reproducibility

Due to the stochastic nature of deep reinforcement learning, exact reproducibility via fixed datasets is not feasible.
Instead, we provide a set of random seeds used in our experiments.

from aftab import aftab_seeds

print(aftab_seeds)

Full experiment replication:

from aftab import Aftab
from aftab import aftab_environments
from aftab import aftab_seeds

for environment in aftab_environments:
    agent = Aftab()
    for seed in aftab_seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

A comprehensive set of Atari environments is available via EnvPool:
https://envpool.readthedocs.io/en/latest/env/atari.html#available-tasks

Procgen environments use their native RGB observations with shape (3, 64, 64). Aftab reads each task's EnvPool configuration and only applies supported options. Atari-only options such as noop, frame_skip, frame_stack, train_episodic_life, and EnvPool reward clipping are therefore not passed to Procgen.

A comprehensive set of Procgen environments is available via EnvPool:

https://envpool.readthedocs.io/en/latest/env/procgen.html#available-tasks

Hardware

Nvidia A40 GPUs were used to run all the experiments in this experiment.

Citation

Repository:

@software{aftab2026,
  author = {Taha Shieenavaz},
  title = {Aftab: A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub Repository},
  howpublished = {\url{https://github.com/tahashieenavaz/aftab}},
}

Preprint:

@misc{2608.07335,
  Author = {Taha Shieenavaz and Shabnam Zareshahraki and Loris Nanni},
  Title = {Aftab: A Progressive Design Study of Visual Encoders and Value Estimation for Replay-Free Parallelized Q-Learning},
  Year = {2026},
  Eprint = {arXiv:2608.07335},
}

Related Works

@misc{2407.04811,
  Title = {Simplifying Deep Temporal Difference Learning},
  Author = {Matteo Gallici and Mattie Fellows and Benjamin Ellis and Bartomeu Pou and Ivan Masmitja and Jakob Nicolaus Foerster and Mario Martin},
  Year = {2024},
  Eprint = {arXiv:2407.04811},
}

@misc{2403.03950,
  Title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
  Author = {Jesse Farebrother and Jordi Orbay and Quan Vuong and Adrien Ali Taïga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
  Year = {2024},
  Eprint = {arXiv:2403.03950},
}

@misc{1511.06581,
  Title = {Dueling Network Architectures for Deep Reinforcement Learning},
  Author = {Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas},
  Year = {2015},
  Eprint = {arXiv:1511.06581},
}

@misc{1806.04613,
  Title = {Improving Regression Performance with Distributional Losses},
  Author = {Ehsan Imani and Martha White},
  Year = {2018},
  Eprint = {arXiv:1806.04613},
}

@misc{1602.04621,
  Title = {Deep Exploration via Bootstrapped DQN},
  Author = {Ian Osband and Charles Blundell and Alexander Pritzel and Benjamin Van Roy},
  Year = {2016},
  Eprint = {arXiv:1602.04621},
}

Useful Links

• Wikipedia: Reinforcement Learning (RL)
• Wikipedia: Deep Reinforcement Learning (DRL)
• Wikipedia: Q-Learning
• Wikipedia: PyTorch
• Wikipedia: Statistical Hypothesis Test
• Wikipedia: Wilcoxon Signed-Rank Test
• PyTorch

Font

The Vazirmatn font is used for both Persian and English text throughout the GitHub repository header and the project's landing page.

GitHub | Google Fonts

Translations

Translations were generated by AI using GPT-5.6 Sol @ Wednesday August 19th, ~14:00.

License

© 2025 Taha Shieenavaz.
Licensed under CC BY-NC 4.0: https://creativecommons.org/licenses/by-nc/4.0/