additive-recurrent-unit

🧠 ARU: Additive Recurrent Unit

"Why do we force our models to forget in order to learn?"

ARU is a next-generation Recurrent Neural Network architecture that challenges the status quo of gated sequence modeling. It decouples memory retention from information injection, achieving true additive accumulation—a feat that standard GRUs and LSTMs struggle to perform mathematically.

---

🎓 Origin Story: Born from the Classroom

The concept of ARU wasn't born in a corporate lab, but from a moment of clarity during a Natural Language Processing (NLP) course.

While studying the evolution of sequence models, from the vanishing gradients of Vanilla RNNs to the complex gating of LSTMs and GRUs, a fundamental architectural flaw became apparent. I realized that existing architectures like the GRU enforce a zero-sum game: to let new information in (update), they mathematically force old information out (decay).

This observation sparked a question: Can we design a unit that can remember forever and learn continuously?

The answer is ARU.

---

⚡ The Theory: The Architecture of Addition

ARU introduces a Three-Gate Architecture that replaces the "convex combination" (weighted averaging) of GRUs with independent control mechanisms.

The Problem with GRUs

The standard GRU update rule is:

$$ h_t = (1 - z_t) h_{t-1} + z_t \tilde{h}_t $$

• The Trap: If $z_t \approx 1$ (learn new), then $(1-z_t) \approx 0$ (forget old). You cannot have both.

The ARU Solution

ARU uses a pure additive update, controlled by three independent gates:

$$ \Large h_t = \rho_t \odot (\pi_t \odot h_{t-1} + \alpha_t \odot v_t) $$

Meet the Gates

This decoupling allows ARU to perform Counting ($h_t = h_{t-1} + 1$) and Copying ($h_t = h_{t-1}$) perfectly, without the mathematical decay inherent in other models.

---

🧪 The Evidence: Benchmark Phases

We didn't just build it; we proved it. The ARU has been subjected to 7 phases of rigorous testing against industry-standard baselines.

tl;dr: ARU matches GRU on standard tasks but crushes it on tasks requiring long-term memory, precise counting, continuous integration, hierarchical temporal reasoning, and long-term time series forecasting.

---

📦 Installation

git clone https://github.com/abderahmane-ai/additive-recurrent-unit.git
cd additive-recurrent-unit
pip install torch rich

💻 Usage

ARU is a drop-in replacement for standard PyTorch RNNs. It is fully JIT-compiled for speed.

import torch
from aru.model import ARU

# 1. Initialize the Masterpiece
model = ARU(
    input_size=128,
    hidden_size=256,
    num_classes=10,       # Optional: Built-in classifier
    use_embedding=False   # Set True for NLP
).cuda()

# 2. Run the Forward Pass
# Shape: (Batch, Sequence Length, Features)
x = torch.randn(32, 100, 128).cuda()

# Returns: (Batch, Num Classes) if classifier is used
# Or:      (Batch, Hidden Size) if just encoder
output = model(x)

🚀 Reproducing Results

Science needs verification. You can reproduce any phase of our benchmarks by running the scripts directly as modules:

# Phase 1: Classification
python -m benchmarks.phase1.ag_news_benchmark

# Phase 2: The Memory Test
python -m benchmarks.phase2.copy_task_benchmark

# Phase 3: Counting
python -m benchmarks.phase3.counting_benchmark

# Phase 4: Sparse Events
python -m benchmarks.phase4.sparse_event_counting_benchmark

# Phase 5: Adding Problem
python -m benchmarks.phase5.adding_problem_benchmark

# Phase 6: Movie-Inspired Benchmarks
python -m benchmarks.phase6.matrix_bullet_dodge
python -m benchmarks.phase6.inception_dream_layers

# Phase 7: Time Series Forecasting
python -m benchmarks.phase7.ett_benchmark

📂 Structure

ARU/
├── aru/                 # Core ARU implementation (JIT optimized)
├── benchmarks/          # The 5 phases of truth (Copy, Counting, etc.)
└── utils/               # Training loops, data loaders, and stats

🖊️ Citation

If you use ARU in your research or project, please cite:

@misc{aru2025,
  author = {Abderahmane Ainouche},
  title = {ARU: The Additive Recurrent Unit},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/abderahmane-ai/additive-recurrent-unit}}
}

📜 License

Distributed under the MIT License. Go forth and add.