agi-transformer

AGi Transformer v5.1 — Advanced Core

High-performance transformer library with FP4-E2M1 quantization, Parallel Blocks, Mamba layers, and Triton double-buffering kernels.

Features

• FP4-E2M1 Format: 1 sign + 2 exponent + 1 mantissa (superior to NF4 for Gaussian weights)
• Parallel Block Architecture: PaLM-style simultaneous Attention + FFN (15% speedup)
• Mamba/TTT Hybrid: Replace middle attention with O(L) linear recurrent layers
• Double Buffering: Triton kernel preloads Block K+1 while computing Block K
• Matrix Swizzling: XOR-based shared memory indexing eliminates bank conflicts
• Dynamic Scaled Clamping: Per-row adaptive bounds based on block maxima
• Vector SmoothAlpha: Per-channel 4096-parameter scaling (not scalar)
• AdaRound + KLD: Learnable rounding with distribution-preserving loss

Quick Start

pip install -e .
python scripts/train.py --steps 5000 --d-model 512 --n-layers 8

Project Structure

src/agi_core/
├── kernels/      # Triton AOT kernels (FP4, Double Buffer, Swizzle)
├── layers/       # Quantized Linear, Mamba, Parallel Block
├── model/        # AGiParallelTransformer
├── ops/csrc/     # C++/CUDA extensions (optional AOT)
└── utils/        # Quantization helpers, logging

License

MIT