abtk

ABTK - A/B Testing Toolkit

ABTK is a comprehensive Python library for statistical analysis of A/B tests. It provides a unified interface for parametric and nonparametric hypothesis tests, variance reduction techniques, and multiple comparisons correction.

Key Features

• 12 Statistical Tests - Parametric (T-Test, Z-Test, CUPED, ANCOVA) and Nonparametric (Bootstrap)
• Cluster-Randomized Experiments - Geo experiments, store tests, market-level randomization
• Simulation-Based Power Analysis - PowerAnalyzer for experiment planning (NEW in v0.4.0)
• Variance Reduction - CUPED, ANCOVA with multiple covariates
• Multiple Comparisons - Bonferroni, Holm, Benjamini-Hochberg, and more
• Quantile Analysis - Analyze treatment effects across the distribution
• Unified Interface - All tests return standardized TestResult objects
• Automatic Diagnostics - ICC, design effect, VIF, and more
• Flexible - Support for relative and absolute effects

Quick Start

Installation

pip install abtk

For visualization support:

pip install abtk[viz]

Basic Example

from core.data_types import SampleData
from tests.parametric import TTest

# Prepare your data
control = SampleData(
    data=[100, 110, 95, 105, 98, 102],
    name="Control"
)

treatment = SampleData(
    data=[105, 115, 100, 110, 103, 108],
    name="Treatment"
)

# Run test
test = TTest(alpha=0.05, test_type="relative")
results = test.compare([control, treatment])

# Get results
result = results[0]
print(f"Effect: {result.effect:.2%}")           # e.g., "Effect: 5.5%"
print(f"P-value: {result.pvalue:.4f}")
print(f"Significant: {result.reject}")          # True/False
print(f"95% CI: [{result.left_bound:.2%}, {result.right_bound:.2%}]")

Using pandas DataFrames

For most analysts, data starts in a pandas DataFrame. Use the helper utilities for easy conversion:

import pandas as pd
from utils.dataframe_helpers import sample_data_from_dataframe
from tests.parametric import TTest

# Your experiment data as DataFrame
df = pd.DataFrame({
    'variant': ['control', 'control', 'treatment', 'treatment'],
    'revenue': [100, 110, 105, 115],
    'baseline_revenue': [90, 100, 92, 102]  # Optional: for CUPED
})

# Convert to SampleData objects automatically
samples = sample_data_from_dataframe(
    df,
    group_col='variant',
    metric_col='revenue',
    covariate_cols='baseline_revenue'  # Optional
)

# Run test
test = TTest(alpha=0.05, test_type="relative")
results = test.compare(samples)

See the DataFrame Usage Examples for more.

Available Tests

Parametric Tests

Nonparametric Tests

Cluster-Randomized Tests (NEW in v0.3.0)

For experiments where randomization occurs at group level (cities, stores, schools, etc.) rather than individual level.

Key Features:

• Accounts for within-cluster correlation (ICC)
• Provides design effect and effective sample size
• Validates cluster structure (minimum clusters, balance)
• All tests return comprehensive cluster diagnostics

See Cluster Experiments Guide for details.

Documentation

• Getting Started - Installation, first steps, basic concepts
• Test Selection Guide - How to choose the right test
• User Guides - Detailed guides for each test type
• API Reference - Complete API documentation
• Examples - Real-world use cases
• FAQ - Frequently asked questions

Example: Variance Reduction with CUPED

from tests.parametric import CupedTTest

# Include historical data for variance reduction
control = SampleData(
    data=[100, 110, 95, 105],           # Current metric
    covariates=[90, 100, 85, 95],       # Historical baseline
    name="Control"
)

treatment = SampleData(
    data=[105, 115, 100, 110],
    covariates=[92, 102, 87, 97],
    name="Treatment"
)

test = CupedTTest(alpha=0.05)
results = test.compare([control, treatment])

# CUPED typically gives narrower CI and lower p-value!

Example: Cluster-Randomized Experiment (NEW in v0.3.0)

For geo experiments where cities/regions are randomized:

from tests.parametric import ClusteredTTest

# Geo experiment: cities randomized to control/treatment
control = SampleData(
    data=[100, 110, 95, 105, 98, 102, 108, 92],  # User metrics
    clusters=[1, 1, 2, 2, 3, 3, 4, 4],            # City IDs (2 users per city)
    name="Control Cities"
)

treatment = SampleData(
    data=[105, 115, 100, 110, 103, 108, 113, 97],
    clusters=[5, 5, 6, 6, 7, 7, 8, 8],
    name="Treatment Cities"
)

test = ClusteredTTest(alpha=0.05, min_clusters=4)
results = test.compare([control, treatment])

result = results[0]
# Access cluster diagnostics
print(f"ICC: {result.method_params['icc_control']:.3f}")
print(f"Design Effect: {result.method_params['design_effect_control']:.2f}")
print(f"Effective N: {result.method_params['effective_n_control']:.0f}")

See Cluster Experiments Guide for complete workflows.

Example: Multiple Comparisons Correction

from tests.parametric import TTest
from utils.corrections import adjust_pvalues

# Test multiple variants
test = TTest(alpha=0.05)
results = test.compare([control, treatment_a, treatment_b, treatment_c])

# Apply Bonferroni correction
adjusted = adjust_pvalues(results, method="bonferroni")

for r in adjusted:
    print(f"{r.name_1} vs {r.name_2}: p={r.pvalue:.4f}, significant={r.reject}")

Example: Quantile Analysis

from tests.nonparametric import BootstrapTest
from utils.quantile_analysis import QuantileAnalyzer

# Analyze treatment effects at different quantiles
bootstrap = BootstrapTest(alpha=0.05, n_samples=10000)
analyzer = QuantileAnalyzer(
    test=bootstrap,
    quantiles=[0.25, 0.5, 0.75, 0.9, 0.95]
)

results = analyzer.compare([control, treatment])
result = results[0]

# View results as table
print(result.to_dataframe())

# Find where effects are significant
sig_quantiles = result.significant_quantiles()
print(f"Effects significant at: {sig_quantiles}")

Experiment Planning

Analytical Approach (Fast)

For quick TTest/ZTest/CUPED planning, use analytical formulas:

from utils.sample_size_calculator import calculate_mde_ttest, calculate_sample_size_ttest

# Question 1: What can I detect with 1000 users?
mde = calculate_mde_ttest(mean=100, std=20, n=1000)
print(f"Can detect {mde:.2%} effect")  # 3.5%

# Question 2: How many users to detect 5% effect?
n = calculate_sample_size_ttest(mean=100, std=20, mde=0.05)
print(f"Need {n:,} users per group")  # 1,571

Simulation-Based Power Analysis (NEW in v0.4.0)

For Bootstrap tests and cluster experiments, use PowerAnalyzer - the ONLY way to get MDE when no analytical formula exists:

from core.data_types import SampleData
from tests.nonparametric import BootstrapTest
from utils.power_analysis import PowerAnalyzer
import numpy as np

# Historical data (NOT split)
historical = SampleData(data=last_month_revenue)

# Configure test
test = BootstrapTest(alpha=0.05, n_bootstrap=500, test_type="relative")

# Create power analyzer
analyzer = PowerAnalyzer(test=test, n_simulations=200, seed=42)

# Calculate MDE for 80% power (ONLY option for Bootstrap!)
mde = analyzer.minimum_detectable_effect(
    sample=historical,
    target_power=0.8,
    effect_type="multiplicative"
)
print(f"MDE: {mde:.1%}")  # e.g., "MDE: 8.5%"

# Or estimate power for planned effect
power = analyzer.power_analysis(
    sample=historical,
    effect=0.05,  # 5% increase
    effect_type="multiplicative"
)
print(f"Power: {power:.1%}")  # e.g., "Power: 65.3%"

Key advantages of PowerAnalyzer:

• ✅ Works with ALL 12 tests (including Bootstrap, ClusteredBootstrap)
• ✅ ONLY way to get MDE for Bootstrap tests (no analytical formula exists)
• ✅ Automatic data splitting (handles simple/cluster/paired designs)
• ✅ Empirical power estimates (no distributional assumptions)

When to use:

See Experiment Planning Guide for complete details.

Test Selection Decision Tree

Is randomization at GROUP level (cities, stores)?
├─ Yes (Cluster-Randomized)
│   ├─ Proportions (CTR, CVR)? → ClusteredZTest
│   └─ Continuous metric
│       ├─ Have covariates? → ClusteredAncovaTest
│       ├─ Assume normality? → ClusteredTTest
│       └─ No assumptions → ClusteredBootstrapTest
└─ No (Individual-Level Randomization)
    ├─ Do you have proportions (CTR, CVR)?
    │   └─ Yes → ZTest
    └─ No (continuous metric)
        └─ Do you have paired data?
            ├─ Yes
            │   ├─ Assume normality? → PairedTTest
            │   └─ No assumptions → PairedBootstrapTest
            └─ No (independent samples)
                └─ Do you have covariates?
                    ├─ Yes
                    │   ├─ Multiple covariates? → AncovaTest (or OLSTest)
                    │   ├─ One covariate, normal → CupedTTest
                    │   └─ One covariate, non-parametric → PostNormedBootstrapTest
                    └─ No covariates
                        ├─ Assume normality? → TTest
                        └─ No assumptions → BootstrapTest

See Test Selection Guide for detailed recommendations.

Requirements

• Python >= 3.8
• numpy >= 1.20.0
• scipy >= 1.7.0
• pandas >= 1.3.0
• statsmodels >= 0.13.0

Optional:

• matplotlib >= 3.5.0 (for visualization)

Installation Options

For Users

# Basic installation from PyPI
pip install abtk

# With visualization support
pip install abtk[viz]

For Developers

If you want to contribute or modify the code:

# Clone the repository
git clone https://github.com/alexeiveselov92/abtk.git
cd abtk

# Install in editable mode with development dependencies
pip install -e ".[dev]"

# Or with visualization support
pip install -e ".[dev,viz]"

Running Tests

# Run all tests
pytest unit_tests/

# Run with coverage
pytest --cov=. unit_tests/

# Run specific test file
pytest unit_tests/test_ancova.py

Project Structure

abtk/
├── core/                    # Core data structures
│   ├── data_types.py        # SampleData, ProportionData
│   ├── test_result.py       # TestResult
│   └── quantile_test_result.py
├── tests/
│   ├── parametric/          # Parametric tests
│   └── nonparametric/       # Nonparametric tests
├── utils/                   # Utilities
│   ├── corrections.py       # Multiple comparisons
│   ├── quantile_analysis.py # Quantile treatment effects
│   └── visualization.py     # Plotting (optional)
├── unit_tests/              # Unit tests
├── examples/                # Runnable examples
└── docs/                    # Documentation

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Citation

If you use ABTK in your research, please cite:

@software{abtk2025,
  title={ABTK: A/B Testing Toolkit},
  author={Alexei Veselov},
  year={2025},
  url={https://github.com/alexeiveselov92/abtk}
}

Support

• Documentation: docs/
• Issues: GitHub Issues
• FAQ: docs/faq.md

Acknowledgments

ABTK implements methods from:

• Deng et al. (2013) - CUPED methodology
• Benjamini & Hochberg (1995) - FDR control
• And many other contributions to A/B testing literature

---

Ready to get started? → Check out the Getting Started Guide