abpy-toolkit

abpy — Frequentist A/B Testing Library

A Python library for rigorous frequentist A/B and multivariate testing, with built-in support for CUPED variance reduction, Delta method for ratio metrics, and multiple comparison corrections.

---

Installation

pip install abpy-toolkit

---

Quick Start

from abpy.router import Experiment

# binary metric — direct input
exp = Experiment(type='ab', metric='binary', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'successes': 100},
    'treatment': {'n': 1000, 'successes': 130}
})
exp.results()
exp.plot()

---

Features

• Three metric types — binary, continuous, count
• Two input modes — CSV file or direct summary statistics
• Automatic test selection — z-test, t-test, Mann-Whitney U, Poisson rate test
• CUPED — variance reduction using pre-experiment data
• Delta method — correct variance estimation for ratio metrics
• Multiple comparison corrections — Bonferroni, Holm, Benjamini-Hochberg
• Multivariate testing — all pairwise comparisons with corrections
• Power analysis — achieved power for every test
• Dashboard visualisation — 6-panel results plot

---

API Reference

Experiment(type, metric, input, alpha, cuped, correction, is_ratio,power,alternative)

---

Usage Examples

Binary Metric — CSV Input

import pandas as pd
from abpy.router import Experiment

exp = Experiment(type='ab', metric='binary', input='csv', alpha=0.05)
exp.fit('experiment.csv')   # requires columns: variant, metric
exp.results()
exp.plot()

CSV structure:

variant,metric
control,0
control,1
treatment,1
treatment,0
...

Continuous Metric — Direct Input

exp = Experiment(type='ab', metric='continous', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'mean': 45.2, 'std': 12.3},
    'treatment': {'n': 1000, 'mean': 48.7, 'std': 11.9}
})
exp.results()

Discrete / Count Metric

exp = Experiment(type='ab', metric='discrete', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'events': 300},
    'treatment': {'n': 1000, 'events': 375}
})
exp.results()

Multivariate with Holm Correction

exp = Experiment(
    type='multivariate',
    metric='binary',
    input='direct',
    alpha=0.05,
    correction='holm'
)
exp.fit({
    'control':    {'n': 2000, 'successes': 200},
    'treatment1': {'n': 2000, 'successes': 240},
    'treatment2': {'n': 2000, 'successes': 260}
})
exp.results()

CUPED — Variance Reduction

Reduces variance using pre-experiment data, requiring fewer users to reach significance.

exp = Experiment(type='ab', metric='continous', input='csv', alpha=0.05, cuped=True)
exp.fit('experiment_with_pre.csv')
exp.results()

CSV structure for CUPED:

variant,metric,pre_metric
control,45.2,44.1
treatment,48.7,43.9
...

Ratio Metric — Delta Method

For metrics like CTR = clicks / impressions where both vary per user.

exp = Experiment(type='ab', metric='continous', input='csv', alpha=0.05, is_ratio=True)
exp.fit('ratio_experiment.csv')
exp.results()

CSV structure for ratio metrics:

variant,metric1,metric2
control,3,10
treatment,5,10
...

CSV structure for ratio metrics (if Cuped applied)

variant,metric1,metric2,pre_metric1,pre_metric2
control,3,10,2,6
treatment,5,10,3,5
...

---

Test Selection Logic

Binary metric    → z-test (pooled proportion)
Continuous
  n >= 30        → t-test (Welch's approximation)
  n < 30         → Shapiro-Wilk normality check
    normal       → t-test
    not normal   → Mann-Whitney U
  is_ratio=True  → Delta method → t-test
Discrete/Count   → Poisson rate test

---

Output

Every test returns:

Test used            which statistical test was selected
Alternative          type 
Test statistic       z, t, or U value
P-value              raw p-value
Adjusted p-value     if correction applied
Confidence interval  95% CI on the lift
Effect size          Cohen's h (binary) or d (continuous)
Power                achieved power given sample size
Required sample size The sample size for the given power to be achieved
Recommendation       "2nd variant wins" / "no significant difference" / "underpowered"

---

Corrections Reference

---

Visualisation

exp.plot()   # renders 6-panel dashboard

Dashboard panels:

1. P-value vs significance threshold
2. Confidence interval on lift
3. Effect size (Cohen's benchmark)
4. Power curve vs sample size
5. Metric distribution by variant
6. Cuped applied metric distribution by variant

---

Dependencies

numpy >= 1.21.0
scipy >= 1.7.0
pandas >= 1.3.0
matplotlib >= 3.4.0

---

License

MIT License — see LICENSE file for details.