aihw-bench

AIHW-Bench

AI Hardware Benchmarking & Profiling Framework

AIHW-Bench is a reproducible Python framework for benchmarking, profiling, comparing, and reporting AI workloads across CPUs, GPUs, embedded systems, simulator-backed environments, and extensible accelerator backends.

Measure the workload. Understand the hardware. Compare the results. Reproduce the experiment.

• Documentation: https://shashi065.github.io/aihw-bench/
• Repository: https://github.com/shashi065/aihw-bench
• Current stable release: v2.0.0

Why AIHW-Bench?

Architecture

Configuration + workload
        │
        ▼
Benchmark service ──► backend + hardware inspection
        │                         │
        ▼                         ▼
immutable session ◄──────── measurements / observations
        │
        ├──► metrics + statistics
        ├──► JSON / CSV / Markdown / HTML reports
        ├──► static dashboard
        └──► local benchmark analysis assistant

Install

python -m pip install aihw-bench

Optional runtime integrations are installed only when needed:

python -m pip install "aihw-bench[pytorch]"
python -m pip install "aihw-bench[onnx]"
python -m pip install "aihw-bench[all-backends]"

Quick start

Run a deterministic reference benchmark and write a JSON report:

aihw-bench benchmark --backend reference --warmup 1 --iterations 10 --report json

Then inspect your environment and generated results:

aihw-bench doctor
aihw-bench report SESSION_ID --format html --format markdown
aihw-bench dashboard --storage-root .aihw-bench/sessions --output-dir dashboard

Common workflows

Compare two sessions

aihw-bench compare BASELINE_SESSION CANDIDATE_SESSION --output table

Use the official reproducible suite

aihw-bench suite list
aihw-bench suite materialize --output-dir benchmarks
aihw-bench suite baselines --output-dir benchmarks

The suite ships deterministic synthetic inputs and reference fixtures. They validate benchmark plumbing and reproducibility; they are not universal real-device performance claims.

Explain a result locally

aihw-bench assistant SESSION_ID --storage-root .aihw-bench/sessions

The built-in assistant is a deterministic, metric-grounded local benchmark analysis assistant. It is not an LLM and does not send benchmark data to an external service.

Hardware support model

AIHW-Bench distinguishes four capability states:

This distinction matters: FPGA and RTL information can be detected or reported without implying synthesis, board programming, or vendor-tool execution. See the hardware support guide.

Backends and runtimes

Reproducibility and reporting

Every benchmark session captures resolved configuration, hardware context, execution samples, metrics, diagnostics, and artifact checksums. Built-in reporters generate JSON, CSV, Markdown, and HTML; the static dashboard supports history browsing, filtering, comparison, and export.

• Quick start
• CLI reference
• Official benchmark suite
• Dashboard
• Assistant
• API reference

Project status

AIHW-Bench v2.0.0 is feature-complete and maintained as a stable local benchmarking and reporting toolkit. Distributed remote execution, marketplace installation, real FPGA programming, and model-backed assistant providers remain extension areas rather than claims of the core package.

Development

python -m pip install poetry
poetry install --with dev,docs
poetry run pytest
poetry run ruff check src tests scripts
poetry run black --check src tests scripts
poetry run mypy src
poetry run mkdocs build --strict

See the contribution guide, security policy, and engineering documentation.

License

AIHW-Bench is licensed under the Apache License 2.0.