absconda

Absconda

Turn conda environments into optimized container images for development and HPC deployment.

Absconda bridges the gap between conda's reproducible environments and container-based workflows. Define your scientific computing environment once with conda, then deploy it anywhere—Docker, Singularity/Apptainer, HPC clusters—with production-ready optimizations.

Key Features

🚀 Multi-Stage Builds - Automatic optimization that reduces image sizes by 40-60%
🔐 Policy Validation - Enforce security and compliance rules organization-wide
🏗️ Remote Builders - Offload builds to cloud instances with automatic provisioning
🧪 HPC Integration - First-class Singularity support with module files and wrappers
📦 R + renv Support - Combine conda environments with R package management
🎯 Flexible Deployment - Multiple modes: full-env, tarball, requirements, export-explicit
🔧 Custom Templates - Jinja2-based system for advanced customization

Quick Start

Installation

# Install from PyPI
pip install absconda

# Or with pipx (recommended)
pipx install absconda

# Verify installation
absconda --version

Basic Usage

1. Create a conda environment file:

# environment.yaml
name: my-analysis
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.11
  - numpy=1.26
  - pandas=2.1
  - scikit-learn=1.3

2. Build a Docker image:

absconda build \
  --file environment.yaml \
  --repository ghcr.io/myorg/my-analysis \
  --tag latest \
  --push

3. Use the image:

docker run --rm ghcr.io/myorg/my-analysis:latest python -c "import numpy; print(numpy.__version__)"

For HPC with Singularity:

# Build Docker image and convert to Singularity
absconda build \
  --file environment.yaml \
  --repository ghcr.io/myorg/my-analysis \
  --tag latest \
  --push

# Convert to Singularity
absconda singularity \
  --image ghcr.io/myorg/my-analysis:latest \
  --output my-analysis.sif

# Generate HPC module
absconda module \
  --image my-analysis.sif \
  --module-path /apps/modules/my-analysis/1.0

See the Quick Start Guide for a complete walkthrough.

Documentation

For New Users

• Installation - Get up and running
• Quick Start - 5-minute tutorial
• Core Concepts - Understanding absconda

Guides

• Basic Usage - Essential workflows
• Building Images - Docker build process
• HPC Deployment - Singularity on HPC clusters
• Remote Builders - Cloud-based builds
• R + renv Integration - Combining conda and renv
• Requirements Mode - Deployment mode comparison
• Advanced Templating - Custom Dockerfiles

How-To Guides

• Multi-Stage Builds - Optimize image size
• Custom Base Images - GPU and specialized bases
• Secrets Management - Handle credentials safely
• CI/CD Integration - Automate with GitHub Actions

Examples

Complete working examples with explanations:

• Minimal Python - Simple Python environment
• Data Science Stack - NumPy/pandas/scikit-learn
• R + Bioconductor - RNA-seq analysis
• GPU + PyTorch - Deep learning with CUDA
• HPC Workflow - Complete HPC deployment

Reference

• CLI Reference - Complete command documentation
• Environment Files - YAML specification
• Configuration - System configuration
• Policies - Policy validation system

For Contributors

• Contributing - How to contribute
• Testing - Running and writing tests
• Architecture - Technical design documentation

Why Absconda?

The Problem

Scientific computing has conflicting requirements:

• Reproducibility: Need exact package versions
• Portability: Must run on laptops, HPC clusters, cloud
• Performance: Large conda environments create huge containers
• HPC Reality: Singularity/Apptainer, not Docker; module systems, not Docker Compose

The Solution

Absconda solves this by:

1. Starting with conda - Use the ecosystem you already know
2. Optimizing automatically - Multi-stage builds reduce image sizes by 40-60%
3. Targeting HPC - Native Singularity support with modules and wrappers
4. Enforcing policies - Organization-wide security and compliance
5. Enabling remote builds - Build on powerful cloud instances, not your laptop

Compared to Alternatives

Real-World Example

Complete workflow for deploying to NCI Gadi HPC:

# 1. Build optimized Docker image (locally or on GCP)
absconda build \
  --file rnaseq-env.yaml \
  --repository ghcr.io/lab/rnaseq \
  --tag v1.0 \
  --remote-builder gcp-builder \
  --push

# 2. Convert to Singularity on HPC
ssh gadi.nci.org.au
singularity pull rnaseq.sif docker://ghcr.io/lab/rnaseq:v1.0

# 3. Generate module file
absconda module \
  --image /apps/rnaseq/v1.0/rnaseq.sif \
  --module-path /apps/Modules/modulefiles/rnaseq/1.0 \
  --wrapper-dir /apps/rnaseq/v1.0/wrappers

# 4. Use in PBS job
module load rnaseq/1.0
python analysis.py  # Uses containerized environment transparently

The result: reproducible, optimized, compliant environments deployed consistently from development through production.

Use Cases

🔬 Research Computing

• Reproducible analysis pipelines
• Sharing environments with collaborators
• Publishing with computational papers
• Archive environments for long-term reproducibility

🏢 Multi-User HPC

• Centralized environment management
• Policy enforcement across teams
• Module system integration
• Singularity deployment at scale

☁️ Cloud + HPC Hybrid

• Build in cloud (fast, powerful VMs)
• Deploy to HPC (Singularity)
• Consistent environments across platforms
• Cost-optimized with remote builders

🧬 Bioinformatics

• Bioconda + R/Bioconductor workflows
• GPU-accelerated analysis
• Large-scale genomics pipelines
• Compliance with data policies

Project Status

Absconda is production-ready and actively maintained. It powers scientific computing workflows for research teams at the Garvan Institute and beyond.

Current version: 0.1.0
Python support: 3.10, 3.11, 3.12
License: MIT

Getting Help

• Documentation: docs/
• Issues: GitHub Issues
• Discussions: GitHub Discussions

Contributing

Contributions welcome! See Contributing Guide.

Areas needing help:

• Documentation improvements
• Example workflows
• Testing on different platforms
• Feature requests and feedback

Acknowledgments

Developed at the Garvan Institute of Medical Research by the Swarbrick Lab.

Built with:

• micromamba - Fast conda package manager
• Jinja2 - Template engine
• Singularity/Apptainer - HPC containers
• Terraform - Infrastructure as code

License

MIT License - see LICENSE for details.

---

Ready to get started? → Installation Guide

Have questions? → GitHub Discussions