access-moppy

ACCESS-MOPPy (Model Output Post-processor in Python)

ACCESS-MOPPy is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets.

Watch the quick start (10 min)

CMORising ACCESS output for CMIP7 FastTrack — a quick start with ACCESS-MOPPy — a narrated walkthrough of the whole NCI Gadi workflow, from loading the analysis3 environment to watching the batch run. It follows the quick start guide step for step.

[!NOTE] The narration voice in the video is synthetic and the script was drafted with AI assistance from this documentation. The workflow it shows is the real, maintained one.

Key Features

• Python API for integration into notebooks and scripts
• Batch processing system for HPC environments with PBS
• Real-time monitoring with web-based dashboard
• Flexible CMORisation of individual variables
• Dask-enabled for scalable parallel processing
• Cross-platform compatibility (not limited to NCI Gadi)
• CMIP6 and CMIP7 FastTrack support
• Publication QC: physical-range checks, the WCRP compliance checker, and diagnostic plots — see below

From native ACCESS output to publishable CMIP7

For CMIP7 Fast Track, ACCESS-MOPPy takes raw ACCESS model output — UM fields files, MOM and CICE history — and produces files that are ready to publish: CMORised, checked against physical ranges, validated against the CMIP controlled vocabularies, and plotted for a human to look at. The four stages run in that order, from a notebook or from a batch run on NCI Gadi.

Reads ACCESS-ESM1.6 atmosphere, land, ocean and sea-ice output directly and writes CMIP7 files — branded variable names, CMIP7 global attributes, DRS paths and file names. No CMOR library: the rewrite is built on xarray and dask, so the same code runs in a notebook or across hundreds of PBS jobs.

→ Fast Track quick start · baseline runs · batch processing

Every CMIP7 file written is checked against a per-variable physical envelope — 293 ACCESS-ESM1-6 variables, with experiment-specific overrides — plus units, missing-value and finite-value checks. The bounds are broad on purpose: they catch a unit, sign or conversion error without rejecting a plausible extreme.

The rules themselves are data, and you can read them without touching a file:

moppy-qc --show-ranges --variable tas --variable pr --experiment piControl

variable  units       min  max  rule
--------  ----------  ---  ---  ---------
tas       K           180  325  piControl
pr        kg m-2 s-1  0    0.1  default

Add --format json for the machine-readable form, ready to pipe into jq or attach to a data-quality record.

→ Every rule, rendered and filterable · running the checks

Runs the CF suite (cf:1.11) and the WCRP CMIP suite (wcrp_cmip7:1.0, backed by esgvoc) on the first file each variable publishes — metadata, controlled-vocabulary values, DRS path and file name. A failure stops that variable before any further file is written, and the JSON report is kept either way. One line of batch config turns it on:

compliance_check: true

→ Enabling it in a batch run · checker backends

Two PNGs per output file: a spatial snapshot of the first timestep, and a timeseries of the global mean with min/max shading and standard deviation. A published ACCESS-ESM1-5 CMIP6 series can be overlaid on the timeseries, so drift against the previous submission is visible at a glance.

moppy-qc-plots /scratch/cmor_output/CMIP7 --comparison-store /g/data/cmip6_store

→ Plots from a batch run · regenerating them

[!NOTE] Stages 2 and 4 run inside the CMORisation job, and stage 3 is one line of batch configuration. The batch report (moppy_batch_report_<UTC>.json) collects the results of all three, so a whole experiment can be signed off from a single file.

Installation

ACCESS-MOPPy requires Python >= 3.11.

On NCI Gadi (recommended for ACCESS users)

The conda/analysis3-latest environment maintained by ACCESS-NRI already includes access_moppy and its dependencies, so no pip install is needed:

module use /g/data/xp65/public/modules
module load conda/analysis3-latest

All command-line tools (moppy-cmorise, moppy-tui, moppy-qc, …) are available immediately after loading the module. You'll need membership of the xp65 NCI project for the module itself, plus whichever projects hold the model output and CV/table data you're processing. Pin a dated release (e.g. conda/analysis3-26.04) instead of -latest if you need a reproducible environment for a production run.

From PyPI

pip install access_moppy

From source

The controlled vocabularies under src/access_moppy/vocabularies/ are pulled in as git submodules. If you install from a local clone, initialise them first, otherwise the CMOR tables/CVs will be missing and imports will fail with an error like No module named 'access_moppy.vocabularies.CMIP6_CVs':

git clone --recurse-submodules https://github.com/ACCESS-NRI/ACCESS-MOPPy.git
cd ACCESS-MOPPy
pip install .

If you already have a clone without the submodules populated, run:

git submodule update --init --recursive
pip install .

Quick Start

New to CMORisation on Gadi? Watch the 10 minute walkthrough first.

Interactive Usage (Python API)

import glob
from access_moppy import ACCESS_ESM_CMORiser

# Select input files
files = glob.glob("/path/to/model/output/*mon.nc")

# Create CMORiser instance
cmoriser = ACCESS_ESM_CMORiser(
    input_data=files,
    compound_name="Amon.pr",  # table.variable format
    experiment_id="historical",
    source_id="ACCESS-ESM1-5",
    variant_label="r1i1p1f1",
    grid_label="gn",
    activity_id="CMIP",
    output_path="/path/to/output"
)

# Run CMORisation
cmoriser.run()
cmoriser.write()

Batch Processing (HPC/PBS)

For large-scale processing on HPC systems:

1. Create a configuration file (batch_config.yml):

variables:
  - Amon.pr
  - Omon.tos
  - Amon.ts

experiment_id: piControl
source_id: ACCESS-ESM1-5
variant_label: r1i1p1f1
grid_label: gn

input_folder: "/g/data/project/model/output"
output_folder: "/scratch/project/cmor_output"

file_patterns:
  Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"
  Omon.tos: "output[0-4][0-9][0-9]/ocean/*temp*.nc"
  Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/*mon.nc"

# PBS configuration
queue: normal
cpus_per_node: 16
mem: 32GB
walltime: "02:00:00"
scheduler_options: "#PBS -P your_project"
storage: "gdata/project+scratch/project"

worker_init: |
  module load conda
  conda activate your_environment

2. Submit batch job:

moppy-cmorise batch_config.yml

3. Monitor progress at http://localhost:8501

Batch Processing Features

The batch processing system provides:

• Parallel execution: Each variable processed as a separate PBS job
• Real-time monitoring: Web dashboard showing job status and progress
• Automatic tracking: SQLite database maintains job history and status
• Error handling: Failed jobs can be easily identified and resubmitted
• Resource optimization: Configurable CPU, memory, and storage requirements
• Environment management: Automatic setup of conda/module environments

Monitoring Tools

• Streamlit Dashboard: Real-time web interface at http://localhost:8501
• Command line: Use standard PBS commands (qstat, qdel)
• Database: SQLite tracking at {output_folder}/cmor_tasks.db
• Log files: Individual stdout/stderr for each job

File Organization

work_directory/
├── batch_config.yml          # Your configuration
├── cmor_job_scripts/          # Generated PBS scripts and logs
│   ├── cmor_Amon_pr.sh       # PBS script
│   ├── cmor_Amon_pr.py       # Python processing script
│   ├── cmor_Amon_pr.out      # Job output
│   └── cmor_Amon_pr.err      # Job errors
└── output_folder/
    ├── cmor_tasks.db         # Progress tracking
    └── [CMORised files]      # Final output

Documentation

Full documentation: https://access-moppy.readthedocs.io

• Quick start: Run your first CMORisation on Gadi
• Tutorials: Notebook walkthroughs
• How-to guides: Batch processing, QC validation, ESMValTool, ILAMB
• Reference: CLI, configuration keys, QC range rules, Python API
• Example Configuration: src/access_moppy/examples/batch_config.yml

Test Data Override

Integration and end-to-end tests require an external test-data tree set via the ACCESS_MOPPY_DATA_ROOT environment variable.

• Covered tests: full CMOR integration and end-to-end real-file tests
• No fallback: test-data fixtures in tests/data/ are not used by these tests
• Requirement: ACCESS_MOPPY_DATA_ROOT must point to a valid dataset root containing output*/atmosphere/netCDF, output*/ocean, and output*/ice

Example:

export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
pixi run -e dev python -m pytest tests/integration/test_full_cmorisation.py
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py
pixi run -e dev python -m pytest tests/e2e/test_end_to_end.py

CMIP7 baseline test note:

• tests/integration/test_cmip7_baseline_cmorisation.py runs one case per CMIP7 baseline variable listed in src/access_moppy/examples/batch_config_esm1-6_cmip7_baseline.yml
• By default, this suite checks end-to-end CMORisation success (run/write/output)
• To additionally enforce WCRP compliance-checker validation for this suite, set ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1

Example with strict WCRP validation enabled:

export ACCESS_MOPPY_DATA_ROOT=/path/to/CMIP7_Test_data/esm-historical
export ACCESS_MOPPY_BASELINE_VALIDATE_WCRP=1
pixi run -e dev python -m pytest tests/integration/test_cmip7_baseline_cmorisation.py --validation-tool=wcrp

Current Status

• Stable project status: ACCESS-MOPPy is suitable for supported CMORisation workflows and ongoing production-oriented use.
• Ocean variables: Ocean variables are supported, including dedicated ocean CMORisers and resource guidance for large 3D variables.
• Variable mapping: Mapping coverage continues to be reviewed and improved for CMIP6/CMIP7 compliance.

Support

• Issues: Submit via GitHub Issues
• Questions: Contact ACCESS-NRI support
• Contributions: Welcome via Pull Requests

License

ACCESS-MOPPy is licensed under the Apache-2.0 License.