aiida-fireworks-scheduler

aiida-fireworks-scheduler

AiiDA plugin for using fireworks as the execution engine for CalcJobProcess.

The main advantage of using the FwScheduler, as provided in this plugin, compared to the standard AiiDA scheduler plugins is that it allows more flexible job placement. For example, your may be forced to submit very large jobs to the cluster (or simply such jobs goes through the queue faster!), or that the cluster has a strict limit on the number of jobs that can be in the queue. Using FwScheduler, a single allocation of the resources from the scheduler (SGE, PBSpro, SLURM etc.) can be used to run multiple AiiDA CalcJobs in serial or in parallel, depending on the user configuration. In addition, AiiDA jobs can be run along side other workflows in fireworks.

Repository contents

• .github/: Github Actions configuration
  • ci.yml: runs tests, checks test coverage and builds documentation at every new commit
  • publish-on-pypi.yml: automatically deploy git tags to PyPI - just generate a PyPI API token for your PyPI account and add it to the pypi_token secret of your github repository
• aiida_fireworks_scheduler/: The main source code of the plugin package
  • fwscheduler.py: A new FWScheduler class.
  • scripts/arlauncher.py: A special rlaunch script for launching jobs respecting the walltime limits.
  • jobs.py: Specialised AiiDAJobFirework for running AiiDA prepared jobs.
  • fworker.py: Specialised AiiDAFWorker to generate query for selecting appropriate jobs from the FireServer.
• docs/: A documentation template ready for publication on Read the Docs
• examples/: An example of how to submit a calculation using this plugin
• tests/: Basic regression tests using the pytest framework (submitting a calculation, ...). Install pip install -e .[testing] and run pytest.
• .coveragerc: Configuration of coverage.py tool reporting which lines of your plugin are covered by tests
• .gitignore: Telling git which files to ignore
• .pre-commit-config.yaml: Configuration of pre-commit hooks that sanitize coding style and check for syntax errors. Enable via pip install -e .[pre-commit] && pre-commit install
• .readthedocs.yml: Configuration of documentation build for Read the Docs
• LICENSE: License for your plugin
• MANIFEST.in: Configure non-Python files to be included for publication on PyPI
• README.md: This file
• conftest.py: Configuration of fixtures for pytest
• pytest.ini: Configuration of pytest test discovery
• setup.json: Plugin metadata for registration on PyPI and the AiiDA plugin registry (including entry points)
• setup.py: Installation script for pip / PyPI

Features

• FWScheduler scheduler plugin to submit jobs to LaunchPad managed by fireworks.

• arlaunch command for launching jobs on the cluster machine.

• verdi data fireworks-scheduler command line tool for duplicating existing Computer/Cold for switching to FwScheduler.

Installation

On the local machine where AiiDA is installed:

pip install aiida-fireworks-scheduler[local]

On the remote machine where jobs to be launched:

pip install aiida-fireworks-scheduler

Usage

Simply create a new computer using verdi computer setup and select the fw scheduler. Configure your fireworks configuration following the guide here.

Note that you must configure the LAUNCHPAD_LOC setting in the file as defined by the FW_CONFIG_FILE environment variable to point to your my_launchpad.yaml file on BOTH the local and remote machines. On the local machine, it will be picked up by the daemon.

In addition, on the remote machine, setup your my_fworker.yaml with special directives for identifying the computer and username. These files can be generated using:

verdi data fireworks-scheduler generate-worker -Y COMPUTER -mpinp NUM_MPI_PROCESSORS

Note that each *worker" can only launch jobs of a particular size (number of MPI processors). But you can always combine multiple workers in one or more cluster jobs.

At runtime, jobs needs to be launched with the arlaunch command on the remote machine.

Adding walltime selectors for standard fireworks jobs

Standard fireworks jobs can also be selected based on the requested walltime using arlaunch. If a job has spec._walltime_seconds key, it will only be selected to run if there is sufficient time left. However, unlike AiiDA jobs, this walltime limit is not enforced, and the launch can proceed even if the requested seconds have elapsed.

Development

git clone https://github.com/zhubonan/aiida-fireworks-scheduler .
cd aiida-fireworks-scheduler
pip install -e .[pre-commit,testing]  # install extra dependencies
pre-commit install  # install pre-commit hooks
pytest -v  # discover and run all tests

See the developer guide for more information.

License

MIT

Contact

zhubonan@outlook.com