aind-airflow-jobs

aind-airflow-jobs

Usage

• This library contains code and classes that can be used globally for aind-airflow-service
• Primarily used for submitting jobs to the HPC
• The Docker image can be used by Airflow KubernetesPodOperators in the aind-airflow-service to run tasks, for example:

from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

check_param_store_connection = KubernetesPodOperator(
    task_id="check_param_store_connection",
    image="ghcr.io/allenneuraldynamics/aind-airflow-jobs:latest",
    in_cluster=True,
    cmds=["python", "-m", "aind_airflow_jobs.dag_tasks.check_connections"],
    env_vars={
        "AIRFLOW_CTX_TASK_ID": "{{ task.task_id }}",
    },
)

Installation

To use the software, in the root directory, run

pip install -e .

To develop the code, run

pip install -e .[dev]

Contributing

Linters and testing

There are several libraries used to run linters, check documentation, and run tests.

• Please test your changes using the coverage library, which will run the tests and log a coverage report:

coverage run -m unittest discover && coverage report

• Use interrogate to check that modules, methods, etc. have been documented thoroughly:

interrogate .

• Use flake8 to check that code is up to standards (no unused imports, etc.):

flake8 .

• Use black to automatically format the code into PEP standards:

black .

• Use isort to automatically sort import statements:

isort .

Pull requests

For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:

<type>(<scope>): <short summary>

where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:

• build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
• ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
• docs: Documentation only changes
• feat: A new feature
• fix: A bugfix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• test: Adding missing tests or correcting existing tests

Semantic Release

The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):

Documentation

To generate the rst files source files for documentation, run

sphinx-apidoc -o doc_template/source/ src 

Then to create the documentation HTML files, run

sphinx-build -b html doc_template/source/ doc_template/build/html

More info on sphinx installation can be found here.