aind-behavior-video-transformation

aind-behavior-video-transformation

This project is used to encode behavior videos and, depending on the user request, will compress those videos while ensuring that they are formatted for correct display across many devices. This may include common image preprocessing steps, such as gamma encoding, that are necessary for correct display, but have to be done post-hoc for our behavior videos.

Goals

This will attempt to compress videos so that the results:

• Retain the majority of the detail of the input video
• Take up as little space as possible for a target amount of visual detail
• Are in a format that can be widely viewed across devices, and streamed by browsers
• Have pixel data in a color space that most players can properly display

This video compression is often lossy, and the original videos are not kept, so this library will attempt to produce the highest-quality video for a target compression ratio. The speed of this compression is strictly secondary to the quality of the compression, as measured by the visual detail retained and the compression ratio. See this section for more details.

Additionally, this package should provide an easy to use interface that:

• Presents users with a curated set of compression settings, which have been rigorously tested in terms of their visual quality using perception-based metrics like VMAF.
• Allow users to also provide their own compression settings, if they have specific requirements

Non-goals

• Sacrifice the visual fidelity of videos in order to decrease encoding time.

Brief benchmarks on video compression with CPU-based encoders and GPU-based encoders

A surprising fact is that video encoders implementing the same algorithm, but written for different compute resources do not have the same visual performance; for a given compression ratio, or similar settings, they do not retain the same amount of visual detail. This is also true for different presets of the same encoder and compute resource even if the other settings are identical. For example, the presets -preset fast and -preset veryslow of the encoder libx264 produce videos with the same compression ratio, but differing visual quality.

This can be seen in the plot below, where the GPU encoder and CPU encoders retain different amounts of visual detail, as assessed with visual perception-based metric VMAF. Also note the difference between presets for the same encoder and compute resource: CPU Fast and CPU Slow.

This figure shows that for compression ratios greater than 100, it often makes sense to take your time and use a slow preset of a CPU-based encoder to retain as much visual information for a given amount of compression.

While it may be tempting to select a faster preset, or faster compute resource like GPU for dramatic speedups shown below, doing will degrade the quality of the resulting video.

Because the output of this package are permanent video artifacts, the compression is lossy, and the intent is to delete the original, taking the CPU time to produce the highest quality video possible might well be worth it.

Usage

• The BehaviorVideoJob.run_job method in the transform_videos should be the primary method to call for processing video files.
• On a merge to main, this package will be published as a docker image, which can easily be run on a SLURM cluster.

Running across multiple nodes

A job can be split across a SLURM array. Every task runs the same command and discovers the same list of files independently, then acts only on the share identified by its own partition_number, so no coordination between tasks is needed.

sbatch --array=1-4 --cpus-per-task=8 --wrap='
  python -m aind_behavior_video_transformation.etl --job-settings "{
    \"input_source\": \"/mnt/input_source\",
    \"output_directory\": \"/mnt/output_directory\",
    \"partition_number\": $SLURM_ARRAY_TASK_ID,
    \"num_partitions\": 4,
    \"ffmpeg_thread_cnt\": 8
  }"'

Notes:

• partition_number is 1-based, so use --array=1-N, not --array=0-(N-1).
• num_partitions must match the array size and be identical on every task.
• Both default to 1, which processes every video in a single task.
• Set ffmpeg_thread_cnt to match --cpus-per-task. It defaults to 0, which lets ffmpeg use every core on the physical node regardless of the allocation.

Videos and the non-video files that are symlinked alongside them are distributed separately, so each task receives a comparable share of the transcoding work rather than one task inheriting every video.

Deprecated settings

parallel_compression is deprecated. It is still accepted, so existing job settings continue to validate, but it no longer has any effect and will be removed in a future release. Compression now runs sequentially within a task, and parallelism comes from the scheduler running one task per partition. Use num_partitions and partition_number instead.

Docker build for local testing

In the same directory as the Dockerfile, run

docker build -t aind-behavior-video-transformation-local .

Now a docker container can be run. You may need to change the mount locations and file permissions:

docker run -v /home/local_videos/input_source:/mnt/input_source -v /home/local_videos/output_directory:/mnt/output_directory aind-behavior-video-transformation-local python -m aind_behavior_video_transformation.etl --job-settings '{"compression_requested": {"compression_enum": "gamma fix colorspace"}, "input_source": "/mnt/input_source", "output_directory": "/mnt/output_directory"}'

Development

To develop the code, run

pip install -e .[dev]

Contributing

Linters and testing

There are several libraries used to run linters, check documentation, and run tests.

• Please test your changes using the coverage library, which will run the tests and log a coverage report:

coverage run -m unittest discover && coverage report

• Use interrogate to check that modules, methods, etc. have been documented thoroughly:

interrogate .

• Use flake8 to check that code is up to standards (no unused imports, etc.):

flake8 .

• Use black to automatically format the code into PEP standards:

black .

• Use isort to automatically sort import statements:

isort .

Pull requests

For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:

<type>(<scope>): <short summary>

where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:

• build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
• ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
• docs: Documentation only changes
• feat: A new feature
• fix: A bugfix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• test: Adding missing tests or correcting existing tests

Semantic Release

The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):

Documentation

To generate the rst files source files for documentation, run

sphinx-apidoc -o doc_template/source/ src

Then to create the documentation HTML files, run

sphinx-build -b html doc_template/source/ doc_template/build/html

More info on sphinx installation can be found here.