98-contributing-recipes.md

Contributing Recipes

First, thank you for considering to contribute your recipe to the wider community! Below are some general guidelines to follow when contributing a recipe to Batch Shipyard.

Follow the Directory Structure of Existing Recipes

Please follow the existing recipe directory structure for the recipe that you wish to add. Each recipe directory structure typically has the following structure:

  recipes/
    SOFTWARE-NAME/
      README.md
      config/
        config.yaml
        credentials.yaml
        jobs.yaml
        pool.yaml
      docker/
        README.md
        Dockerfile
        ...

The docker directory is optional if your Docker image is an automated build on public Docker Hub, however, please read the other requirements regarding links to licenses.

Provide a Clear README File

Each recipe should contain a clear README.md file that links to the application website and steps through each configuration file (pool, global, jobs). Provide links to configuration files within the README.md file as sample configurations users can utilize to get started with your recipe.

Provide any additional pointers or thinge to look out for when running the Docker image and the application. For instance, if running a Multi-Instance task with MPI and the application expects a shared file system for output files, then explicitly mention this with an example or reference on how to set this up properly within Batch Shipyard.

Provide a Sample Execution

Each recipe should provide a sample execution to validate that Batch Shipyard works with the provided Docker image. It is preferable to embed a sample in the Docker image which you can then use during your jobs configuration text in the README.md file to explain how to launch a job.

Create Generic Configuration Files

Create a set of generic configuration files without specific links to credential names, aliases or other sensitive info. The credentials config file should not contain any pre-filled information except for a storage account endpoint. Please take a look at the pre-existing recipe sample config files to see an example.

Use YAML Syntax for Configuration Files

Configuration files should use YAML syntax rather than JSON.

Docker Image References

If you are providing your own Docker image reference in the global configuration and jobs config files, then the Dockerfile must be publically accessible. The docker image can be hosted as an automated build on public Docker Hub which will automatically upload the Dockerfile to your repository, or you will need to explicitly provide a link in Docker Hub and in the README for the recipe to the Dockerfile which can be viewable by anyone.

Ensuring transparency in the Docker images used for recipes will create a more open and better experience for all users that wish to use your recipe.

Provide Links to Licenses

You must provide links to licenses used by the application in the recipe README and the associated docker directory, if provided.

Optimize Dockerfiles

If possible, try to create Docker images with the minimal amount of layers. This involves coalescing multiple RUN or other statements together. Reducing the number of layers can potentially reduce the size of the Docker image which is of concern for image replication to compute nodes.

If you are able to completely remove the source files of your application, then it is recommended to do so after the binaries are installed on the system (e.g., through make install). Additionally, if some software is only needed to build the application and can be safely removed, then it is recommended to do so. For instance, if your software requires development headers to compile, these can be removed after the binary is produced. Similarly, temporary files as a result of the build process (e.g., .o or .dep files) can be removed to reduce the size of the final Docker image.

Multi-Instance (MPI) Recipes

Please ensure that the necessary SSH, MPI runtime, or other software is installed into the Docker image (if licensing allows) such that Multi-Instance tasks can be run on them without requiring users to import your image and add them later. Please refer to the Multi-Instance doc for more information.

There are sample recipes which showcase MPI on Batch Shipyard, please reference those as examples.

Existing Recipes

Please see this directory for existing recipes.