Draft of noaa-oisst recipe

[rabernat]

closes #19

This is the recipe used in the tutorial.

The point of this PR is to get the ball rolling on the "steel thread" identified by @CiaranEvans in pangeo-forge/roadmap#9 (comment).

This is a very simple recipe that is already fully supported by the pangeo_forge package in its current form. So working through this can help us discuss questions around the infrastructure and automation

• What files are required in a new recipe PR? What should they contain? (see Define contribution process pangeo-forge-recipes#71). I have included two files: recipe.py and meta.yaml. This is purely a brainstorming exercise at this point.
• What sort of testing and validation should happen automatically as part of a PR? (see Need for common vocabulary/visibility of work related to high-level concepts roadmap#9 (comment))
• How do we specify which bakery this recipe should be routed to? (see Bakery-specific datasets pangeo-forge-recipes#68)
• What happens once the PR is accepted and merged?

[TomAugspurger]

Feel free to ping me if I become a blocker anywhere here. I feel like a few pieces of the original CI structure are still useful:

1. General linting of recipes (static errors with flake8, style / formatting)
2. pangeo-forge-specific linting (currently at

   staged-recipes/.github/workflows/main.yaml

   Lines 23 to 28 in 0d6fa51

   	- name: Lint recipes
   	shell: bash -l {0}
   	run: |
   	for dir in recipes/*; do
   	pangeo-forge check ${dir} --verbose
   	done;

   , needs to be updated for the new structure)
3. Setting up new repositories on merge: https://github.com/pangeo-forge/staged-recipes/blob/master/.github/workflows/create-repository.yaml. Including registering a new pipeline with prefect: https://github.com/pangeo-forge/staged-recipes/blob/0d6fa51a1189a7bdb1927e7987c7a74216d13ab2/.github/workflows/scripts/register_pipeline.yaml

The interaction with prefect will need to be updated. Previously the recipes were prefect Flows.

The current prefect instance running in the GCP pangeo kubernetes cluster probably shouldn't be used anymore. I'd recommend starting fresh.

[TomAugspurger]

This could perhaps be a top-level field. I believe conda-forge puts it in the extra field since they're building on the existing conda meta.yaml, which didn't include maintainers. In this case we're defining the schema for the recipe meta.yaml.

[rabernat]

I basically just made all of this up. What process should we follow to design the meta.yaml file?

[rabernat]

Bakery section

bakery: aws-1

[rabernat]

Resources section

resources:
   memory: 10 GB

[rabernat]

Release tag, version, etc.

[TomAugspurger]

We probably want a top-level version key indicating the version of the pangeo-forge metadata file itself, it case we want to evolve it over time.

[rabernat]

Also perhaps a pangeo_forge library version. If the API changes over time, it will be important to pin the recipe to a specific version of the library.

[rabernat]

I have put together a document heading towards a spec for meta.yaml: https://hackmd.io/@rabernat/rkOsDk6QO/edit

Q: where should this live?