From 6242c1af14c99756997e729f71be973bf3fcb334 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 2 Jun 2023 18:24:08 +0200 Subject: [PATCH 01/12] add cep 20.1 for only the YAML format --- cep-20.1.md | 166 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 cep-20.1.md diff --git a/cep-20.1.md b/cep-20.1.md new file mode 100644 index 00000000..8df2e062 --- /dev/null +++ b/cep-20.1.md @@ -0,0 +1,166 @@ +# A new recipe format – part 1 + + + + + + + + + +
Title A new recipe format
Status Proposed
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Created May 23, 2023
Updated May 23, 2023
Discussion
Implementation https://github.com/prefix-dev/rattler-build
+ +## Abstract + +We propose a new recipe format that is heavily inspired by conda-build. The main change is a pure YAML format without arbitrary Jinja or comments with semantic meaning. + +## Motivation + +The conda-build format has grown over the years to become quite complex. Unfortunately it has never been formally "specified" and it has grown some features over time that make it hard to parse as straightforward YAML. + +The CEP attempts to introduce a subset of the conda build format that allows for fast parsing and building of recipes. + +### History + +A discussion was started on what a new recipe spec could or should look like. The fragments of this discussion can be found here: https://github.com/mamba-org/conda-specs/blob/master/proposed_specs/recipe.md +The reason for a new spec are: + +- Make it easier to parse ("pure yaml"). conda-build uses a mix of comments and jinja to achieve a great deal of flexibility, but it's hard to parse the recipe with a computer +- iron out some inconsistencies around multiple outputs (build vs. build/script and more) +- remove any need for recursive parsing & solving + +## Major differences with conda-build + +- no full Jinja2 support: no conditional or `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid YAML +- Jinja string interpolation needs to be quoted at the beginning of a string, e.g. `- "{{ version }}"` in order for it to be valid YAML +- Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- sel(osx): somepkg` instead of `- somepkg # [osx]` + +## Selectors + +Selectors in the new spec take the following format: + +`sel(unix): selected_value` + +This is a valid YAML dictionary. Selector contents are simple boolean expressions and follow Python syntax. The following selectors are all valid: + +``` +win and arm64 +(osx or linux) and aarch64 +something == "test" +``` + +### The cmp function for variant selection + +Furthermore, we have a special "cmp" function that can be used to run a check against a selected variant version. The `cmp` function looks like the following: + +``` +cmp(python, "3.6") +cmp(python, ">=3.6") +cmp(python, ">=3.8,<3.10") +etc +``` + +This can be used in a selector like so: + +``` +requirements: + build: + - sel(cmp(python, ">=3.6,<3.10")): dataclasses +``` + +This functionality generalizes and replaces the previous special variables such as `py2k`, `py3k`, `py36`, `py37`, and works just as well for NumPy, Ruby, R, or any other variant that might be of interest in the future. + +### Preprocessing selectors + +You can add selectors to any item, and the selector is evaluated in +a preprocessing stage. If a selector evaluates to `true`, the item is +flattened into the parent element. If a selector evaluates to `false`, +the item is removed. + +```yaml +source: + - sel(not win): + url: http://path/to/unix/source + - sel(win): + url: http://path/to/windows/source +``` + +Because the selector is a valid Jinja expression, complicated logic +is possible: + +```yaml +source: + - sel(win): + url: http://path/to/windows/source + - sel(unix and cmp(python, "2")): + url: http://path/to/python2/unix/source + - sel(unix and cmp(python, "3")): + url: http://path/to/python3/unix/source +``` + +Lists are automatically "merged" upwards, so it is possible to group multiple items under a single selector: + +```yaml +test: + commands: + - sel(unix): + - test -d ${PREFIX}/include/xtensor + - test -f ${PREFIX}/include/xtensor/xarray.hpp + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake + - sel(win): + - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) + - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfig.cmake (exit 1) + - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) + +# On unix this is rendered to: +test: + commands: + - test -d ${PREFIX}/include/xtensor + - test -f ${PREFIX}/include/xtensor/xarray.hpp + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake +``` + +## Templating with Jinja + +The spec supports simple Jinja templating in the `recipe.yaml` file. + +You can set up Jinja variables in the context YAML section: + +```yaml +context: + name: "test" + version: "5.1.2" + major_version: "{{ version.split('.')[0] }}" +``` + +Later in your `recipe.yaml` you can use these values in string interpolation +with Jinja. For example: + +```yaml +source: + url: https://github.com/mamba-org/{{ name }}/v{{ version }}.tar.gz +``` + +Jinja has built-in support for some common string manipulations. + +In the new spec, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. +So you should not use any `{% if ... %}` or similar Jinja constructs that produce invalid yaml. +Furthermore, quotes need to be applied when starting a value with double-curly brackets like so: + +```yaml +package: + name: {{ name }} # WRONG: invalid yaml + name: "{{ name }}" # correct +``` + +Some Jinja functions remain valid, but this is out of the scope of this spec. However, as an example, +the `compiler` Jinja function will still be available, with the main difference being the quoting of the +brackets. + +```yaml +requirements: + build: + - "{{ compiler('cxx') }}" +``` \ No newline at end of file From 34917a89879592bc037c94942bcbe7985a9eb477 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 2 Jun 2023 19:44:09 +0200 Subject: [PATCH 02/12] add example conversion --- cep-20.1.md | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 1 deletion(-) diff --git a/cep-20.1.md b/cep-20.1.md index 8df2e062..e60bd139 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -163,4 +163,73 @@ brackets. requirements: build: - "{{ compiler('cxx') }}" -``` \ No newline at end of file +``` + +## Examples + +### xtensor + +Original recipe [found here](https://github.com/conda-forge/xtensor-feedstock/blob/feaa4fd8015f96038168a9d67d69eaf06a36d63f/recipe/meta.yaml). + +```yaml +context: + name: xtensor + version: 0.24.6 + sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655 + +package: + name: "{{ name|lower }}" + version: "{{ version }}" + +source: + fn: "{{ name }}-{{ version }}.tar.gz" + url: https://github.com/xtensor-stack/xtensor/archive/{{ version }}.tar.gz + sha256: "{{ sha256 }}" + +build: + number: 0 + sel(win and vc<14): + skip: true + +requirements: + build: + - "{{ compiler('cxx') }}" + - cmake + - sel(unix): make + host: + - xtl >=0.7,<0.8 + run: + - xtl >=0.7,<0.8 + run_constrained: + - xsimd >=8.0.3,<10 + +test: + commands: + sel(unix): + - test -d ${PREFIX}/include/xtensor + - test -f ${PREFIX}/include/xtensor/xarray.hpp + - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake + - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake + sel(win): + - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) + - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfig.cmake (exit 1) + - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) + +about: + home: https://github.com/xtensor-stack/xtensor + license: BSD-3-Clause + license_family: BSD + license_file: LICENSE + summary: The C++ tensor algebra library + description: Multi dimensional arrays with broadcasting and lazy computing + doc_url: https://xtensor.readthedocs.io + dev_url: https://github.com/xtensor-stack/xtensor + +extra: + recipe-maintainers: + - SylvainCorlay + - JohanMabille + - wolfv + - davidbrochart +``` + From 7a22c07fb45f3618b854e4fb2904867ece2f0ab0 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 2 Jun 2023 20:08:19 +0200 Subject: [PATCH 03/12] add a little bit of text about shortcomings --- cep-20.1.md | 103 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 77 insertions(+), 26 deletions(-) diff --git a/cep-20.1.md b/cep-20.1.md index e60bd139..f836885c 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -12,28 +12,42 @@ ## Abstract -We propose a new recipe format that is heavily inspired by conda-build. The main change is a pure YAML format without arbitrary Jinja or comments with semantic meaning. +We propose a new recipe format that is heavily inspired by conda-build. The main +change is a pure YAML format without arbitrary Jinja or comments with semantic +meaning. ## Motivation -The conda-build format has grown over the years to become quite complex. Unfortunately it has never been formally "specified" and it has grown some features over time that make it hard to parse as straightforward YAML. +The conda-build format has grown over the years to become quite complex. +Unfortunately it has never been formally "specified" and it has grown some +features over time that make it hard to parse as straightforward YAML. -The CEP attempts to introduce a subset of the conda build format that allows for fast parsing and building of recipes. +The CEP attempts to introduce a subset of the conda build format that allows for +fast parsing and building of recipes. ### History -A discussion was started on what a new recipe spec could or should look like. The fragments of this discussion can be found here: https://github.com/mamba-org/conda-specs/blob/master/proposed_specs/recipe.md +A discussion was started on what a new recipe spec could or should look like. +The fragments of this discussion can be found here: +https://github.com/mamba-org/conda-specs/blob/master/proposed_specs/recipe.md The reason for a new spec are: -- Make it easier to parse ("pure yaml"). conda-build uses a mix of comments and jinja to achieve a great deal of flexibility, but it's hard to parse the recipe with a computer -- iron out some inconsistencies around multiple outputs (build vs. build/script and more) +- Make it easier to parse ("pure yaml"). conda-build uses a mix of comments and + jinja to achieve a great deal of flexibility, but it's hard to parse the + recipe with a computer +- iron out some inconsistencies around multiple outputs (build vs. build/script + and more) - remove any need for recursive parsing & solving ## Major differences with conda-build -- no full Jinja2 support: no conditional or `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid YAML -- Jinja string interpolation needs to be quoted at the beginning of a string, e.g. `- "{{ version }}"` in order for it to be valid YAML -- Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- sel(osx): somepkg` instead of `- somepkg # [osx]` +- no full Jinja2 support: no conditional or `{% set ...` support, only string + interpolation. Variables can be set in the toplevel "context" which is valid + YAML +- Jinja string interpolation needs to be quoted at the beginning of a string, + e.g. `- "{{ version }}"` in order for it to be valid YAML +- Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- + sel(osx): somepkg` instead of `- somepkg # [osx]` ## Selectors @@ -41,7 +55,8 @@ Selectors in the new spec take the following format: `sel(unix): selected_value` -This is a valid YAML dictionary. Selector contents are simple boolean expressions and follow Python syntax. The following selectors are all valid: +This is a valid YAML dictionary. Selector contents are simple boolean +expressions and follow Python syntax. The following selectors are all valid: ``` win and arm64 @@ -51,7 +66,8 @@ something == "test" ### The cmp function for variant selection -Furthermore, we have a special "cmp" function that can be used to run a check against a selected variant version. The `cmp` function looks like the following: +Furthermore, we have a special "cmp" function that can be used to run a check +against a selected variant version. The `cmp` function looks like the following: ``` cmp(python, "3.6") @@ -68,14 +84,16 @@ requirements: - sel(cmp(python, ">=3.6,<3.10")): dataclasses ``` -This functionality generalizes and replaces the previous special variables such as `py2k`, `py3k`, `py36`, `py37`, and works just as well for NumPy, Ruby, R, or any other variant that might be of interest in the future. +This functionality generalizes and replaces the previous special variables such +as `py2k`, `py3k`, `py36`, `py37`, and works just as well for NumPy, Ruby, R, or +any other variant that might be of interest in the future. ### Preprocessing selectors -You can add selectors to any item, and the selector is evaluated in -a preprocessing stage. If a selector evaluates to `true`, the item is -flattened into the parent element. If a selector evaluates to `false`, -the item is removed. +You can add selectors to any item, and the selector is evaluated in a +preprocessing stage. If a selector evaluates to `true`, the item is flattened +into the parent element. If a selector evaluates to `false`, the item is +removed. ```yaml source: @@ -85,8 +103,7 @@ source: url: http://path/to/windows/source ``` -Because the selector is a valid Jinja expression, complicated logic -is possible: +Because the selector is a valid Jinja expression, complicated logic is possible: ```yaml source: @@ -98,7 +115,8 @@ source: url: http://path/to/python3/unix/source ``` -Lists are automatically "merged" upwards, so it is possible to group multiple items under a single selector: +Lists are automatically "merged" upwards, so it is possible to group multiple +items under a single selector: ```yaml test: @@ -145,9 +163,10 @@ source: Jinja has built-in support for some common string manipulations. -In the new spec, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. -So you should not use any `{% if ... %}` or similar Jinja constructs that produce invalid yaml. -Furthermore, quotes need to be applied when starting a value with double-curly brackets like so: +In the new spec, complex Jinja is completely disallowed as we try to produce +YAML that is valid at all times. So you should not use any `{% if ... %}` or +similar Jinja constructs that produce invalid yaml. Furthermore, quotes need to +be applied when starting a value with double-curly brackets like so: ```yaml package: @@ -155,9 +174,9 @@ package: name: "{{ name }}" # correct ``` -Some Jinja functions remain valid, but this is out of the scope of this spec. However, as an example, -the `compiler` Jinja function will still be available, with the main difference being the quoting of the -brackets. +Some Jinja functions remain valid, but this is out of the scope of this spec. +However, as an example, the `compiler` Jinja function will still be available, +with the main difference being the quoting of the brackets. ```yaml requirements: @@ -165,11 +184,43 @@ requirements: - "{{ compiler('cxx') }}" ``` +## Shortcomings + +Since we deliberately limit the amount of "Jinja" that is allowed in recipes +there will be several shortcomings. + +For example, using a `{% for ... %}` loop is prohibited. After searching through +the conda-forge recipes with the Github search, we found for loops mostly used +in tests. + +In our view, for loops are a nice helper, but not necessary for many tasks: the +same functionality could be achieved in a testing script, for example. At the +same time we also plan to formalize a more powerful testing harness ([prototyped +in +boa](https://github.com/mamba-org/boa/blob/main/docs/source/recipe_spec.md#check-for-file-existence-in-the-final-package)). + +This could be used instead of a for loop to check the existence of shared +libraries or header files cross-platform (instead of relying on Jinja templates +as done +[here](https://github.com/conda-forge/opencv-feedstock/blob/2fc7848655ca65419050336fe38fcfd87bec0649/recipe/meta.yaml#L131) +or +[here](https://github.com/conda-forge/boost-cpp-feedstock/blob/699cfb6ec3488da8586833b1500b69133f052b6f/recipe/meta.yaml#L53)). + +Other uses of `for` loops should be relatively easy to refactor, such as +[here](https://github.com/conda-forge/libiio-feedstock/blob/1351e5846b753e4ee85624acf3a14aee4bcf321d/recipe/meta.yaml#L45-L51). + +However, since the new recipe format is "pure YAML" it is very easy to create +and pre-process these files using a script, or even generating them with Python +or any other scripting language. That means, many of the features that are +currently done with Jinja could be done with a simple pre-processing step in the +future. + ## Examples ### xtensor -Original recipe [found here](https://github.com/conda-forge/xtensor-feedstock/blob/feaa4fd8015f96038168a9d67d69eaf06a36d63f/recipe/meta.yaml). +Original recipe [found +here](https://github.com/conda-forge/xtensor-feedstock/blob/feaa4fd8015f96038168a9d67d69eaf06a36d63f/recipe/meta.yaml). ```yaml context: From a4f15ae9aed56ed7ba40f177bd34eea2e076bead Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 8 Jun 2023 11:33:28 +0200 Subject: [PATCH 04/12] Update cep-20.1.md Co-authored-by: Jannis Leidel --- cep-20.1.md | 1 + 1 file changed, 1 insertion(+) diff --git a/cep-20.1.md b/cep-20.1.md index f836885c..27c769d1 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -38,6 +38,7 @@ The reason for a new spec are: - iron out some inconsistencies around multiple outputs (build vs. build/script and more) - remove any need for recursive parsing & solving +- cater to needs for automation and dependency tree analysis via a determinstic format ## Major differences with conda-build From 628bffb652371ef19cea1dc087cb52157a3f0d4f Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Thu, 8 Jun 2023 11:33:40 +0200 Subject: [PATCH 05/12] Update cep-20.1.md Co-authored-by: Jannis Leidel --- cep-20.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-20.1.md b/cep-20.1.md index 27c769d1..96c0c91a 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -44,7 +44,7 @@ The reason for a new spec are: - no full Jinja2 support: no conditional or `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid - YAML + YAML (all new features should be native to YAML specs) - Jinja string interpolation needs to be quoted at the beginning of a string, e.g. `- "{{ version }}"` in order for it to be valid YAML - Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- From 918ae80d3333b7bb6e194c6a4a4f0af84a2e5a88 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Tue, 13 Jun 2023 09:50:34 +0200 Subject: [PATCH 06/12] update to new syntax --- cep-20.1.md | 165 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 115 insertions(+), 50 deletions(-) diff --git a/cep-20.1.md b/cep-20.1.md index 96c0c91a..f0ec7a72 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -38,25 +38,56 @@ The reason for a new spec are: - iron out some inconsistencies around multiple outputs (build vs. build/script and more) - remove any need for recursive parsing & solving -- cater to needs for automation and dependency tree analysis via a determinstic format +- cater to needs for automation and dependency tree analysis via a determinstic + format ## Major differences with conda-build -- no full Jinja2 support: no conditional or `{% set ...` support, only string +- no full Jinja2 support: no block support `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid YAML (all new features should be native to YAML specs) -- Jinja string interpolation needs to be quoted at the beginning of a string, - e.g. `- "{{ version }}"` in order for it to be valid YAML -- Selectors use a YAML dictionary style (vs. comments in conda-build). E.g. `- - sel(osx): somepkg` instead of `- somepkg # [osx]` +- Jinja variable syntax is changed to begin with `${{` so that it becomes valid + YAML, e.g. `- ${{ version }}` +- Selectors use a YAML dictionary with `if / then / else` (vs. comments in + conda-build) and are only allowed in lists (dependencies, scripts, ...). The + syntax looks like: + ```yaml + - if: win + then: this + else: that # optional + ``` +- for inline values, the Jinja ternary operator can be used, e.g. `number: ${{ 0 + if linux else 100 }}` ## Selectors -Selectors in the new spec take the following format: +Selectors in the new spec are only allowed in lists and take an explicit `if / +then / else` syntax. -`sel(unix): selected_value` +For example the following `script` section: -This is a valid YAML dictionary. Selector contents are simple boolean +```yaml +script: + - if: unix + then: | + # this is the unix script + - if: win + then: | + @rem a script for batch +``` + +The same could have been expressed with an `else`: + +```yaml +script: + - if: unix + then: | + # this is the unix script + else: | + @rem a script for batch +``` + +This is a valid YAML dictionary. Selector if statements are simple boolean expressions and follow Python syntax. The following selectors are all valid: ``` @@ -65,6 +96,22 @@ win and arm64 something == "test" ``` +If the value of a selector statement is a list, it extends the "outer" list. For +example: + +```yaml +build: + - ${{ compiler('cxx') }} + - if: unix + then: + - make + - cmake + - pkg-config +``` + +evaluates for `unix == true` to a list with elements `[${{ compiler('cxx') }}, +make, cmake, pkg-config]`. + ### The cmp function for variant selection Furthermore, we have a special "cmp" function that can be used to run a check @@ -79,10 +126,13 @@ etc This can be used in a selector like so: -``` +```yaml requirements: build: - - sel(cmp(python, ">=3.6,<3.10")): dataclasses + - if: cmp(python, ">=3.6,<3.10") + then: dataclasses + # The following syntax is also valid and equivalent + - ${{ "dataclasses" if cmp(python, ">=3.6,<3.10") }} ``` This functionality generalizes and replaces the previous special variables such @@ -98,21 +148,28 @@ removed. ```yaml source: - - sel(not win): + - if: not win + then: + # note that we omit the `-`, both is valid url: http://path/to/unix/source - - sel(win): - url: http://path/to/windows/source + sha256: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b + else: + - url: http://path/to/windows/source + sha256: 06f961b802bc46ee168555f066d28f4f0e9afdf3f88174c1ee6f9de004fc30a0 ``` Because the selector is a valid Jinja expression, complicated logic is possible: ```yaml source: - - sel(win): + - if: win + then: url: http://path/to/windows/source - - sel(unix and cmp(python, "2")): + - if: (unix and cmp(python, "2")) + then: url: http://path/to/python2/unix/source - - sel(unix and cmp(python, "3")): + - if: unix and cmp(python, "3") + then: url: http://path/to/python3/unix/source ``` @@ -122,15 +179,17 @@ items under a single selector: ```yaml test: commands: - - sel(unix): - - test -d ${PREFIX}/include/xtensor - - test -f ${PREFIX}/include/xtensor/xarray.hpp - - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake - - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake - - sel(win): - - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) - - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfig.cmake (exit 1) - - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) + - if: unix + then: + - test -d ${PREFIX}/include/xtensor + - test -f ${PREFIX}/include/xtensor/xarray.hpp + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfig.cmake + - test -f ${PREFIX}/lib/cmake/xtensor/xtensorConfigVersion.cmake + - if: win + then: + - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) + - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfig.cmake (exit 1) + - if not exist %LIBRARY_PREFIX%\lib\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) # On unix this is rendered to: test: @@ -151,7 +210,7 @@ You can set up Jinja variables in the context YAML section: context: name: "test" version: "5.1.2" - major_version: "{{ version.split('.')[0] }}" + major_version: ${{ version.split('.')[0] }} ``` Later in your `recipe.yaml` you can use these values in string interpolation @@ -159,30 +218,31 @@ with Jinja. For example: ```yaml source: - url: https://github.com/mamba-org/{{ name }}/v{{ version }}.tar.gz + url: https://github.com/mamba-org/${{ name }}/v${{ version }}.tar.gz ``` Jinja has built-in support for some common string manipulations. In the new spec, complex Jinja is completely disallowed as we try to produce YAML that is valid at all times. So you should not use any `{% if ... %}` or -similar Jinja constructs that produce invalid yaml. Furthermore, quotes need to -be applied when starting a value with double-curly brackets like so: +similar Jinja constructs that produce invalid YAML. We also do not use the +standard Jinja delimiters (`{{ .. }}`) because that is confused by the YAML +parser as a dictionary. We follow Github Actions and others and use `${{ ... }}` +instead: ```yaml package: name: {{ name }} # WRONG: invalid yaml - name: "{{ name }}" # correct + name: ${{ name }} # correct ``` -Some Jinja functions remain valid, but this is out of the scope of this spec. -However, as an example, the `compiler` Jinja function will still be available, -with the main difference being the quoting of the brackets. +Jinja functions work as usual. As an example, the `compiler` Jinja function will +look like this: ```yaml requirements: build: - - "{{ compiler('cxx') }}" + - ${{ compiler('cxx') }} ``` ## Shortcomings @@ -216,6 +276,9 @@ or any other scripting language. That means, many of the features that are currently done with Jinja could be done with a simple pre-processing step in the future. +Another option would be to allow "full" Jinja inside the test script text blocks +(as long as it doesn't change the structure of the YAML). + ## Examples ### xtensor @@ -230,24 +293,27 @@ context: sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655 package: - name: "{{ name|lower }}" - version: "{{ version }}" + name: ${{ name|lower }} + version: ${{ version }} source: - fn: "{{ name }}-{{ version }}.tar.gz" - url: https://github.com/xtensor-stack/xtensor/archive/{{ version }}.tar.gz - sha256: "{{ sha256 }}" + fn: ${{ name }}-${{ version }}.tar.gz" + url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz + sha256: ${{ sha256 }} build: number: 0 - sel(win and vc<14): - skip: true + # note: in the new recipe format, `skip` is a list of conditional expressions + # but for the "YAML format" discussion we pretend that we still use the + # `skip: bool` syntax + skip: ${{ true if (win and vc14) }} requirements: build: - - "{{ compiler('cxx') }}" + - ${{ compiler('cxx') }} - cmake - - sel(unix): make + - if: unix + then: make host: - xtl >=0.7,<0.8 run: @@ -257,12 +323,14 @@ requirements: test: commands: - sel(unix): + if: unix + then: - test -d ${PREFIX}/include/xtensor - test -f ${PREFIX}/include/xtensor/xarray.hpp - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake - sel(win): + if: win + then: - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfig.cmake (exit 1) - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfigVersion.cmake (exit 1) @@ -279,9 +347,6 @@ about: extra: recipe-maintainers: - - SylvainCorlay - - JohanMabille - - wolfv - - davidbrochart + - some-maintainer ``` From 878053c5e8c6aa9fa605fe3b72fa08bf1fa01ce5 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Tue, 13 Jun 2023 10:46:55 +0200 Subject: [PATCH 07/12] Update cep-20.1.md Co-authored-by: jaimergp --- cep-20.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-20.1.md b/cep-20.1.md index f0ec7a72..74d904b0 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -329,7 +329,7 @@ test: - test -f ${PREFIX}/include/xtensor/xarray.hpp - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfig.cmake - test -f ${PREFIX}/share/cmake/xtensor/xtensorConfigVersion.cmake - if: win + - if: win then: - if not exist %LIBRARY_PREFIX%\include\xtensor\xarray.hpp (exit 1) - if not exist %LIBRARY_PREFIX%\share\cmake\xtensor\xtensorConfig.cmake (exit 1) From 85a5a3f96da67dc4a3108fef834177495bacea77 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Tue, 13 Jun 2023 10:49:01 +0200 Subject: [PATCH 08/12] Update cep-20.1.md Co-authored-by: jaimergp --- cep-20.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-20.1.md b/cep-20.1.md index 74d904b0..79926c45 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -323,7 +323,7 @@ requirements: test: commands: - if: unix + - if: unix then: - test -d ${PREFIX}/include/xtensor - test -f ${PREFIX}/include/xtensor/xarray.hpp From c49d58b74bccc20516da3be8a5c92317fa4d9fae Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 20 Oct 2023 14:40:36 +0200 Subject: [PATCH 09/12] remove cmp function for now --- cep-20.1.md | 27 --------------------------- 1 file changed, 27 deletions(-) diff --git a/cep-20.1.md b/cep-20.1.md index 79926c45..0d1bf1bd 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -112,33 +112,6 @@ build: evaluates for `unix == true` to a list with elements `[${{ compiler('cxx') }}, make, cmake, pkg-config]`. -### The cmp function for variant selection - -Furthermore, we have a special "cmp" function that can be used to run a check -against a selected variant version. The `cmp` function looks like the following: - -``` -cmp(python, "3.6") -cmp(python, ">=3.6") -cmp(python, ">=3.8,<3.10") -etc -``` - -This can be used in a selector like so: - -```yaml -requirements: - build: - - if: cmp(python, ">=3.6,<3.10") - then: dataclasses - # The following syntax is also valid and equivalent - - ${{ "dataclasses" if cmp(python, ">=3.6,<3.10") }} -``` - -This functionality generalizes and replaces the previous special variables such -as `py2k`, `py3k`, `py36`, `py37`, and works just as well for NumPy, Ruby, R, or -any other variant that might be of interest in the future. - ### Preprocessing selectors You can add selectors to any item, and the selector is evaluated in a From 2c8ad62f45f5d6322ac4324ef6b39a582f7de318 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 20 Oct 2023 14:50:56 +0200 Subject: [PATCH 10/12] update dates, status == proposed --- cep-20.1.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cep-20.1.md b/cep-20.1.md index 0d1bf1bd..9fa0cd3e 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -2,11 +2,11 @@ - + - - + +
Title A new recipe format
Status Proposed
Status Proposed
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Created May 23, 2023
Updated May 23, 2023
Discussion
Updated October 20, 2023
Discussion https://github.com/conda-incubator/ceps/pull/54
Implementation https://github.com/prefix-dev/rattler-build
From cd196f479120fcd0f3867a6aac54d9cbf9de5762 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 20 Oct 2023 19:09:50 +0200 Subject: [PATCH 11/12] Update cep-20.1.md Co-authored-by: jaimergp --- cep-20.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-20.1.md b/cep-20.1.md index 9fa0cd3e..a7debe31 100644 --- a/cep-20.1.md +++ b/cep-20.1.md @@ -270,7 +270,7 @@ package: version: ${{ version }} source: - fn: ${{ name }}-${{ version }}.tar.gz" + fn: ${{ name }}-${{ version }}.tar.gz url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz sha256: ${{ sha256 }} From b7d677bc1281c187e27bb59a6f365df9690d73bb Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Mon, 30 Oct 2023 11:22:01 +0100 Subject: [PATCH 12/12] rename cep to cep 13 and update status to accepted --- cep-20.1.md => cep-13.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename cep-20.1.md => cep-13.md (99%) diff --git a/cep-20.1.md b/cep-13.md similarity index 99% rename from cep-20.1.md rename to cep-13.md index a7debe31..5d62e35b 100644 --- a/cep-20.1.md +++ b/cep-13.md @@ -1,8 +1,8 @@ -# A new recipe format – part 1 +# A new Recipe format (Part 1) - +
Title A new recipe format
Status Proposed
Status Accepted
Author(s) Wolf Vollprecht <wolf@prefix.dev>
Created May 23, 2023
Updated October 20, 2023