WIP: Build customization integration

[msarahan]

This works towards integration of #848 with work that has occurred since (primarily the separation of rendering recipes from build.)

I am suggesting to remove --bootstrap, and instead roll its functionality into --build-config, which has dynamic behavior depending on whether a file or a folder/env name is provided.

When complete, this PR will:

• Provide a central configuration file that can contain sets of configurations or variants of packages.
• Discuss applications, and potentially provide a local override for that central configuration, for local (recipe-level) control of configurations/variants.
• Provide an API for examining the union of this/these configurations, along with the recipe contents. This API will be the tool for matrix build tools to understand the loop-able dimensions of a given recipe in a given configuration system.
• Provide a first-pass syntax for pinning under a few configurations:
  • Pin to version defined by configuration files (similar to numpy x.x)
  • Pin to minimum version, with upper bound of next major version (default "compatible" pin)
  • Pin to installed version

I'm leaving some related stuff out of this PR. Namely, I think that we need to have conda-build be much more meticulous about recording system information, such as compiler versions, ABI tags, and anything else that may affect compatibility. I anticipate that this PR will help integrate this information into "compatible" pins, but leave that for future work.

CC @ukoethe @stuarteberg @pelson @jakirkham @mcg1969 @kalefranz

[ukoethe]

Copy & paste typo python

[msarahan]

Good catch, thanks.

[jankatins]

re pinning: I think there are multiple angles here: first one to influence the build matrix (numpy x.x) and then there is the problem that link-time requirements currently can change on rebuild which then leads to crashes when the user installs a package which fits the conda requirement but not anymore the link-time requirement (example).

Fot the second part, I think that the solution to that problem is similar solution the the shlibs/SYMBOLS files in debian (description in the debian policy document). E.g.

• Include a file in the library which contains the pinning requirements (e.g. >= 1.1,<1.2 or only >= when the package name changes when the new version is not API/ABI compatible to the lower ones). ideally this file is automatically generated, but if not it's probably easy to build by hand.
• let the build restrictions be whatever version is the minimum requirement of that app -> Manually chosen by the package maintainer
• let the runtime restrictions be the requirements from the installed files (e.g libpng {{installed["libpng"].versions_requirement}} will get the pinning file from wherever that gets installed and insert it here). This requirement fits the version which the app is build against and so the above problem can't happen anymore (as long as the requirement file is up to date...).

-> The library is responsible for creating that information, not the package. If the package wants stricter requirements it has to do something manually.

This could even work as {{installed_packages_dependencies}} variable which would spit out the names and version-pins for all installed packages in the build environment.

[msarahan]

@JanSchulz, thanks. This is exactly what I want to develop once this PR is done. However, I did not want to make the scope of this PR too large. I have a bad habit of that.

Is there anything this PR specifically should include (or avoid) to ensure that pursuing what you suggest is as easy as possible?

[fschlimb]

A dynamic configuration option is something we would also greatly appreciate.

Here is an alternative (potential) approach to dynamically change or set entries in meta.yaml in general:
The configuration file could be just another yaml file which can specify whatever meta.yaml can. This config file should be provided on the command line. Let's call it config.yaml for now (even though the user could use any other name and place it where s/he wants). conda-build would combine the two files by - conceptually - concatenating them: cat meta.yaml config.yaml > final.yaml. When parsing the resulting file, it would - conceptually -

• overwrite existing entries if found again
• add new entries to sections that have been found before
• add new sections that have not been found before
• delete entries specified for deletion in config.yaml (syntax to be defined)

This will probably not work by simply concatenating the two files, but semantically that's what I have in mind.

The resulting mechanism would be more flexible, because config.yaml can change, add and delete whatever it wants, without requiring jinja2 stuff in meta.yaml. The author of meta.yaml might not have thought about what someone else would like to change - with this we could still support this without changing meta.yaml at all. meta.yaml would be the author's recommended (default) configuration.

Even though more flexible, this mechanism is also simpler in concept: it does not need an additional indirection (through jinja2) to think about.

I have not thought about all details raised in these discussions, but in general it seems worth being considered/discussed.

@msarahan Any thoughts?

[msarahan]

CC @ukoethe @stuarteberg @pelson @jakirkham @johanneskoester - you all have had good input on this topic. Please chime in with any thoughts you have on Frank's idea.

I think Jinja2 makes naming templated regions more explicit at a finer level than overriding meta.yaml keys, and templates also are more reusable than overwriting meta.yaml (reuse of a given value)

I think it's reasonable to ask and work with recipe authors to add templated regions wherever you need to add them. I don't see the necessity of overriding non-templated regions.

I think that more complicated cascades of overrides are necessary to achieve a given level of nontrivial collection of modifications. I personally would be error prone and forgetful, and would spend a lot of time debugging why I wasn't getting a given value.

[ukoethe]

Thanks for your interesting thoughts, @fschlimb.

This config file should be provided on the command line. Let's call it config.yaml for now

The current PR already includes the possibility to specify a config file on the command line. At present, this file may only specify the requirements: build: section (everything else is ignored). Moreover, existing build requirements in meta.yaml are not overwritten, but concatenated. That is, version resolution must fulfill the requirements of both meta.yaml and config.yaml. The idea behind this rule is that meta.yaml lists dependencies in general (e.g. numpy 1.10), whereas config.yaml specializes these dependencies to particular versions (which must not contradict the general specification, e.g. numpy 1.10.2).

In the lengthy discussions leading up to this PR, we found that other aspects of meta.yaml (e.g. features: and requirements: run:) can be made flexible by jinja2 templating in a very elegant way. Do you have concrete use cases where your approach is superior?

[fschlimb]

Yes, I do understand that this solution comes with a different mindset. Rather than than trying to define which aspects could be modified in which manner this simply provides full freedom. At some point someone will want to change something that we have not thought of. What seems well-motivated now might seem arbitrary later.

People can always mess up, there is no way to protect from this. I prefer combining 2 simple concepts in an orthogonal manner rather than linking them with a (complicated and) fixed rule-set..

My current use-case is that I want to define a different source for the source, e.g. I want to use sources from git rather than a ftp url. Or I want to add a patch that I know will not get accepted widely.

[msarahan]

Superseded by #1585 - I didn't want to fight git to bring that branch up to date.

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically locked because it has not had recent activity after being closed.

Please open a new issue or pull request if needed.

Thanks!