Conda-build 3 May Change Everything #799
Comments
See #799 We will need to make major changes to our build environment before we move to conda-build 3
See omnia-md/conda-recipes#799 We will need to make changes before we move to conda-build 3
|
I have started delving into the new Conda-Build, and it looks like it may simplify some things, but make our name checking to see if the package already exists tricky. Instead of names like I'm also still trying to figure out how we reduce build duplication since each build is now considered a variant, which is hard to aggregate over many many packages while also checking for the package already existing on the cloud. Has anyone else tried using conda-build 3 for multiple package/variant builds yet? |
This PR converts the `conda-build-all` script to be compatible with
Conda Build 3 and Conda 4.4, using the API as much as I can. Its
imperfect due to the imperfect API, but I think I managed to get what
we need without high-maintenance work arounds.
In theory, this should make the conda-build all script much more stable
and we can switch back to using the `conda-forge` docker image as a
base after some iterations
We've kicked this can down the road about as far as it can go at this
point so I think its time to implement a proper fix.
All existing functionality should remain unchanged from how the script
is used and invoked.
Complete list of everything I tweaked:
* Simplify imports to minimal set
* Import only from the Conda(-build) API when possible
* Added docstrings everywhere
* Use metapackage functions directly instead of reading from various metapackage dicts
* Reduce calls to the CLI directly where possible, replacing with API
* Convert previous multi-python multi-numpy build-mixing to Conda Build 3 Variant system
* Variants compiled from command line, still supports `--python` and `--numpy`
* Build enumeration automatically checks for duplications
* Still uses old package string styles instead of hashed dependencies for checking if package already on Anaconda (Will need to convert at some point)
* Adds ability to check if package is already built locally, and can be overridden with `--rebuild`
* Preserved checking against specific channels, both at command line and meta.yaml level
* Package building handled by API instead of CLI
* Functions more generalized to allow importing the conda-build-all script if need be
* Switched all print statements to the Python 3 `.format()` mini-language over `%` formatting
* Minimized amount of excess data passed between functions, handled mostly by Metadata now
* Added ability for built and uploaded tarballs to be removed to save build space (e.g. Travis-CI) through the `-c` or `--clean` command line arg. This is False by default (related: omnia-md/conda-recipes#877)
* Unpinned the conda and conda-build versions from the scripts. Will need to make changes to the Docker Image though
Related issues: omnia-md/conda-recipes#799
Potentially resolved issues from these changes:
* omnia-md/conda-recipes#712
* omnia-md/conda-recipes#403 (not likely, but something to consider)
Conda-build 3 is a major change to the way builds are handled. For instance, it looks like single
meta.yamlfiles can build multiple versions of Python in one go. We may even be able to vastly simply if not outright remove our "conda-build-all" script.Because of the major changes conda-build 3 will introduce, and some build issues we are seeing on our local packages, I am going to pin the conda-build version to 2.1.17 (last of the 2 series) until we can properly use conda build 3.
This issue should stay open until we can use conda build 3.
additional reading: https://conda.io/docs/building/variants.html
The text was updated successfully, but these errors were encountered: