bootstrap-conda: Refactor, generate versioned environment files
#36405
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mkoeppe
force-pushed
the
bootstrap_conda_refactor
branch
from
a6f8c03
to
6e1f955
Compare
mkoeppe
force-pushed
the
bootstrap_conda_refactor
branch
2 times, most recently
from
9ae39bf
to
939aa1b
Compare
|
@tobiasdiez This is a cleaned up version of parts of #35593, which is independent from other changes that are still in the pipeline. This PR should be neutral on the CI workflows, i.e., neither fixing nor breaking something. Please take a look |
|
Thanks. But there is now a |
|
Ah, I see. I'll back this change (creating the pip section not only for environment-optional and environment-dev, but also for environment) out from here until the package lists are fixed. |
mkoeppe
force-pushed
the
bootstrap_conda_refactor
branch
from
939aa1b
to
fdeb0f1
Compare
tobiasdiez
approved these changes
Oct 5, 2023
|
Thanks! |
|
merge conflict |
added 8 commits
October 8, 2023 13:07
…nor versions (environment-3.9.yml etc.) This is convenient for conda-lock. Partial cherry-pick from "Use conda-lock for reproducible conda env"
…: Use configure --enable-system-site-packages
mkoeppe
force-pushed
the
bootstrap_conda_refactor
branch
from
fdeb0f1
to
b02ef8e
Compare
tobiasdiez
previously requested changes
Oct 9, 2023
|
Along similar lines, should we include the Python version in the default environment name written to the file? |
Care to elaborate why you think it would overwrite it? I don't think it does. |
I tried it out and it was not working (that's how I discovered it in the first place). |
|
I also tried it out and it worked, so some details are needed. |
|
Might be a version mismatch, I don't know. Either way, I don't think its a good idea to have two python specifiers in the file. |
|
Well ok, here you go. |
5 tasks
|
Thanks! Are the |
|
I haven't tried. This PR, as requested, is focused narrowly on what is promised in its title. Other fixes are on other PRs. |
|
On |
|
OK, here's a temporary workaround. |
|
Documentation preview for this PR (built with commit 73ab316; changes) is ready! 🎉 |
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Dec 27, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 2, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 5, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
vbraun
pushed a commit
to vbraun/sage
that referenced
this pull request
Jan 13, 2024
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
--enable-system-site-packagesfrom Use system python packages with --enable-system-site-packages #36141 by @orlitzkySplit out from
📝 Checklist
⌛ Dependencies