`pkgs/sagemath-standard`: Move metadata from `setup.cfg` to `pyproject.toml` #36951

mkoeppe · 2023-12-23T20:06:53Z

Modernize the metadata.

This is a trivial "chore" PR. It updates Python metadata to the latest format. No controversies about the current format are known about in the Python community. In a typical open source project, someone in a Maintainer role would open a PR and then immediately merge it, or when receiving such a PR from the outside, quickly review and merge it (examples: my PRs pytest-dev/pytest-mock#410 (merged in within 1 day), pyodide/pyodide#4472, pytest-dev/pytest-xdist#1020,
sagemath/cypari2#158, fplll/fpylll#258, polymake/JuPyMake#2, cvxpy/cvxpy#2276, sagemath/cysignals#177).

Also fixes the pyproject.toml build requirements of sagemath-standard, broken since 10.2, hence "critical".

Part of Migrate from setup.cfg to pyproject.toml #33577

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

Depends on Switch more Python toolchain packages to use (platform-independent) wheels #36885 (merged here)
Depends on Use conda-lock for reproducible conda env #35986 (merged here to resolve merge conflict)
Depends on Sphinx ext links for Sage source files #37546 (merged here to resolve merge conflict)

tobiasdiez · 2023-12-30T06:37:06Z

src/pyproject.toml.m4

+name = "sagemath-standard"
+description = "Sage: Open Source Mathematics Software: Standard Python Library"
+dependencies = [
+    SPKG_INSTALL_REQUIRES_sage_conf


I think you can simply use dynamic metadata and

dependencies = {file = ["build/pkgs/whatever/install-requires.txt", "build/pkgs/whatever2/install-requires.txt", ...]}

Thanks, but that would not be an improvement.
It would make the sdist of this distribution depend on the monorepo / the Sage distribution.

setuptools automatically includes the linked files (and only these) in the sdist.

These file paths build/pkgs/... do not exist in this directory.

In any case, changes to this design are outside of the scope of this PR. This design is already in place in

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-bliss/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-categories/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-coxeter3/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-environment/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-mcqd/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-meataxe/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-objects/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-repl/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-sirocco/pyproject.toml.m4

https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-tdlib/pyproject.toml.m4

mkoeppe · 2023-12-30T18:52:11Z

Let's get it in.

orlitzky · 2023-12-30T23:42:41Z

Rebase onto develop please?

…ve src/setup.cfg.m4, update docs

mkoeppe · 2023-12-31T00:13:23Z

Rebased.

orlitzky · 2023-12-31T00:47:36Z

Both choices for the metadata (./bootstrap or file=[...]) reference the build tree from src and therefore look comparably bad to me w.r.t. maintaining library/distro separation. But we also don't need to redesign it right now. Does this cause problems for anyone? If not, this is standard python packaging churn and is fine with me.

mkoeppe · 2023-12-31T01:10:17Z

Both choices for the metadata (./bootstrap or file=[...]) reference the build tree from src

Yes, but bootstrap only runs at ... bootstrapping time, so it is a "repository concern" only and not a "distribution concern" if that makes sense. After running bootstrap, each subdirectory in pkgs is a self-contained source directory of a Python distribution.

mkoeppe · 2023-12-31T01:13:56Z

Also, in PRs such as the following, I make less code dependent on the file layout within build/pkgs. So poking into there directly from the pyproject.toml would run counter to this work.

Refactor bootstrap-conda, m4/sage_spkg_collect.m4 through sage-package; handle dependencies_build #36740

mkoeppe · 2023-12-31T01:14:53Z

Does this cause problems for anyone?

No, I wouldn't know how it could.

mkoeppe · 2024-01-01T18:16:55Z

Let's get this in please.

kwankyu

It works well. LGTM.

tobiasdiez · 2024-01-02T13:42:20Z

Both choices for the metadata (./bootstrap or file=[...]) reference the build tree from src and therefore look comparably bad to me w.r.t. maintaining library/distro separation. But we also don't need to redesign it right now.

@kwankyu Sorry, forgot to mention that this suggestion was implemented in #36982.

To not hold up this PR here until #36982 is merged, I would be fine for now with putting the dynamically created information in file that is generated during bootstrap and included in pyproject.toml via the file tag.

orlitzky · 2024-01-02T13:42:53Z

It works well. LGTM.

There's now a conflicting proposal in #36982, both are probably headed to the dispute pile.

mkoeppe · 2024-01-02T16:23:59Z

To not hold up this PR here until #36982 is merged, I would be fine for now with putting the dynamically created information in file that is generated during bootstrap and included in pyproject.toml via the file tag.

This PR does not have to wait for any such innovations.

…d_pyproject_toml

github-actions · 2024-04-09T04:01:56Z

Documentation preview for this PR (built with commit 18cb712; changes) is ready! 🎉

…files, remove remaining `# coding: utf-8`   These directives at the top of the file inform developers about the intended assignment of modules to pip-installable distributions. As of this PR, there should be no change to the existing distributions (sagemath-categories...) nor the monolithic build of the Sage library.  - Cherry-picked from sagemath#35095   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   - Depends on sagemath#36951 (merged here) - Depends on sagemath#36676 (merged here) - Depends on sagemath#37667 (merged here) URL: sagemath#36964 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez

jhpalmieri · 2024-04-13T15:55:23Z

+1 from me.

mkoeppe · 2024-04-13T16:23:36Z

Thanks, John.

sagemath#36964 was inappropriately merged, since two dependencies were still disputed. This PR attempts to revert the merge. It was created by ``` git revert -m 1 6ecb1d8 ``` ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies None, though it will also revert sagemath#36676 and sagemath#36951. URL: sagemath#37796 Reported by: roed314 Reviewer(s): Dima Pasechnik

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

sagemath#36964 was inappropriately merged, since two dependencies were still disputed. This PR attempts to revert the merge. It was created by ``` git revert -m 1 6ecb1d8 ``` ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies None, though it will also revert sagemath#36676 and sagemath#36951. URL: sagemath#37796 Reported by: roed314 Reviewer(s): Dima Pasechnik

saraedum · 2024-04-29T12:09:21Z

The changes from this PR are now at #37902 which reverts the revert from #37796.

@kwankyu

…arate jobs for pyright, build, modularized tests, long tests    Running the containers explicitly, instead of using the declarative `container:` feature of GH Actions gives us more control: - we can create more space on the host if necessary; we just scraped by an out of space condition in sagemath#36473 / sagemath#36469 (comment) - we can run some operations outside of the container but in the same job; this will make the separate "Get CI fixes" jobs unnecessary, addressing the cosmetic concerns from sagemath#36338 (comment), sagemath#36349 - it enables caching between the various workflows (as first discussed in sagemath#36446). We split out static type checking with pyright into its separate workflow: - **pyright.yml**: As static checking does not need a build of the Sage library, for PRs that do not make any changes to packages, there's nothing to build, and the workflow gives a fast turnaround just after 10 minutes. For applying the CI fixes from blocker tickets, this workflow uses the technique favored in sagemath#36349. The workflow **build.yml** first launches a job: - **test-new:** It builds incrementally (using a tox-generated `Dockerfile` and https://github.com/marketplace/actions/build-and-push- docker-images) and does the quick incremental test. This completes within 10 to 20 minutes when there's no change. After this is completed, more jobs are launched: - **test-mod:** It again builds incrementally and tests a modularized distribution. Later (with more from sagemath#35095), more jobs will be added to this matrix job for other distributions. - **test-long:** It again builds incrementally and runs the long test. The workflows **doc-build.yml** and **doc-build-pdf.yml** again build incrementally and then build the documentation. The diffing code for the HTML documentation now runs in the host, not the container, which simplifies things. (To show that diffing still works, we include a small change to the Sage library.) Splitting the workflow jobs implements @kwankyu's suggestion from: - sagemath#35652 (comment) (Fixes sagemath#35814) The steps "again builds incrementally" actually use the GH Actions cache, https://docs.docker.com/build/ci/github-actions/cache/#cache- backend-api. When nothing is cached and the 3 workflows are launched in parallel, they may each build the same thing. But when there's congestion that leads to the workflows to be run serially, the 2nd and 3rd workflow will be able to use the cache from the 1st workflow. This elasticity may be helpful in reducing congestion. There is a rather small per-project limit of 10 GB for this cache, so we'll have to see how effectively caching works in practice. See https://github.com/sagemath/sage/actions/caches   ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies  - Depends on sagemath#36938 (merged here) - Depends on sagemath#36951 (merged here) - Depends on sagemath#37351 (merged here) - Depends on sagemath#37750 (merged here) - Depends on sagemath#37277 (merged here)  URL: sagemath#36498 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

As noted in sagemath#36951 (comment), the fact that the dependency declaration of sage-the-library depends on information contain in sage-the-distro's build folder is problematic for the separation of concerns. Sage-the-distro should rely on information of sage-the-library, not the other way around. Here we fix this for the build requirements by making `src/pyproject.toml` the single source of thruth. The corresponding `install-requires.txt` are demoted to mere indicator files (to be removed later). Moreover, this change allows us to make `pyproject.toml` a static file (no longer generated by configure), which should help with downstream packaging. --- Happy new year!    ### 📝 Checklist     - [ ] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#37796   URL: sagemath#36982 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Gonzalo Tornaría, Matthias Köppe, Tobias Diez

mkoeppe added the c: distribution label Dec 23, 2023

mkoeppe self-assigned this Dec 23, 2023

mkoeppe added the s: needs review label Dec 23, 2023

mkoeppe requested a review from orlitzky December 25, 2023 23:21

mkoeppe added the p: critical / 2 label Dec 29, 2023

mkoeppe requested review from kwankyu and kiwifb December 29, 2023 01:54

tobiasdiez reviewed Dec 30, 2023

View reviewed changes

src/pyproject.toml.m4: Merge all metadata from src/setup.cfg.m4, remo…

1a71556

…ve src/setup.cfg.m4, update docs

mkoeppe force-pushed the sagemath_standard_pyproject_toml branch from 03f8e9c to 1a71556 Compare December 31, 2023 00:13

tobiasdiez mentioned this pull request Dec 31, 2023

Make pyproject.toml the source for build dependencies #36982

Merged

5 tasks

kwankyu approved these changes Jan 2, 2024

View reviewed changes

kwankyu added s: positive review and removed s: needs review labels Jan 2, 2024

tobiasdiez added s: needs work and removed s: positive review labels Jan 2, 2024

mkoeppe added s: positive review disputed PR is waiting for community vote, see https://groups.google.com/g/sage-devel/c/IgBYUJl33SQ and removed s: needs work labels Jan 2, 2024

mkoeppe removed the s: needs work label Apr 8, 2024

mkoeppe requested review from culler and saraedum April 8, 2024 02:03

tornaria removed the s: positive review label Apr 8, 2024

mkoeppe added the s: needs review label Apr 8, 2024

mkoeppe requested a review from williamstein April 8, 2024 17:01

mkoeppe added s: needs work and removed s: needs review labels Apr 8, 2024

Merge remote-tracking branch 'upstream/develop' into sagemath_standar…

18cb712

…d_pyproject_toml

mkoeppe added s: needs review and removed s: needs work labels Apr 8, 2024

mkoeppe requested a review from jhpalmieri April 11, 2024 20:43

vbraun merged commit 5ae0bc7 into sagemath:develop Apr 12, 2024
13 of 33 checks passed

mkoeppe added this to the sage-10.4 milestone Apr 12, 2024

mkoeppe deleted the sagemath_standard_pyproject_toml branch April 12, 2024 23:42

github-actions bot removed the s: needs review label Apr 13, 2024

roed314 mentioned this pull request Apr 13, 2024

Revert PR with disputed dependencies #37796

Merged

5 tasks

This was referenced Apr 29, 2024

Add # sage_setup: distribution directives to all files #37901

Open

pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #37902

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`pkgs/sagemath-standard`: Move metadata from `setup.cfg` to `pyproject.toml` #36951

`pkgs/sagemath-standard`: Move metadata from `setup.cfg` to `pyproject.toml` #36951

mkoeppe commented Dec 23, 2023 •

edited

Loading

tobiasdiez Dec 30, 2023

mkoeppe Dec 30, 2023

tobiasdiez Dec 30, 2023

mkoeppe Dec 30, 2023

mkoeppe Dec 30, 2023

mkoeppe commented Dec 30, 2023

orlitzky commented Dec 30, 2023

mkoeppe commented Dec 31, 2023

orlitzky commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Jan 1, 2024

kwankyu left a comment

tobiasdiez commented Jan 2, 2024

orlitzky commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

github-actions bot commented Apr 9, 2024

jhpalmieri commented Apr 13, 2024

mkoeppe commented Apr 13, 2024

saraedum commented Apr 29, 2024

pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #36951

pkgs/sagemath-standard: Move metadata from setup.cfg to pyproject.toml #36951

Conversation

mkoeppe commented Dec 23, 2023 • edited Loading

📝 Checklist

⌛ Dependencies

tobiasdiez Dec 30, 2023

Choose a reason for hiding this comment

mkoeppe Dec 30, 2023

Choose a reason for hiding this comment

tobiasdiez Dec 30, 2023

Choose a reason for hiding this comment

mkoeppe Dec 30, 2023

Choose a reason for hiding this comment

mkoeppe Dec 30, 2023

Choose a reason for hiding this comment

mkoeppe commented Dec 30, 2023

orlitzky commented Dec 30, 2023

mkoeppe commented Dec 31, 2023

orlitzky commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Dec 31, 2023

mkoeppe commented Jan 1, 2024

kwankyu left a comment

Choose a reason for hiding this comment

tobiasdiez commented Jan 2, 2024

orlitzky commented Jan 2, 2024

mkoeppe commented Jan 2, 2024

github-actions bot commented Apr 9, 2024

jhpalmieri commented Apr 13, 2024

mkoeppe commented Apr 13, 2024

saraedum commented Apr 29, 2024

`pkgs/sagemath-standard`: Move metadata from `setup.cfg` to `pyproject.toml` #36951

`pkgs/sagemath-standard`: Move metadata from `setup.cfg` to `pyproject.toml` #36951

mkoeppe commented Dec 23, 2023 •

edited

Loading