sagemath-standard: include sage_setup in sdist

[tornaria]

This makes it so that sage_setup is included in sagemath_standard
for convenience and ease of installation.

Having sage_setup as a separate distribution leads to chicken-and-egg problem. For example, this is what happens when I try to build sagemath-standard from #37270:

cd pkgs/sagemath-standard
$ python -m build -s 
* Creating venv isolated environment...
* Installing packages in isolated environment... (cypari2 >=2.1.1, cysignals >=1.10.2, cython >=3.0, != 3.0.3, <4.0, gmpy2 ~=2.1.b999, jinja2 >=3.0, jupyter_core >=4.6.3, memory_allocator, numpy >=1.19, pkgconfig, pplpy >=0.8.6, sage-conf ~= 10.3b7, sage-setup ~= 10.3b7, sage_setup[autogen], setuptools >= 68.1.1, wheel >=0.36.2)

[... snip ...]

ERROR: Could not find a version that satisfies the requirement sage-setup~=10.3b7 (from versions: 9.4rc1)
ERROR: No matching distribution found for sage-setup~=10.3b7

[...]

This is because the version of sage-setup which is available on pypi requires python <= 3.11. Even if the branch I'm trying to build allows python 3.12, I cannot use it.

The main idea of this change is that sagemath-standard sdists are self-contained and include the exact matching version of sage-setup. The self-contained part is good for the pypi sdist, but the matching version allows for greater flexibility when testing, given that the implementations of sagemath-standard/setup.py and sage_setup are so intrincately connected and really need to match.

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.

[mkoeppe]

-1.

sage-setup is declared as PEP 518 build requirement.

Including a separate copy of a part of it in the sdist can only create confusion.

[mkoeppe]

No thanks, strong -1.

[tornaria]

-1.

sage-setup is declared as PEP 518 build requirement.

Including a separate copy of a part of it in the sdist can only create confusion.

You are right. I forgot to remove it from there. Done now.

I've got used to build without --skip-dependency-check because of the non-required requirements (sage-setup + sage_conf).

[kiwifb]

I know you are having difficulties with this but I cannot really agree to that while there is a separate sage_setup package. It is really a kind of vendoring. And I spent some time organising stuff in Gentoo so those parts move smoothly together as separate packages. I really would need to remove the whole sage_setup as a separate package to rebalance thing.

[tornaria]

I know you are having difficulties with this but I cannot really agree to that while there is a separate sage_setup package. It is really a kind of vendoring. And I spent some time organising stuff in Gentoo so those parts move smoothly together as separate packages. I really would need to remove the whole sage_setup as a separate package to rebalance thing.

The issue is that sage_setup is not really a separate package. Building sagemath-standard is very, very intertwined with the exact version of sage-setup. Lots of changes happen in sync, refactoring from one to the other, etc. This means any change in one affects the other, etc.

The situation would be different if sage-setup was a separate project with different release schedule (separate repo, etc) and with a clear-cut API between one and the other, but it doesn't really work like that.

For a tiny example: can someone tell me how I can build sagemath-standard from #37270 using python 3.12? Given that #37270 changes metadata for both sagemath-standard and sage-setup to allow 3.12, it should be possible but it isn't.

BTW, don't get me started with circular dependencies: sage-setup depends on sagemath-standard even if it's not declared afaict.

[mkoeppe]

circular dependencies: sage-setup depends on sagemath-standard even if it's not declared afaict.

sage_setup does import sage.env and sage.misc.package_dir.
To access these packages, the current version of sagemath-standard setup.py makes the source of the package being installed importable - https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-standard/setup.py#L31

The correct solution for this is

• for sage-setup to declare a dependency (install-requires) on sagemath-environment, see https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-environment/MANIFEST.in,
• or, equivalently, for sagemath-standard to declare a build dependency (build-system requires) on sagemath-environment.

The modularized distributions sagemath-objects etc. already do the latter: https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-objects/pyproject.toml.m4#L8

If you'd like to know why this correct solution is not yet implemented, you don't need to look much farther than #36951

[kiwifb]

Yes, sage_setup has some run time dependencies on sage to build sage stuff. Maybe some of the stuff from sage.misc.packagedir should be moved across. But sage is not a build time dependency of sage_setup at all. It will install quite fine without sage being there.

[tornaria]

In any case, this solves a real issue that I've raised, and nobody has shown any real downsides (as in: breaks this or that).

[mkoeppe]

In any case, this solves a real issue that I've raised, and nobody has shown any real downsides

As you know, that's not the standard that we use in this project.

[tornaria]

In any case, this solves a real issue that I've raised, and nobody has shown any real downsides

As you know, that's not the standard that we use in this project.

Please be respectful.

[tobiasdiez]

I think this is a great idea. sage_setup is just a bunch of scripts with very little potential to be helpful to anyone outside of the sage-xyz packages (eg who else would like to auto-generate interpreters?). Inlining it streamlines the installation and is eg a step towards pip installation with build isolation.

@tornaria Could you please also remove the sage-setup mentions in the conda install docs and the conda ci. Otherwise this is good to go from my side.

[mkoeppe]

eg who else would like to auto-generate interpreters?

The modularized distributions sagemath-* do.

[mkoeppe]

a step towards pip installation with build isolation.

this is nonsense.

[tornaria]

I think this is a great idea. sage_setup is just a bunch of scripts with very little potential to be helpful to anyone outside of the sage-xyz packages (eg who else would like to auto-generate interpreters?). Inlining it streamlines the installation and is eg a step towards pip installation with build isolation.

@tornaria Could you please also remove the sage-setup mentions in the conda install docs and the conda ci. Otherwise this is good to go from my side.

Let me know if what I did works for you. I don't use conda myself.

[github-actions]

Documentation preview for this PR (built with commit 0c8d742; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[tornaria]

Documentation preview for this PR (built with commit e280cba; changes) is ready! 🎉

Could we please disable the spam bot?

[tobiasdiez]

Thanks, this works well indeed. CI is also passing, so good to go from my side!

[tobiasdiez]

a step towards pip installation with build isolation.

this is nonsense.

Then please explain how pip install src works with build isolation and a separate sage_setup package.

[tornaria]

I know you are having difficulties with this but I cannot really agree to that while there is a separate sage_setup package. It is really a kind of vendoring. And I spent some time organising stuff in Gentoo so those parts move smoothly together as separate packages. I really would need to remove the whole sage_setup as a separate package to rebalance thing.

I don't think you need to remove sage_setup as a separate package: it should not interfere with building sagelib / sagemath-standard. It will still be needed to build the split distributions.

[tornaria]

So we have a positive review (@tobiasdiez) and a -1 (@mkoeppe). Not sure how to count @kiwifb (neutral?)

I believe negative reviews are not a thing, but we should still give Matthias the chance to:

• show an issue caused by this PR
• propose an alternative

Let's try to move forward in a way that satisfies everybody.

[mkoeppe]

@tornaria If your workflow has a problem with sage-setup being a "build-system requires" of sagemath-standard, then you are going to have the same problem with the modularization of the Sage library.

• https://github.com/sagemath/sage/blob/develop/pkgs/sagemath-categories/pyproject.toml.m4#L7

[mkoeppe]

@tornaria I previously asked on another ticket whether you can point us to policy documents in voidlinux that would help us understand where these rigid rules ("package A cannot build-depend on a specific version of package B") that you seem to be citing here and in similar tickets are coming from. This would be really helpful.

[mkoeppe]

In any case, this solves a real issue that I've raised, and nobody has shown any real downsides

As you know, that's not the standard that we use in this project.

@tornaria To elaborate: The standard in our project is to go towards best practices, not away from them.

One such best practice in packaging is to ship the same file in exactly one distribution, not several. sagemath-standard of course already violates that, as it ships files that are also shipped by the modularized distributions sagemath-environment, sagemath-objects, etc.

Making sage-setup a separate distribution and NOT ship it with sagemath-standard -- that was a move towards best practices, as it is needed by multiple distributions.

The path forward is to remove the existing duplications, making sagemath-standard depend on the modularized distributions. In the branch of #35095 you can see it in its final form: sagemath-standard does not ship anything; it only depends on the modularized distributions.

[kwankyu]

... it will allow me to switch to build from the sagemath-standard sdist instead of having to build from the git repo tarball. This is more compatible with the principles of modularization.

Note that a lot of people have been working very hard for more than 15 years to make sagemath more modular with the goal of having distro packages of sagemath.

You seem to use the word "modularization" to mean the efforts to break the "batteries-included"ness of sage. Yes, by the efforts of 15 or so years, sage became to use system packages and system pip packages instead of carrying them. That is nice.

... or explain in which ways my approach here conflicts with modularization so I can improve it.

In this PR, the "modularization" should mean making distribution packages (=pip-installable packages) splitting the sage library. I meant conflicts with the design of this modularization. Obviously, this PR goes against with the design. There's no way to improve this PR to avoid the conflict. My question is whether the cause of your PR is worth to change the design.

[kwankyu]

@kwankyu with this PR, mkoeppe's modularisation, admittedly a low-priority task compared with the need to have up to date Sage in major and not so major Linux etc distros, would need some amends. So what?

If you want to help the author, you better use this argument to recruit positive voters from sage-devel, instead of attacking negative voters.

[tornaria]

Most distributions ship a single version of a given package. In this case, we would be shipping a package sage-setup that is unable to build any version of sagemath-standard except the exact same version. That is a useless package from the point of view of a distribution, and I'm afraid it won't be accepted (it's the kind of things that make distros unwilling to package sagemath).
Moreover, sage-setup is unusable and untestable without sagemath-standard. We take testing seriously and it's quite important for us to test the software in our setting (with the particular packages we ship, etc).

For next version up in distro packaging, both should be deployed together. Right?

If there are two separate packages, the distro needs to build and test each one in sequence. The 1st package could be installed when the 2nd package is built, but we cannot require the 2nd package to be installed when we build the 1st one, as that would lead to a bootstrapping problem.

In this case, we would do:

• 1st package: sage-setup, cannot depend on sagemath-standard; can be built but not tested.
• 2nd package: sagemath-standard, can depend on sage-setup; can be built and tested.

The 1st package could "checkdepend" on the 2nd package. But this would not work when 1st and 2nd package have to be updated in sync as is the case for sagemath.

Circular dependencies like this are troublesome and often lead to untested software.

Let's say that sage-setup 10.3 and sagemath-standard 10.3 are the deployed distro packages. Now you start to prepare sage-setup 10.4 and sagemath-standard 10.4. The corresponding distribution packages are in pypi.

Now by "circular dependencies", do you mean this:

(1) The deployed distro package is still sage-setup 10.3. (2) but you want to test the new distro package sagemath-standard 10.4. (3) There is no deployed distro package sage-setup 10.4. (4) Hence you cannot test the new distro package sagemath-standard 10.4.

(a) The deployed distro package is still sagemath-standard 10.3. (b) but you want to test the new distro package sage-setup 10.4. (c) There is no deployed distro package sagemath-standard 10.4. (d) Hence you cannot test the new distro package sage-setup 10.4.

Yes those two are problems of the current design. The issue also affects all versions 10.4.beta*. This is not unimportant. I do track prereleases and make draft distro packages for them (e.g., void-linux/void-packages#49571). This is quite important as it means I'm preemptively discovering issues and this is a great feedback loop for sagemath. I also build on git branches to test and for the PRs that affect build system is important that the way I build on a git checkout is identical to the way I build on the distro packages.

As an example, let's look at jupyter, which is a valid comparison being very highly modularized:

• There is hatch-jupyter-builder, currently on version 0.9.1. This is a standalone package that can be tested on its own. Some jupyter packages (notebook, jupyter-server, jupyterlab, etc) depend on it for building, but not on a specific version. E.g., notebook depends on hatch-jupyter-builder>=0.5, jupyter-server depends on hatch-jupyter-builder>=0.8.1, and jupyterlab on hatch-jupyter-builder>=0.3.2. In particular, the version of hatch-jupyter-builder` that we ship in the distro can be used to build releases but also to build from the latest git tip. When a new feature is needed, they can first release a new version of the builder and then start using it.

... it will allow me to switch to build from the sagemath-standard sdist instead of having to build from the git repo tarball. This is more compatible with the principles of modularization.

Note that a lot of people have been working very hard for more than 15 years to make sagemath more modular with the goal of having distro packages of sagemath.

You seem to use the word "modularization" to mean the efforts to break the "batteries-included"ness of sage. Yes, by the efforts of 15 or so years, sage became to use system packages and system pip packages instead of carrying them. That is nice.

No, that's not what I mean. Or rather, not only that. There's also lots of functionality that used to be part of sage that has been modularized into separate packages. Some core examples: cysignals, cypari2, etc. These are good examples: they are packages that were "modularized" out of sagemath, they have independent releases, they can be used and tested independently of sagemath, etc.

... or explain in which ways my approach here conflicts with modularization so I can improve it.

In this PR, the "modularization" should mean making distribution packages (=pip-installable packages) splitting the sage library. I meant conflicts with the design of this modularization. Obviously, this PR goes against with the design.

"Obviously" is a subjective word. Can you spell out in which way this PR conflicts or goes against "modularization"?
(I'm using quotes like you just to indicate the particular definition you mention above)

Indeed, some of my ideas and opinions might conflict with "modularization". But I do respect the ideals of modularization, I made this PR taking that into account, and I really don't think there's any conflict here.

There's no way to improve this PR to avoid the conflict.

I can't argue with this since I still don't know what is the conflict. I don't think there's any conflict.

My question is whether the cause of your PR is worth to change the design.

Again: I do think that the design should be open for a (technical) discussion, yes. But this PR is not that.

[tornaria]

@dimpase please chill out. I appreciate your concerns, opinions, and collaboration, when you do it constructively (which you do a lot, the same as @kwankyu and @mkoeppe).

[orlitzky]

For next version up in distro packaging, both should be deployed together. Right?

If there are two separate packages, the distro needs to build and test each one in sequence. The 1st package could be installed when the 2nd package is built, but we cannot require the 2nd package to be installed when we build the 1st one, as that would lead to a bootstrapping problem.

We have similar problems in Gentoo but they affect the end users. Installing a package on Gentoo builds it from source and (optionally) tests it. If you can't test one package without the other, you have to install both without testing, and then reinstall them both with testing. That not only doubles the amount of time involved, but can possibly leave you with a broken package -- once you know that the tests fail, it's too late -- the new version is already installed.

[kwankyu]

No, that's not what I mean. Or rather, not only that. There's also lots of functionality that used to be part of sage that has been modularized into separate packages. Some core examples: cysignals, cypari2, etc. These are good examples: they are packages that were "modularized" out of sagemath, they have independent releases, they can be used and tested independently of sagemath, etc.

Right. I forgot to include that.

[kwankyu]

In this case, we would do:

• 1st package: sage-setup, cannot depend on sagemath-standard; can be built but not tested.
• 2nd package: sagemath-standard, can depend on sage-setup; can be built and tested.

The 1st package could "checkdepend" on the 2nd package. But this would not work when 1st and 2nd package have to be updated in sync as is the case for sagemath.

Yes those two are problems of the current design. The issue also affects all versions 10.4.beta*. This is not unimportant. I do track prereleases and make draft distro packages for them (e.g., void-linux/void-packages#49571). This is quite important as it means I'm preemptively discovering issues and this is a great feedback loop for sagemath. I also build on git branches to test and for the PRs that affect build system is important that the way I build on a git checkout is identical to the way I build on the distro packages.

I agree that these (testing, and testing betas) are important.

If there are two separate packages, the distro needs to build and test each one in sequence. The 1st package could be installed when the 2nd package is built, but we cannot require the 2nd package to be installed when we build the 1st one, as that would lead to a bootstrapping problem.

We have similar problems in Gentoo but they affect the end users. Installing a package on Gentoo builds it from source and (optionally) tests it. If you can't test one package without the other, you have to install both without testing, and then reinstall them both with testing. That not only doubles the amount of time involved, but can possibly leave you with a broken package -- once you know that the tests fail, it's too late -- the new version is already installed.

So the problem affects sagemath distro package on all distros if sagemath-standard sdist is used for the packaging.

@mkoeppe I think that the above difficulty of packagers could be a cause enough to change the design of the modularization. I hope that having sage-setup as a separate distribution package is not essential in the design. Perhaps sage-setup can be included to every distribution packages. Would this work?

[orlitzky]

I should clarify that I don't oppose the existence of a sage-setup (and I have not voted here, because Francois is the only one who has to deal with this at the moment). Lots of packages have build dependencies, and sage-setup is, in theory, just another build dependency. I mentioned how it works on Gentoo because I don't want you to get the idea that Gonzalo is doing something weird/wrong. Any time you have packages with circular dependencies it is a headache for packagers and users, and indicates that maybe those packages were not split optimally.

[mkoeppe]

sage-setup is unusable and untestable without sagemath-standard.

@tornaria That's false. It's testable using just sagemath-repl, which provides the doctester.

The whole talk about "circular dependencies" -- I already replied to it 2 months ago in #37287 (comment). In short: sage-setup does not needs sagemath-standard. It only needs sagemath-environment at runtime.

(That that is not declared as a runtime dependency is simply because sage-setup still caters to the monolithic Sage library, in which sagemath-environment is not a thing.)

[kwankyu]

sage-setup is unusable and untestable without sagemath-standard.

@tornaria That's false. It's testable using just sagemath-repl, which provides the doctester.

He seems to mean the testing of the distro package sage-setup for the linux platform, which is a different thing from testing the distribution package.

[tornaria]

If there are two separate packages, the distro needs to build and test each one in sequence. The 1st package could be installed when the 2nd package is built, but we cannot require the 2nd package to be installed when we build the 1st one, as that would lead to a bootstrapping problem.

We have similar problems in Gentoo but they affect the end users. Installing a package on Gentoo builds it from source and (optionally) tests it. If you can't test one package without the other, you have to install both without testing, and then reinstall them both with testing. That not only doubles the amount of time involved, but can possibly leave you with a broken package -- once you know that the tests fail, it's too late -- the new version is already installed.

So the problem affects sagemath distro package on all distros if sagemath-standard sdist is used for the packaging.

Indeed.

@mkoeppe I think that the above difficulty of packagers could be a cause enough to change the design of the modularization. I hope that having sage-setup as a separate distribution package is not essential in the design. Perhaps sage-setup can be included to every distribution packages. Would this work?

In fact, I will echo what @orlitzky said: having sage-setup split is not a problem per se. It would be just fine to have sage-setup as a separate package, but a separate package has to be a separate package (a bit tautological, isn't it?).

The sagemath source code is very large, and split in a large hierarchy of very many packages/modules. But for legacy reasons, this separation is far from perfect, in the sense that there is no clear separation of concerns, clear ownership, etc. and there are lots of interdependencies (explicit and implicit). This also affects sage-setup, which is the main reason why we can't use a fixed version of sage-setup to build several versions of sagemath-standard (we could, but then it would be difficult to work on the setup code without breaking something).

The more I think on the issue, the more I lean towards the idea that the split of sage-setup was premature. A good split would be identifying the features and requirements of building sagemath that are general and make sense to split in a separate package, design and implement such a separate package (using the existing code, of course), and then switch sagemath building to that separate package.

On the other hand, it seems setuptools is showing its limitations for building sagemath, and even with all the disagreements it seems that there is some kind of consensus that a move to meson migth be the best way. Then maybe investing much effort in sage-setup is really not worth it, and we can use this work to design a better meson build system that is suitable for building separate modules while being nice for distro packaging.

Thus: can we please have this -- think of it as a temporary workaround? There might be some philosophical concern in the fact that we would have some code that is repeated in more than one sdist on pypi. However (a) the repeated code comes from a single source tree (no code duplication in that sense) and (b) the repeated code will not be shipped by any binary distribution other than sage-setup (avoiding any overlap or conflicts).

Note: the sdist for sagemath-standard is I think about 19-20M, and adding sage-setup makes it maybe 100-200K larger. But the git repo tarball that I have to use to build is about 30M, so if source size is a concern this is actually better.

@tornaria That's false. It's testable using just sagemath-repl, which provides the doctester.

This sounds a little bit agressive. We are both making an effort to communicate in better terms, so let's try to keep it cool and learn to agree and disagree.

You know what I mean. I'm not currently packaging sagemath-repl since that way of packaging is not ready and not working and, AFAIK, sagemath-repl and sagemath-standard do overlap in the binary distributions. Am I mistaken?

[mkoeppe]

The sagemath source code is very large, and split in a large hierarchy of very many packages/modules. But for legacy reasons, this separation is far from perfect, in the sense that there is no clear separation of concerns, clear ownership, etc. and there are lots of interdependencies (explicit and implicit). This also affects sage-setup, [...]

The more I think on the issue, the more I lean towards the idea that the split of sage-setup was premature. A good split would be identifying the features and requirements of building sagemath that are general and make sense to split in a separate package, design and implement such a separate package (using the existing code, of course), and then switch sagemath building to that separate package.

You seem to be missing that doing this work -- figuring out where and how to split it, and implementing it -- is exactly what I have been doing over the past 4 years.

[kwankyu]

Thus: can we please have this -- think of it as a temporary workaround? There might be some philosophical concern in the fact that we would have some code that is repeated in more than one sdist on pypi. However (a) the repeated code comes from a single source tree (no code duplication in that sense) and (b) the repeated code will not be shipped by any binary distribution other than sage-setup (avoiding any overlap or conflicts).

As I now understand the situation, having this PR clears up some difficulty in preparing sagemath distro package from sagemath-standard sdist package, but breaks the design of the modularization. The broken part of the design seems "philosophical" or "getting away from the standard". We have to choose between this PR and the current design. Fortunately, moving to meson would clear up the packager's difficulty without getting away from the standard. I don't know the technical details, but all except me seem to understand this.

Based on this, I now withdraw my vote. Consider my vote as 0.

[mkoeppe]

I'm not currently packaging sagemath-repl since that way of packaging is not ready and not working

There's nothing wrong with providing a downstream package for sagemath-repl.

and, AFAIK, sagemath-repl and sagemath-standard do overlap in the binary distributions.

They do overlap currently, so in typical downstream packaging, you would mark them as mutual conflicts.
Or you could be fancy and remove the overlap and thus the conflict, and declare your downstream package of sagemath-repl a dependency.

[tornaria]

Setting back to positive review given the vote is 5-1:
👍 @tornaria (as the author)
👍 @tobiasdiez
👍 @dimpase
👍 @dkwo
👍 @saraedum
👎 @mkoeppe

Withdrawn votes:

• @kiwifb in sagemath-standard: include sage_setup in sdist #37287 (comment)
• @kwankyu in sagemath-standard: include sage_setup in sdist #37287 (comment))

To avoid merge conflicts, I've rebased this PR on top of #37138 and #37796 (both with positive review). In both cases the merge was trivial but not automatic.

The CI failures are unrelated.