Use the system GAP #36792

orlitzky · 2023-11-30T13:00:26Z

Resurrect #29644 and add an spkg-configure.m4 for GAP.

Some other changes were made to make this possible / nicer:

GAP_LIB_DIR and GAP_SHARE_DIR are merged into GAP_ROOT_PATHS. This was suggested by @tornaria and is the right approach, especially now that GAP_SO has been removed. Nothing else needs those two directories and GAP itself doesn't care about them -- it only cares about the package search path, i.e. GAP_ROOT_PATHS. So changing the variable(s) brings us in line with the way GAP works.
We no longer pass the -r flag to GAP. Particularly when using the system GAP, we do want the user to be able to install his own packages and run his own initialization.
We begin to pass -A to GAP. This tells GAP not to autoload the big set of "recommended" packages at start-up, which avoids the inevitable error messages when those packages are not installed on the system GAP. We could check for all of them in spkg-configure.m4, but it's a long list, and loading them slows down GAP initialization for no benefit -- Sage itself uses only one such package. Users can autoload them via gaprc or gap.ini in light of (2).
After adding -A to the initialization, we try to load the PolyCyclic package if it's installed. This is the one "recommended" package that Sage itself uses.
The "recommended" packages are all moved from the gap SPKG to gap_packages because we no longer need them, except (maybe) for PolyCyclic. Should keep polycyclic installed by default? The tests all pass without it, but it is used a few places in sagelib.
Various stale doctest fixes.
The expected output from a few tests has changed. Where possible, I've made the test more robust. In one case I had to drop the printing of a matrix, because if you dig into the source code for GAP's NormalSubgroups(), it chooses a Representative() inconsistently, and that eventually affects the entries of the matrix.

All tests are passing afterwards... except one, a heisenbug:

sage -t --warn-long 187.7 --random-seed=96688270013898650573232209016248663009 src/sage/groups/matrix_gps/finitely_generated_gap.py
**********************************************************************
File "src/sage/groups/matrix_gps/finitely_generated_gap.py", line 123, in sage.groups.matrix_gps.finitely_generated_gap.FinitelyGeneratedMatrixGroup_gap.as_permutation_group
Failed example:
    P == Psmaller
Expected:
    False
Got:
    True
**********************************************************************

If anyone knows what's going on there, I'd be grateful. The GAP docs for SmallerDegreePermutationRepresentation() say,

The methods used might involve the use of random elements and the permutation representation (or even the degree of the representation) is not guaranteed to be the same for different calls of SmallerDegreePermutationRepresentation.

So maybe this isn't really a bug, but I would find it strange if that randomness could mean that sometimes the smaller degree option just wouldn't work at all.

Anyway, have at it, and let me know what you think.

dimpase · 2023-11-30T15:36:14Z

is there GAP on a Gentoo overlay?

dimpase · 2023-11-30T15:39:38Z

these "recommended" packages better be installed on the system. Otherwise it's hard to predict exact results (they amend behaviour of some GAP routines)

tobiasdiez

I cannot say much about the gap interface, so just two general suggestions.

Also the (conda) tests are failing (there are more, but most of them are false positives as #36372 was not applied):

File "src/doc/en/reference/spkg/gap_packages.rst", line 18, in doc.en.reference.spkg.gap_packages
Failed example:
    gap.eval('LoadPackage("Grape")')
Expected nothing
Got:
    'fail'
**********************************************************************
File "src/doc/en/reference/spkg/gap_packages.rst", line 19, in doc.en.reference.spkg.gap_packages
Failed example:
    libgap.LoadPackage("Grape")
Expected nothing
Got:
    #I  grape package is not available. Check that the name is correct
    #I  and it is present in one of the GAP root directories (see '??RootPaths')
    fail

tobiasdiez · 2023-11-30T16:09:35Z

src/sage/env.py

+# in the usual $PATH) over the system ones. In any case, the only
+# thing to watch out for here is the possibility of the empty string
+# after calling split(";").
+GAP_ROOT_PATHS = ";".join([join(SAGE_LOCAL, "lib", "gap"),


I propose to store this as an array of strings (without empty ones) and not as a string with semicolons. Makes some of the logic below more easy to read.

I'll think about this along with @tornaria's suggestion about overriding GAP_ROOT_PATHS. GAP itself uses a semicolon-separated string, and the semicolons have meaning beyond just separating the paths. At the beginning of the string they mean "append these paths" and at the end of the string a semicolon means "prepend these paths." I think it would be more awkward to encode that magic behavior as an empty string in a list than the current level of awkwardness.

I moved the GAP_ROOT_PATHS mangling from sage.env into sage-conf and I still don't think using a list here would help, for two main reasons:

The semicolon appending/prepending issue I already mentioned. How to we encode a leading or trailing semicolon in a python list of strings?

Both the source (./configure, and hopefully soon, meson) and the sink (gap -l) for GAP_ROOT_PATHS use a semicolon-separated string. Neither can use a python list, so we would have to parse the string we get from the build system into a python list, and then immediately convert back to a semicolon string in sage.env. A list would be a tiny bit nicer for a human to type into conf.py by hand, but I think we all want to stop doing that ASAP.

My proposal was to have something like GAP_ROOT_PATHS = var(...).split(';') in sage.env, but since you now need it as often as a colon-separated list and as a list of strings (and given the appending/prepending issue), I no longer think this makes sense.

I think it's better (less confusing) that sage.env.GAP_ROOT_PATHS matches the environment variable GAP_ROOT_PATHS, which is by definition a string.

tobiasdiez · 2023-11-30T16:10:41Z

src/sage/env.py

+# Passing -A allows us to use a minimal GAP installation without
+# producing errors at start-up. The files sage.g and sage.gaprc can be
+# used to load any additional packages that may be available.
+_gap_cmd  = f'gap -A -l "{GAP_ROOT_PATHS}"'


Can we move this logic of determining the fallback of the gap command to interfaces/gap_workspace.py? It feels like that there is more context why this default command is the correct choice.

I think this is here so it can be overriden by environment SAGE_GAP_COMMAND

Right, the whole chunk is,

_gap_cmd = f'gap -A -l "{GAP_ROOT_PATHS}"' if SAGE_GAP_MEMORY is not None: _gap_cmd += " -s " + SAGE_GAP_MEMORY + " -o " + SAGE_GAP_MEMORY SAGE_GAP_COMMAND = var('SAGE_GAP_COMMAND', _gap_cmd)

So _gap_cmd is being used as the default for SAGE_GAP_COMMAND. (And thus needs to be set before that call to var).

Maybe it's reasonable to have here just:

SAGE_GAP_COMMAND = var("SAGE_GAP_COMMAND")

and move the logic for the "default" gap command to interfaces/gap_workspace.py as something like:

gap_cmd = SAGE_GAP_COMMAND if gap_cmd is None: gap_cmd = f'gap -A -l "{GAP_ROOT_PATHS}"' if SAGE_GAP_MEMORY is not None: gap_cmd += " -s " + SAGE_GAP_MEMORY + " -o " + SAGE_GAP_MEMORY

Yes, thanks @tornaria, that's what I had in mind.

Ok, done. Although I wound up moving it to sage.interfaces.gap (and not sage.interface.gap_workspace) because that's where most of the existing gap command processing takes place.

tornaria · 2023-11-30T16:54:04Z

src/sage/env.py

+GAP_ROOT_PATHS = ";".join([join(SAGE_LOCAL, "lib", "gap"),
+                           join(SAGE_SHARE, "gap"),
+                           var("GAP_ROOT_PATHS", "")])


This seems to make it impossible to really override the value of GAP_ROOT_PATHS. If either SAGE_LOCAL/lib/gap or SAGE_SHARE/gap is undesired for some reason, they are nevertheless forced.

Also, can we do SAGE_LOCAL/share/gap instead of SAGE_SHARE/gap? For a regular installation of sage these two are the same, but for a system wide gap they are not.

For instance, I define SAGE_SHARE to be /usr/share/sagemath/ and I place sagemath specific data inside (e.g. all databases that are only used from sage). And for a system sagemath we possibly have SAGE_LOCAL=/usr (taken from SAGE_VENV which itself defaults to sys.prefix)

(ideally, SAGE_SHARE can be phased out as in conway_polynomials, etc.)

I do have a similar setup for SAGE_SHARE, I am hoping it is on its way out in the near future but yes, I do a similar distinction and patching of some paths for it. So, I second the request of @tornaria

Also, can we do SAGE_LOCAL/share/gap instead of SAGE_SHARE/gap? For a regular installation of sage these two are the same, but for a system wide gap they are not.

Done. Our sdh_configure() doesn't specify --datadir anyway so it's literally using $SAGE_LOCAL/share.

This seems to make it impossible to really override the value of GAP_ROOT_PATHS. If either SAGE_LOCAL/lib/gap or SAGE_SHARE/gap is undesired for some reason, they are nevertheless forced.

Done. I moved the hackiness out of sage.env and into sage-conf.

dimpase · 2023-11-30T17:05:49Z

regrading SmallerDegree... failing test, it's really an heuristic. (Finding the smallest one would mean finding a subgroup of minimal index, a hard task)

It might be affected by the set of installed GAP packages too.

dimpase · 2023-11-30T17:13:04Z

not loading recommend GAP packages is akin to using Python without ssl module. It sort of works, but YMMV.

pkgs/sage-conf/_sage_conf/_conf.py.in

orlitzky · 2023-11-30T21:27:11Z

is there GAP on a Gentoo overlay?

Yes, thanks to Francois: https://github.com/cschwan/sage-on-gentoo

orlitzky · 2023-11-30T21:32:08Z

regrading SmallerDegree... failing test, it's really an heuristic. (Finding the smallest one would mean finding a subgroup of minimal index, a hard task)

It might be affected by the set of installed GAP packages too.

It works in a CLI session but not while doctesting with the same set of packages. Big WTF. I'd ultimately be OK with declaring it not an error, it just strikes me as particularly suspicious.

orlitzky · 2023-11-30T22:16:30Z

not loading recommend GAP packages is akin to using Python without ssl module. It sort of works, but YMMV.

Everything I know about GAP I learned in the past week, so I'll listen to the experts on this. FWIW my motivation was:

Reduce the start-up time. There are 18 fewer packages to load whenever libgap is used for the first time.
Support minimal installations. On Gentoo, those 18 packages are dependencies that would otherwise be unnecessary to run sage or for its test suite to pass.
Make the ./configure (and eventually, meson) check easier. It's a lot easier to check for 3 packages than 21.

orlitzky · 2023-11-30T22:29:59Z

File "src/doc/en/reference/spkg/gap_packages.rst", line 18, in doc.en.reference.spkg.gap_packages
Failed example:
    gap.eval('LoadPackage("Grape")')
Expected nothing
Got:
    'fail'
**********************************************************************
File "src/doc/en/reference/spkg/gap_packages.rst", line 19, in doc.en.reference.spkg.gap_packages
Failed example:
    libgap.LoadPackage("Grape")
Expected nothing
Got:
    #I  grape package is not available. Check that the name is correct
    #I  and it is present in one of the GAP root directories (see '??RootPaths')
    fail

Fixed this one. I forgot SPKG.rst was doctested.

dimpase · 2023-12-01T09:05:15Z

not loading recommend GAP packages is akin to using Python without ssl module. It sort of works, but YMMV.

Everything I know about GAP I learned in the past week, so I'll listen to the experts on this. FWIW my motivation was:
1. Reduce the start-up time. There are 18 fewer packages to load whenever libgap is used for the first time.

only for the 1st time. Then it's in the workspace which we load.

2. Support minimal installations. On Gentoo, those 18 packages are dependencies that would otherwise be unnecessary to run sage or for its test suite to pass.

GAP does not test such super-minimal GAP configurations, nor should we.

3. Make the `./configure` (and eventually, meson) check easier. It's a lot easier to check for 3 packages than 21.

write a loop which runs 21 times isn't harder than a loop which runs 3 times :-)

Please don't let the packaging convenience override the intentions of GAP releases. They have these extra packages out of necessity, for historical reasons - people preferred adding packages to contributing to the core GAP, as GAP was telling the users that publishing GAP packages is like publishing papers - and contributors liked it an did that.
These "recommended" packages are an integral part of GAP proper.

dimpase

please undo this package reshuffling.

build/pkgs/gap_packages/spkg-install.in

src/sage/env.py

orlitzky · 2023-12-02T11:59:01Z

All remaining conda CI failures are fixed in #36372

tobiasdiez

LGTM, but I leave setting it to "positive-review" to the gap experts.

dimpase · 2023-12-03T15:13:03Z

we need to test for presence of system-wide library libgap, with headers (e.g., test for <gap/gap.h>), not only the executable gap.

dimpase · 2023-12-03T15:30:01Z

Please also add distros/gentoo.txt (and perhaps for some other distros?) for gap and gap_packages, and libsemigroups). Figuring out what packages to install to cover all gap_packages isn't trivial. So far I only know about sci-libs/libsemigroups dev-gap/semigroups (the latter does install few other GAP packages)

When the system GAP is installed and the SPKG gap is not, the extra gap_packages need to know where to find sysinfo.gap. We can execute a "gap -c ..." command for this.

Based on, * https://packages.fedoraproject.org/pkgs/gap/ * https://build.opensuse.org/package/show/science/gap

With only libgap-dev, we are missing the "gap" executable.

We added this for the CI to try, and it didn't work.

Without this, when certain tests fail, we print a bunch of junk to the screen (and to config.log).

The GAP subpackages listed at, https://packages.fedoraproject.org/pkgs/gap/ are not included with the superpackage install, so we have to list the ones that we use explicitly.

…SPKG If the user doesn't want to use either the sage package manager or his system's package manager, then this should be available immediately.

let's get this is, and look into gap_packages later

dimpase

OK, let's get this in. We'd sort gap_packages later

github-actions · 2023-12-15T01:36:03Z

Documentation preview for this PR (built with commit 404f23a; changes) is ready! 🎉

orlitzky · 2023-12-15T13:18:09Z

Thanks. I've been working on the Gentoo GAP packages this week, trying to get them ready for the main tree.

dimpase · 2023-12-16T10:34:27Z

@antonio-rojas - does this work on Arch?

antonio-rojas · 2023-12-17T10:23:04Z

@antonio-rojas - does this work on Arch?

Works fine for the distro package (as in there are no regressions wrt the current beta). Does this address #31761 ? It looks like gap_reset_workspace still loads all installed packages.

dimpase · 2023-12-17T10:57:49Z

No, #31761 has not been looked at.
This might require more work, but not in this PR.

kiwifb · 2023-12-19T00:15:51Z

I do get an error when building the documentation in sage-on-gentoo. But it may be that I have to remove some patch or settings I had in place before this ticket.

#I  gapdoc package is not available. Check that the name is correct
#I  and it is present in one of the GAP root directories (see '??RootPaths')
#I  GAP: needed package gapdoc cannot be loaded
Error, failed to load needed package `gapdoc' (version >= 1.2)
Syntax warning: Unbound global variable in /usr/share/gap/lib/init.g:740
    ColorPrompt( UserPreference( "UseColorPrompt" ) );
    ^^^^^^^^^^^
Error, SetGasmanMessageStatus: function is not yet defined
Error, Variable: 'L1_IMMUTABLE_ERROR' must have a value

and GAPDoc is installed at /usr/lib64/gap/pkg/gapdoc.

kiwifb · 2023-12-19T00:19:11Z

I need to update sage-conf in sage-on-gentoo, and then my problem should vanish.

orlitzky · 2023-12-19T02:06:35Z

I need to update sage-conf in sage-on-gentoo, and then my problem should vanish.

I think all you have to do is replace GAP_LIB_DIR and GAP_SHARE_DIR with GAP_ROOT_PATHS (semicolon-separated, how gap itself expects it).

kiwifb · 2023-12-19T04:40:30Z

Yes, that was all it was.

…stalling packages   As observed in sagemath#36792 (comment), the "CI Linux incremental" workflow, which is run when package scripts are updated, is not helpful when testing new or modified `spkg-configure.m4` scripts when the SPKG has been installed: Although we uninstall packages with changed spkg-configure.m4 scripts already, a re-run of configure is not (always?) triggered. So here we invoke it explicitly, using a new makefile target `make reconfigure`. (This new target also simplifies instructions issued by the system package advice facility.)    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36829 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe

orlitzky requested review from dimpase, tornaria, tobiasdiez and kiwifb November 30, 2023 13:00

orlitzky mentioned this pull request Nov 30, 2023

Compile everything with meson #36524

Merged

5 tasks

tobiasdiez suggested changes Nov 30, 2023

View reviewed changes

tornaria reviewed Nov 30, 2023

View reviewed changes

pkgs/sage-conf/_sage_conf/_conf.py.in Outdated Show resolved Hide resolved

dimpase previously requested changes Dec 1, 2023

View reviewed changes

build/pkgs/gap_packages/spkg-install.in Outdated Show resolved Hide resolved

tornaria reviewed Dec 1, 2023

View reviewed changes

src/sage/env.py Outdated Show resolved Hide resolved

orlitzky mentioned this pull request Dec 2, 2023

Fix/ignore all remaining test failures when run under conda #36372

Closed

5 tasks

orlitzky added c: packages: standard s: needs review c: build: configure labels Dec 2, 2023

tobiasdiez approved these changes Dec 3, 2023

View reviewed changes

dimpase added s: needs work and removed s: needs review labels Dec 3, 2023

dimpase self-requested a review December 3, 2023 15:22

orlitzky added 8 commits December 15, 2023 00:20

build/pkgs/gap_packages/spkg-install.in: fix install with system gap

a73d0ec

When the system GAP is installed and the SPKG gap is not, the extra gap_packages need to know where to find sysinfo.gap. We can execute a "gap -c ..." command for this.

tox.ini: it's OK if libgap-dev is missing on ubuntu-focal

c2984f1

build/pkgs/gap: add fedora and opensuse distro information

0102206

Based on, * https://packages.fedoraproject.org/pkgs/gap/ * https://build.opensuse.org/package/show/science/gap

build/pkgs/gap: require "gap" in addition to libgap-dev on debian

bf72457

With only libgap-dev, we are missing the "gap" executable.

build/pkgs/gap: remove opensuse distro package information

e6858a0

We added this for the CI to try, and it didn't work.

build/pkgs/gap: redirect stderr in spkg-configure.m4 tests

b0ec017

Without this, when certain tests fail, we print a bunch of junk to the screen (and to config.log).

build/pkgs/gap: add more fedora package information

5d10bee

The GAP subpackages listed at, https://packages.fedoraproject.org/pkgs/gap/ are not included with the superpackage install, so we have to list the ones that we use explicitly.

build/pkgs/gap{,_packages}: install packagemanager with the gap core …

404f23a

…SPKG If the user doesn't want to use either the sage package manager or his system's package manager, then this should be available immediately.

dimpase force-pushed the system-gap branch from bd7d07e to 404f23a Compare December 15, 2023 00:20

dimpase approved these changes Dec 15, 2023

View reviewed changes

dimpase added s: positive review and removed s: needs work labels Dec 15, 2023

dimpase requested a review from antonio-rojas December 16, 2023 10:34

orlitzky mentioned this pull request Dec 17, 2023

Stop loading all installed GAP packages #31761

Closed

vbraun merged commit d6a2dcc into sagemath:develop Dec 19, 2023
15 of 26 checks passed

mkoeppe added this to the sage-10.3 milestone Dec 19, 2023

github-actions bot removed the s: positive review label Dec 19, 2023

orlitzky deleted the system-gap branch January 24, 2024 14:45

Use the system GAP #36792

Use the system GAP #36792

Conversation

orlitzky commented Nov 30, 2023

dimpase commented Nov 30, 2023

dimpase commented Nov 30, 2023

tobiasdiez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimpase commented Nov 30, 2023

dimpase commented Nov 30, 2023

orlitzky commented Nov 30, 2023

orlitzky commented Nov 30, 2023

orlitzky commented Nov 30, 2023

orlitzky commented Nov 30, 2023

dimpase commented Dec 1, 2023

dimpase left a comment

Choose a reason for hiding this comment

orlitzky commented Dec 2, 2023

tobiasdiez left a comment

Choose a reason for hiding this comment

dimpase commented Dec 3, 2023

dimpase commented Dec 3, 2023

dimpase left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 15, 2023

orlitzky commented Dec 15, 2023

dimpase commented Dec 16, 2023

antonio-rojas commented Dec 17, 2023

dimpase commented Dec 17, 2023

kiwifb commented Dec 19, 2023

kiwifb commented Dec 19, 2023

orlitzky commented Dec 19, 2023

kiwifb commented Dec 19, 2023