Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental gap3_jm package #20107

Closed
stumpc5 opened this issue Feb 24, 2016 · 167 comments
Closed

Add experimental gap3_jm package #20107

stumpc5 opened this issue Feb 24, 2016 · 167 comments

Comments

@stumpc5
Copy link
Contributor

stumpc5 commented Feb 24, 2016

We have been trying to have an optional GAP3 package for a while now. Instead of the official GAP3, we instead use a forked package gap3_jm which has regularly been updated.

Previously, this was made a Sage package (#8906) but it contained binaries, so it was removed again (#19164).

I removed all binaries, and prepared a new tarball at ​http://downloads.findstat.org/gap3-jm5-2015-02-01.tar.gz .

CC: @saliola @tscrim @sagetrac-jmichel @simon-king-jena

Component: packages: experimental

Keywords: gap3, experimental package

Author: Christian Stump

Branch: a0d0e18

Reviewer: Jeroen Demeyer, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/20107

@stumpc5 stumpc5 added this to the sage-7.1 milestone Feb 24, 2016
@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 24, 2016

comment:1

How do I actually find all binaries? Such as

gap3_test$ grep -r -m 1 "^"  . | grep "^Binary file"
Binary file ./pkg/monoid/doc/manual.dvi matches
Binary file ./pkg/anusq/bin/Sq matches
Binary file ./pkg/anusq/doc/sq94.dvi matches
Binary file ./pkg/anusq/doc/sqn92.dvi matches
Binary file ./pkg/nq/bin/nq matches
Binary file ./pkg/cryst/doc/paper.dvi matches
Binary file ./pkg/meataxe/tests/m11.2 matches
Binary file ./pkg/meataxe/tests/Mat9 matches
Binary file ./pkg/meataxe/tests/a2 matches
Binary file ./pkg/meataxe/tests/a1 matches
Binary file ./pkg/meataxe/tests/Mat256 matches
Binary file ./pkg/meataxe/tests/Perm2 matches
Binary file ./pkg/meataxe/tests/Mat25 matches
Binary file ./pkg/meataxe/tests/Mat125 matches
Binary file ./pkg/meataxe/tests/ac.2 matches
Binary file ./pkg/meataxe/tests/Mat67 matches
Binary file ./pkg/meataxe/tests/Perm1 matches
Binary file ./pkg/meataxe/tests/Mat5 matches
Binary file ./pkg/meataxe/tests/C0.2 matches
Binary file ./pkg/meataxe/tests/C0.1 matches
Binary file ./pkg/meataxe/tests/Mat2 matches
Binary file ./pkg/meataxe/tests/ac.1 matches
Binary file ./pkg/meataxe/tests/m11.1 matches
Binary file ./pkg/sisyphos/bin/sis-i586-unknown-linux matches
Binary file ./pkg/sisyphos/doc/sisman.0.8.3.dvi matches
Binary file ./pkg/ve/examples/meout.pb matches
Binary file ./pkg/ve/bin/qme matches
Binary file ./pkg/ve/bin/me matches
Binary file ./pkg/ve/docs/nme.dvi matches
Binary file ./pkg/arep/bin/leonin matches
Binary file ./pkg/arep/bin/desauto matches
Binary file ./pkg/arep/bin/leonout matches
Binary file ./pkg/arep/src/leon/doc/manual.dvi matches
Binary file ./manual.pdf matches
Binary file ./bin/gapdjg.exe matches
Binary file ./bin/gap.mac matches
Binary file ./bin/gap.x86linux64 matches
Binary file ./bin/gapcyg.exe matches
Binary file ./bin/gap.macx86 matches
Binary file ./bin/gap.x86linux matches
Binary file ./bin/gapdjg.pif matches
Binary file ./bin/cygwin1.dll matches

Are any of these okay to keep, such as the dvi files?

@jdemeyer
Copy link

Replying to @stumpc5:

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 25, 2016

comment:3

Replying to @jdemeyer:

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

The development of GAP3 has stopped a long time ago. But there are several GAP3 packages that have never been ported to GAP4, and which are still developed.

From that webpage:

To help people who are just interested in GAP3 because they need a package
which has not been ported to GAP4, I have prepared an easy-to install
minimal GAP3 distribution containing an up-to-date versions of the packages.

So this is the only place where we can find up-to-date versions of the GAP3 package chevie to work with unitary reflection groups and their representation theory.

I fail to see what might be wrong with adding support for those packages using that "random repackaging" since I and several others need them.

I install the packages myself and then use them from within Sage, but since several things inside Sage are about to depend on that package - see #11187, #11010, #10819 - I think it is worth providing the option for others to automatically install that version of GAP3 with the newest chevie package.

@jdemeyer
Copy link

comment:4

Replying to @stumpc5:

Replying to @jdemeyer:

I think this is the problem: it doesn't seem to be the "official" GAP3 package, just some random repackaging of GAP3.

The development of GAP3 has stopped a long time ago. But there are several GAP3 packages that have never been ported to GAP4, and which are still developed.

From that webpage:

To help people who are just interested in GAP3 because they need a package
which has not been ported to GAP4, I have prepared an easy-to install
minimal GAP3 distribution containing an up-to-date versions of the packages.

All the above doesn't explain why you use the repackaged version and not the official GAP 3 release from http://www.gap-system.org/Gap3/Download3/download.html

So this is the only place where we can find up-to-date versions of the GAP3 package chevie to work with unitary reflection groups and their representation theory.

No, the official gap3r4p4 release also contains chevie.

I fail to see what might be wrong with adding support for those packages using that "random repackaging" since I and several others need them.

Because the repackaging seems to made in a messy way, see #19164.

@jdemeyer
Copy link

comment:5

Frank Lübeck's page How to get CHEVIE also says to download the official GAP3 release and doesn't talk about the "repackaged" version.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 25, 2016

comment:6

Replying to @jdemeyer:

Frank Lübeck's page How to get CHEVIE also says to download the official GAP3 release and doesn't talk about the "repackaged" version.

Neither this nor the above is up-to-date. Afaik, Jean Michel is the only person still actively developing chevie and his website is the only place where these new versions can be found. E.g., I found a small bug in chevie last month (so years after the release of gap3r4p4) through my Sage usage -- Jean fixed it within a few hours and posted the new version on his website.

Using any other version of gap3 and chevie will result in running into old mistakes that will slow down my understanding of research problems I compute!

@jdemeyer
Copy link

comment:7

Jean fixed it within a few hours and posted the new version on his website.

With "new version", I guess you mean a new version of the repackaged GAP3?

It should be possible to just get the up-to-date chevie sources without all the extra clutter.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 26, 2016

comment:8

From my email discussion with Jean Michel:

>> These  days I sometimes change files in other places than Chevie, so indeed
>> each  time I change something one must take the whole distribution (however
>> I  almost  never  change  the  C  sources).

> Is the stuff that you change in other places needed for chevie to run?

Well,  yes. There were  patches to the  library loaded when loading Chevie.
Since  I maintain the whole distribution, I  decided it makes more sense to
*do*  these  patches  to  the  library  once  and  for  all (this could fix
undiscovered  bugs in other  packages). So the  correct behaviour of Chevie
depends on having my version of gap3/lib.

@jdemeyer, what do you think how to proceed from here?

@jdemeyer
Copy link

comment:9

First of all, thanks for the clarifications!

I think that somebody (either Jean Michel himself or somebody on this Sage ticket) needs to package a source-only repackaged GAP3.

Further questions to be asked to Jean Michel:

  1. Could he state more clearly on his website that his version is not just a repackaged GAP3, it's really a fork of GAP3 with additional fixes which are not in GAP3 proper.

  2. Can he provide a clean source-only repackaged GAP3?

  3. If no to the previous question: is he willing to host such a source-only repackaged GAP3 on his website, if it is created by some Sage contributor?

@jdemeyer

This comment has been minimized.

@jdemeyer jdemeyer changed the title Add again optional GAP3 package Add optional gap3_jm package Feb 26, 2016
@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 26, 2016

comment:11

Replying to @jdemeyer:

I think that somebody (either Jean Michel himself or somebody on this Sage ticket) needs to package a source-only repackaged GAP3.

I don't understand how this differs from my initial suggestion on this ticket.

@jdemeyer
Copy link

comment:12

Replying to @stumpc5:

I don't understand how this differs from my initial suggestion on this ticket.

It does not differ. It's just that the situation has been clarified, making sure that your initial suggestion is the right thing to do.

@jdemeyer

This comment has been minimized.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 26, 2016

comment:14

Dear Christian,
I cannot post directly on sage-trac so I reply to you.

Further questions to be asked to Jean Michel:

  1. Could he state more clearly on his website that his version is not
    just a repackaged GAP3, it's really a fork of GAP3 with additional fixes
    which are not in GAP3 proper.

OK, I will try to be clear on my webpages. The fact is I do not distribute
a complete GAP3 distribution (all packages, all libraries) to save space.
Perhaps I should distribute a complete distribution and this would obsolete
the St Andrews distribution -- it would go from 15MB to 70MB.

  1. Can he provide a clean source-only repackaged GAP3?

I could try if I understand exactly what this means. But this should
certainly not be the default distribution since my distribution contains
Mac and Windows executables (and executables for some packages which have
executables separate from the gap3 executable). We thus have to agree to a
name for this for-Sage distribution which I will keep separate from the
default distribution.

  1. If no to the previous question: is he willing to host such a source-
    only repackaged GAP3 on his website, if it is created by some Sage
    contributor?

Yes.

@jdemeyer

This comment has been minimized.

@jdemeyer
Copy link

comment:16

Replying to @stumpc5:

I could try if I understand exactly what this means.

It means: delete all binaries (i.e. system-specific) from the package. Make sure that the package has a build system such that it can be built/installed in a standard Unix environment.

We thus have to agree to a name for this for-Sage distribution which I will keep separate from the default distribution.

Well, I wouldn't call it a "for-Sage distribution". Call it a "source distribution". The actual filename doesn't matter much, most likely we will rename it for Sage anyway.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Feb 26, 2016

comment:17

Replying to @jdemeyer:

It means: delete all binaries (i.e. system-specific) from the package. Make sure that the package has a build system such that it can be built/installed in a standard Unix environment.

I will try to do that.

@jdemeyer
Copy link

comment:18

Also (this is mostly a comment for Jean Michel): ideally the binary distribution would be made from the source distribution, not the other way around.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Mar 9, 2016

Branch: u/stumpc5/20107

@stumpc5
Copy link
Contributor Author

stumpc5 commented Mar 9, 2016

comment:20

I prepared a first version by

  1. taking the tarball from 2016-02-01 from Jean Michel's website
  2. deleting all binaries that were listed using grep -r -m 1 "^" . | grep "Binary file"
  3. putting the result into a new tarball with the name gap3-jm5-2015-02-01.tar.gz, this can be downloaded at https://drive.google.com/file/d/0B1POEh8bhq1fLXdfTEZkZjVLOGs/view?usp=sharing
  4. preparing an optional package using this upstream tarball. This was basically a blind trip through the documentation for creating packages + looking at the old optional package patch Optional package for gap3 #8906.

@jdemeyer and @tscrim: I would very much appreciate if you could have a look!

If the linked tarball is okay, I would discuss with Jean Michel whether he could provide such a source distribution directly on his website.


New commits:

5483a58first working version, wandering in the darkness though

@stumpc5
Copy link
Contributor Author

stumpc5 commented Mar 9, 2016

Commit: 5483a58

@stumpc5

This comment has been minimized.

@stumpc5

This comment has been minimized.

@jdemeyer
Copy link

comment:23

This is far from an exhaustive list:

+if [[ $UNAME == 'Linux' ]]; then
+    if [[ $(uname -m) == 'x86_64' ]]; then
+        MAKE_COMMAND='x86linux-gcc64'
+    else
+        MAKE_COMMAND='x86linux-gcc'
+    fi;
+elif [[ $UNAME == 'Darwin' ]]; then
+        MAKE_COMMAND='macosx-gcc'
+fi

Is gap3 only supported on those architectures? In that case, it will need to be an experimental package.

@jdemeyer
Copy link

comment:24

You should check for errors in spkg-install.

@stumpc5
Copy link
Contributor Author

stumpc5 commented Mar 30, 2016

comment:126

Replying to @dimpase:

About packages that need to be built:
these ones which are pretty useless without this ought to be just removed.

Jean, it is okay to comment out the lines

#RequirePackage("anusq");
#RequirePackage("arep");
#RequirePackage("meataxe");
#RequirePackage("nq");
#RequirePackage("sisyphos");
#RequirePackage("ve");

in init.g and delete those folders in gap3/pkg, without getting into any trouble for chevie?

Otherwise, I could also try to build those packages that needs compilation, but already meataxe failed on a first try.

@sagetrac-jmichel
Copy link
Mannequin

sagetrac-jmichel mannequin commented Mar 30, 2016

comment:127

Jean, it is okay to comment out the lines

#RequirePackage("anusq");
#RequirePackage("arep");
#RequirePackage("meataxe");
#RequirePackage("nq");
#RequirePackage("sisyphos");
#RequirePackage("ve");

in init.g and delete those folders in gap3/pkg}}, without getting into any trouble for {{{chevie?

Otherwise, I could also try to build those packages that needs compilation, but already meataxe failed on a first try.

You can comment out all of them but arep.g
The function MovedPointsPerm defined in arep/lib/tools.g is used in various places including chevie.

Generally speaking, arep has some functionality which does not need its binaries. You could delete the subdirectories bin and src of arep if you keep the rest.

In the version I am preparing (complete gap3 distribution), I will make sure all included packages can be compiled
and work on x86 64bits at least.

@sagetrac-jmichel
Copy link
Mannequin

sagetrac-jmichel mannequin commented Mar 30, 2016

comment:128

Earlier releases of GAP4 had GAP3 compatibility modes (a choice of various options).
Did you try any of these?

Of course. They do not deal at all with the operations records which is the crux of the matter.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 30, 2016

Changed commit from 38dbd60 to 7696a2c

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 30, 2016

Branch pushed to git repo; I updated commit sha1. New commits:

d4f3a1dupdated the spkg-src to delete gap3 packages that need compilation
7696a2cadded patch for init.g and fixed typo in patch for gap.sh

@stumpc5
Copy link
Contributor Author

stumpc5 commented Mar 30, 2016

comment:130

Dima, thanks for checking!

I deleted all packages that need compilation. This solved your request on this behalf.

I would appreciate if you could retry to build it on OSX (don't forget to download the new tarball first); at least the two broken symlinks

cp: gap3/pkg/ve/bin/me.exe: No such file or directory
cp: gap3/pkg/ve/bin/qme.exe: No such file or directory

are now gone together with the complete package.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 2, 2016

Changed commit from 7696a2c to 3d123df

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 2, 2016

Branch pushed to git repo; I updated commit sha1. New commits:

3d123dfmerged 20107 to the newest develop

@dimpase
Copy link
Member

dimpase commented Apr 2, 2016

comment:132

it does not fly on OSX:

sage: gap3_console()
             ########            Lehrstuhl D fuer Mathematik, RWTH Aachen
           ###    ####             #######            #########          
          ##         ##           #      ##          ## #     ##         
         ##          #           #       ##             #      ##        
        ##                       ##       #             #      ##        
        ##           #           ##      ##             ##    ##         
        ####        ##            #########             #######          
         #####     ###                                  #                
           ######### #  Version 3 Release 4.4 18 Apr 97 #                
                     #                                  #                
                    ##  Alice Niemeyer, Werner Nickel,  Martin Schoenert 
                   ###  Johannes Meier, Alex Wegner,    Thomas Bischops  
                  ## #  Frank Celler,   Juergen Mnich,  Udo Polis        
                 ##  #  Thomas Breuer,  Goetz Pfeiffer, Hans U. Besche   
                ##   #  Volkmar Felsch, Heiko Theissen, Alexander Hulpke 
               ##    #  Ansgar Kaup,    Akos Seress,    Erzsebet Horvath 
              ##     #  Bettina Eick                                     
              ###   ##  
               ######   For help enter: ?<return>                        
 lib: jm 12Mar2016, src: jm 28Feb2015, sys: macosx gcc
 gap3-jm5 final 21Mar2016 -- see webusers.imj-prg.fr/~jmichel/gap3
 Minimal distribution -- loading the following packages:
--- Loading package arep --------- version 1.0, 16Mar1998--------------------
(C) Sebastian Egner, Markus Pueschel -- Abstract REPresentations
--- Loading package autag -------- version Jan1997---------------------------
Gasman: last bag of type integer (> 2^28) and size       16 has overwritten the free bag.
sage: 

@dimpase
Copy link
Member

dimpase commented Apr 2, 2016

Changed branch from u/stumpc5/20107 to public/20107

@dimpase
Copy link
Member

dimpase commented Apr 2, 2016

comment:133

The reason for this must be that the flag -DSYS_IS_64_BIT is not passed during compilation on OSX. Indeed, gap3_makefile.patch has no ifeq ($(ARCH),64/32) block for Darwin. Fixed in this branch.


New commits:

f5cdde2correct options for OSX make targets

@dimpase
Copy link
Member

dimpase commented Apr 2, 2016

Changed commit from 3d123df to f5cdde2

@dimpase
Copy link
Member

dimpase commented Apr 2, 2016

comment:134

also, is this meant to fail:

$ sage -tp --optional=ccache,gap_packages,mpir,python2,gap3,sage src/sage/combinat/root_system/coxeter_group.py 
too few successful tests, not using stored timings
Running doctests with ID 2016-04-03-00-22-03-2f808575.
Git branch: g3
Using --optional=ccache,gap3,gap_packages,mpir,python2,sage
Doctesting 1 file using 4 threads.
sage -t src/sage/combinat/root_system/coxeter_group.py
**********************************************************************
File "src/sage/combinat/root_system/coxeter_group.py", line 226, in sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class
Failed example:
    W._element_class() is W.element_class                      # optional - gap3
Exception raised:
    Traceback (most recent call last):
      File "/home/dima/software/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/dima/software/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class[1]>", line 1, in <module>
        W._element_class() is W.element_class                      # optional - gap3
      File "sage/structure/parent.pyx", line 854, in sage.structure.parent.Parent.__getattr__ (build/cythonized/sage/structure/parent.c:7994)
        attr = getattr_from_other_class(self, self._category.parent_class, name)
      File "sage/structure/misc.pyx", line 253, in sage.structure.misc.getattr_from_other_class (build/cythonized/sage/structure/misc.c:1666)
        raise dummy_attribute_error
    AttributeError: 'CoxeterMatrixGroup_with_category' object has no attribute '_element_class'
**********************************************************************
1 item had failures:
   1 of   3 in sage.combinat.root_system.coxeter_group.CoxeterGroupAsPermutationGroup._element_class
    [53 tests, 1 failure, 3.95 s]
----------------------------------------------------------------------
sage -t src/sage/combinat/root_system/coxeter_group.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 13.2 seconds
    cpu time: 0.8 seconds
    cumulative wall time: 3.9 seconds

@stumpc5
Copy link
Contributor Author

stumpc5 commented Apr 3, 2016

comment:135

Replying to @dimpase:

Thanks for rechecking and for fixing the make target for osx!

----------------------------------------------------------------------
sage -t src/sage/combinat/root_system/coxeter_group.py  # 1 doctest failed
----------------------------------------------------------------------

I doubt this worked before, I only replaced # optional - chevie by # optional - gap3. It has been there since 2013:

$ git blame -L220,+10 src/sage/combinat/root_system/coxeter_group.py
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 220)         A temporary workaround for compatibility with Sage's
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 221)         permutation groups
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 222) 
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 223)         TESTS::
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 224) 
1452e665 (Jeroen Demeyer 2013-03-02 13:53:59 +0100 225)             sage: W = CoxeterGroup(["H",3])                                  # optional - chevi
1452e665 (Jeroen Demeyer 2013-03-02 13:53:59 +0100 226)             sage: W._element_class() is W.element_class                      # optional - chevi
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 227)             True
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 228)         """
db0cda35 (Anne Schilling 2013-01-22 15:31:57 -0800 229)         return self.element_class

Getting this properly working will be part of our finally-get-reflection-groups-smoothly-in-sage-days next week.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 3, 2016

Changed commit from f5cdde2 to a0d0e18

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Apr 3, 2016

Branch pushed to git repo; I updated commit sha1. New commits:

a0d0e183 trivial doctest fixes

@dimpase
Copy link
Member

dimpase commented Apr 3, 2016

Reviewer: Jeroen Demeyer, Dima Pasechnik

@dimpase dimpase changed the title Add optional gap3_jm package Add experimental gap3_jm package Apr 3, 2016
@stumpc5
Copy link
Contributor Author

stumpc5 commented Apr 4, 2016

comment:138

Many thanks to both of you for your reviewing work on this ticket!

@jdemeyer
Copy link

jdemeyer commented Apr 4, 2016

comment:139

Replying to @stumpc5:

I doubt this worked before, I only replaced # optional - chevie by # optional - gap3.

Exactly. This ticket is not about fixing all doctests, just about adding the package.

@vbraun
Copy link
Member

vbraun commented Apr 4, 2016

Changed branch from public/20107 to a0d0e18

@jdemeyer
Copy link

Changed commit from a0d0e18 to none

@jdemeyer
Copy link

comment:141

gap3 is currently broken: #21722

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants