Add Matsumoto's matroid database and AllMatroids() driver function

[gmou3]

This enables the generation of all matroids of certain number of elements (and, optionally, of certain rank or type). There exist restrictions based on the underlying database (Yoshitake Matsumoto's Database of Matroids: https://www-imai.is.s.u-tokyo.ac.jp/~ymatsu/matroid/index.html). Note that the database files consume ~75MB of space, which drops to ~1.5MB upon a standard compression.

Partially addresses #35464 (the type can be set to paving).

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

[gmou3]

@tscrim

[tscrim]

LGTM overall. The one other comment to consider would be to have an explicit link/reference to the database in AllMatroids.

[github-actions]

Documentation preview for this PR (built with commit e4e2d24; changes) is ready! 🎉

[tscrim]

Thanks. LGTM.

[antonio-rojas]

This doesn't work on an installed sagemath because the data files are not being installed. You need to add them to src/setup.cfg.m4 under [options.package_data]

[gmou3]

Thank you. Can you please check the PR I made for this (#37231)? This issue did not appear for me so I can't witness any functional difference.

[tornaria]

This PR almost doubles the size of src/sage:

$ du -sh src/sage
180M	src/sage
$ du -sh src/sage/ext_data/matroids/
75M	src/sage/ext_data/matroids/

This is not the way databases are handled in sagemath. The normal way would be to make an optional spkg containing the database. The nicer/easier way would be to create a pypi package similar to https://github.com/sagemath/conway-polynomials (btw, in conway-polynomials the actual data files are xz compressed and transparently decompressed as needed)

Also, as @antonio-rojas pointed out, this broke sagemath-standard.

[gmou3]

I am not familiar with your proposed method of resolving this but I could check it out.

Would compressing the files down to ~1.5MB be satisfactory? Then a decompression (and possibly a deletion of the decompressed file) would need to occur when the function is called.

[tornaria]

I am not familiar with your proposed method of resolving this but I could check it out.

Have a look at https://github.com/sagemath/conway-polynomials. It's an independent python package so it needs some boilerplate but not too much (you could base your package on this, which is quite modern).

The advantage of this is that the database can be used completely independent of sage, can have its own release schedule, etc. This would be a package very easy to install using pip or create a distro package (it is for conway-polynomials).

What I think would be reasonable is that your package exports a function analogous to the AllMatroids() function (with all the search and/or filter functionality you want to provide) but returning the data as python types so it's usable independently of sage (IMO, ideally the database package returns the data as close to the original source as possible).

Then your AllMatroids() function in sagemath just imports the function from this package and wraps the data in sage types (calling the Matroid() constructor, etc).

Would compressing the files down to ~1.5MB be satisfactory?

I still think this is better handled in a separate package. It may seem small, but you have to bear in mind that sage uses a several databases, some standard and some optional, and if we included every database in the source code, it would all add to a lot, not just space but maintaining work, etc. A separate package is easier to maintain and update independently of sagemath (as long as the interface doesn't change) and can be used without sagemath (this might be useful for someone, I don't know).

Then a decompression (and possibly a deletion of the decompressed file) would need to occur when the function is called.

That's not necessary. The way you use the files, python makes it very easy to switch to reading the compressed file, decompressing on the fly. See the function _open_database() here https://github.com/sagemath/conway-polynomials/blob/master/src/conway_polynomials/__init__.py#L94

Most probably you can copy this function _open_database() almost verbatim, except that since you have several data files you would add a filename argument. This function also uses importlib.resources so the files are placed in the python package itself as package data and you don't have to worry at all about file locations, etc. So really, all you need to do in your code is essentially replace open() by _open_database().

I hope this helps. Maybe @orlitzky has something useful to add (he wrote the conway_polynomials python package).

[gmou3]

The _open_database() recommendation sounds good. I don't think that a separate package would be of any other use (unless maybe if I also added the decoding function from the revlex encoding to a list of matroid bases).

[orlitzky]

I also think a separate package is a good idea. A big benefit is that it de-couples the download & upgrade procedures of the thing that changes rarely (the database) from the thing that changes often (the code in sagelib). Users might install the database package once, and then upgrade sagelib twenty times before having to update to the database package. It's nice in that example to not have to re-download and re-install the big database those twenty times.

The database link in the original post isn't working for me right now, but another benefit is that a separate package can have a separate license if those restrictions would cause problems for sage itself.

[tornaria]

@vbraun @mkoeppe what's the way to proceed with this PR?

Is it ok to add 75M to the source code just like that? No trigger warnings? No sage-devel discussion?

This also broke sagemath-standard for a --disable-editable build, and also packages installed via wheels (either from pypi or built from pkgs/sagemath-standard) including distro packages.

The issue is some files not installed, so in-place editable builds work. I guess testing only covers editable builds?

A better way for this is being worked out in #37231, but IMHO this PR should be reverted for the time being.

[mkoeppe]

Is it ok to add 75M to the source code just like that? No trigger warnings? No sage-devel discussion?

I think it was a simple oversight in the review of this PR. I agree that it needs an urgent fix. To me it does not make much of a difference whether to revert the PR or to merge the current version of #37231 (as a hotfix only; not the final state, which should be adding an optional spkg).
I think a more relevant question is: Is it problematic to have the 75M addition in the git history of the develop branch, so should the fix be done by a force push to develop?

$ git clone --single-branch --branch 10.3.beta6 https://github.com/sagemath/sage.git clone-single-branch-10.3.beta6
$ du -sh clone-single-branch-10.3.beta6
549M	clone-single-branch-10.3.beta6
$ git clone --single-branch https://github.com/sagemath/sage.git clone-single-branch
$ du -sh clone-single-branch
625M	clone-single-branch
$ git clone --single-branch --branch ext_data_matroids https://github.com/gmou3/sage.git clone-single-branch-37231
$ du -sh clone-single-branch-37231
547M	clone-single-branch-37231

So -- thanks to the added database's extraordinary compression ratio -- at least in terms of repo size there's not a problem that would warrant a force push. (I don't know if we should be concerned about other potential adverse effects, perhaps on the speed of some git operations.)

[tscrim]

We have no policy about adding database-type files, much less anything that is documented in our dev guide. As such, there is no mistake/oversight with the review.

(Side note: We might want to be careful about the definition of a database. What if I write a bunch of Python functions, one for each object?)

The broken builds in certain situations is a problem, but we don’t have any standard testing for this. This is why we have beta releases.

A straight reversion of this PR would also set a bad precedent IMO; it should be handled according to our standard practices of a followup PR. We can also have @vbraun wait for the next release until the issue is settled as we are all trying to figure out the best way forward.

[mkoeppe]

@tscrim It's not like we're trying to charge you a fine for this.

We might want to be careful about the definition of a database. What if I write a bunch of Python functions, one for each object?

Well, adding 75 MB of source code in one PR would certainly also raise some objections.

[tscrim]

@tscrim It's not like we're trying to charge you a fine for this.

You’re still trying to lay the blame at my feet for this, which is unfair.

[mkoeppe]

I guess testing only covers editable builds?

@tornaria That's right.

Two things that we might want to do:

• CI: Add pull request size labeler / checker #37254
• CI: Add testing of the built distributions (sdists, wheels) #37255

[mkoeppe]

You’re still trying to lay the blame at my feet for this, which is unfair.

It's true that your feet should not be blamed for any of this!