Update classical modular form data on www.lmfdb.org to reflect updates on beta.lmfdb.org (levels 1-24, weights 2-24, trivial character)

[sehlen]

I was under the impression that the database serving www.lmfdb.org should be updated every day or so.
Nevertheless, data I inserted a few days ago does still not appear on www.lmfdb.org, yet it does appear on beta.lmfdb.org

See:
http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/ranges/?level=1-24&weight=2-40&group=0

Screenshot:

Beta:

[AndrewVSutherland]

@sehlen, @edgarcosta We are still in the process of configuring a mechanism for database updates, but, with the exception of knowls, the plan is to keep it infrequent (we need to do it in a controlled way to make sure the update does not happen while someone is in the process of inserting data, which might result in an inconsistent state).

Having said that, it might make sense to put in place a procedure specifically for the database modularforms2. Can you tell us exactly which collections are actually used by the current code (there is a lot of "junk" currently in modularforms2 and we only want to copy the collections that are actually relevant, this are also the collections that should be documented on https://github.com/LMFDB/lmfdb-inventory/blob/master/db-modularforms2.md, per issue LMFDB/lmfdb-inventory#13.

[AndrewVSutherland]

@sehlen So the main issue is that we want to have a review process in place for the data (just as we do for the code). As we have seen several times in the past few weeks (and not just in modular forms) updates to data can break things just as easily as updates to code.

Having said that, I realize that we need to address #1248 as quickly as we can and will continue to adhere to the LMFDB policy of promptly pushing updates that "make things less bad" even if they don't make things perfect. What I would propose in the short term is that when you have a set of updates that you think are ready to be pushed to production (e.g. now), you create a new issue that indicates the scope of the changes (which spaces or range of levels/weights have changed or been added), and I will promptly review them on beta (I may even try to automate this, at least in a crude way). If everything looks good, I will make a copy of the relevant collections and give Edgar the go ahead to move them to prod (www.lmfdb.org) and close the issue.

This does mean that you should hold off making updates to the data while the issue is open and I am reviewing.

[AndrewVSutherland]

@sehlen; BTW, I'm happy to take your screen shots above as a description of the scope of the changes (but please tell me if there is more I should know). I'll start looking at the new spaces on beta now.

[AndrewVSutherland]

One problem so far (still checking). If you go to:
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/20/2/

there is a link to:
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/20/2/3/
which gives an error message.

Also, just FYI (I realize these are not in the range of the rectangle you indicated):

Hecke orbit data in webnewforms for space 56.10.27 is incomplete or inconsistent
[u'b', u'a'] versus [u'a']
Hecke orbit dimensions [2, 2, 2] do not sum to 58 for space 55.11.54
Hecke orbit data in webnewforms for space 56.11.13 is incomplete or inconsistent
[u'd', u'a', u'b', u'c'] versus [u'a', u'c', u'b']

[sehlen]

@AndrewVSutherland: Did I miss this discussion about the review process for
the data?
It definitely makes sense to have such a review process but, as you said,
currently we're fixing things (or indeed make them less bad as you said)
and not break them.
We should make such a discussion in public. But the very least is to inform
people and the last message I got was that 1/3 of the mongodb servers has
1-day old data.

Also, I really have to say that I start to get upset by all the bashing
about the modular forms pages, from all sorts of people and it makes me
lose the motivation of fixing things that are not my fault (or just working
on this whole thing at all if people are simply getting unfriendly). Most
likely, you didn't mean it that way and I'm talking to the wrong person
here but you when you write "not just modular forms", it just seems like
mentioning these pages became just the perfect example for anything that's
bad - when in fact I don't think anything has been broken by Fredrik or me
within the last few weeks so this just doesn't apply. On the contrary, I
definitely fixed bugs that were introduced by other people on the modular
forms pages.

Things are not perfect at all (wrong data for instance is certainly not
good ;-) but neither Fredrik nor me ever claimed them to be (nor did we set
the release date).

I can open an issue in the future for data changes if that's the way we do
it now. But again, we should have a discussion about it and document this
somewhere.

Your examples are known and will be fixed as part of a larger update coming
soon but they're all not in my rectangle which is only for Gamma0(N).

On Sun, May 15, 2016 at 5:50 PM Andrew Sutherland notifications@github.com
wrote:

@sehlen https://github.com/sehlen; BTW, I'm happy to take your screen
shots above as a description of the scope of the changes (but please tell
me if there is more I should know). I'll start looking at the new spaces on
beta now.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1376 (comment)

[sehlen]

@AndrewVSutherland, @edgarcosta,
Anyway, modularforms2.webnewforms, modularforms2.webmodformspace, modularforms2.dimension_table and modularforms2.webeigenvalues should be the ones that are affected by new data.

I will add documentation this week, in particular to the inventory, but we will also change a few things.

However, you don’t have to do this now - I was only asking how it works now because I changed the knowl when that square was completed because I expected it to appear on prod according to what was said earlier.

On May 15, 2016, at 17:25, Andrew Sutherland notifications@github.com wrote:

@sehlen https://github.com/sehlen, @edgarcosta https://github.com/edgarcosta We are still in the process of configuring a mechanism for database updates, but, with the exception of knowls, the plan is to keep it infrequent (we need to do it in a controlled way to make sure the update does not happen while someone is in the process of inserting data, which might result in an inconsistent state).

Having said that, it might make sense to put in place a procedure specifically for the database modularforms2. Can you tell us exactly which collections are actually used by the current code (there is a lot of "junk" currently in modularforms2 and we only want to copy the collections that are actually relevant, this are also the collections that should be documented on https://github.com/LMFDB/lmfdb-inventory/blob/master/db-modularforms2.md https://github.com/LMFDB/lmfdb-inventory/blob/master/db-modularforms2.md, per issue LMFDB/lmfdb-inventory#13 LMFDB/lmfdb-inventory#13.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[AndrewVSutherland]

@sehlen I apologize if my remarks came across the wrong way, I definitely did not mean to bash modular forms, in fact I am devoting a lot of my own time and effort to trying to help make them better! I have set aside time from my own research to work on this. I am currently trying to set up some computations in Magma that we can use to sanity check some of the data (I have 112 cores with magma running on them available to do this). Also, if there are any computationally intensive tasks you need done (e.g. Sage computations), please let me know (I can run Sage on Google's Compute Engine on up to 2048 32-core cpus), I'm happy to do anything I can to accelerate the process.

I don't know if you are aware of this, but www.lmfdb.org actually crashed on the release day (May 10) and all the cloud servers were down for almost an hour because of a change that was made to the L-function database in Warwick. It was at that point that we decided to disconnect the databases and realized that we really needed to make updates to the database on the cloud server in a controlled fashion, and should try to minimize the frequency of updates.

Thanks for the information on the collections, and for clarifying that the rectangle is only for Gamma0(N). I will just run a few more tests and then ask @edgarcosta to update the collections on www.lmfdb.org.

[sehlen]

On May 15, 2016, at 19:19, Andrew Sutherland notifications@github.com wrote:

@sehlen https://github.com/sehlen I apologize if my remarks came across the wrong way, I definitely did not mean to bash modular forms, in fact I am devoting a lot of my own time and effort to trying to help make them better! I have set aside time from my own research to work on this. I am currently trying to set up some computations in Magma that we can use to sanity check some of the data (I have 112 cores with magma running on them available to do this). Also, if there are any computationally intensive tasks you need done (e.g. Sage computations), please let me know (I can run Sage on Google's Compute Engine on up to 2048 32-core cpus), I'm happy to do anything I can to accelerate the process.

@AndrewVSutherland I know that you are helping (you’re sanity checks have already proven very useful) and that’s why I apologized in advance that I’m addressing the wrong person.

Please let us come back to your offer of running computations on GCE in a few days!
I’m just making sure (together with Fredrik), that if we touch data for non-trivial characters in a big way, we’re improving more things than just removing the wrong data. I’m cleaning up a lot of the code right now but I can only spend so much time on it as I have many other projects going on and did not plan on spending any time on it before or after the workshop last week. Anyway, once my cleanup and restructuring is done, running recomputations on GCE would probably give us an extension of what we already have in no time. (I used GCE myself already a bit and I really love it but I don’t have any grant for it right now.)

I don't know if you are aware of this, but www.lmfdb.org http://www.lmfdb.org/ actually crashed on the release day (May 10) and all the cloud servers were down for almost an hour because of a change that was made to the L-function database in Warwick. It was at that point that we decided to disconnect the databases and realized that we really needed to make updates to the database on the cloud server in a controlled fashion, and should try to minimize the frequency of updates.

I know, of course, as I have been at the AIM workshop and we saw it just a few minutes before John Cremona’s talk.
I completely agree that we should have a well-defined procedure for this and I just think that the downtime showed that in fact LMFDB was not ready for the release to be done in the way it was done but that’s a completely different issue. But now we have 1.0 and everything seems to have gotten so much more serious just a few days before the release. We asked many times for support during the last few years but no one wanted to work on classical modular forms and we had a special workshop for fixing issues regarding classical modular forms in December but all those who have to say a lot now have not been there.

Thanks for the information on the collections, and for clarifying that the rectangle is only for Gamma0(N). I will just run a few more tests and then ask @edgarcosta https://github.com/edgarcosta to update the collections on www.lmfdb.org http://www.lmfdb.org/.

Great - thank you!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[AndrewVSutherland]

@sehlen So my automated test is failing because there is a form with Hecke orbit label 2.12.1.a in webnewforms, but this space has dimension 0. Should that record be there?

[sehlen]

I deleted it now.
It shouldn’t be there but it also didn’t cause any harm as the record for the space 2.12.1 indicated that there were no hecke_orbits (note that a 0-dim space can have a record but it will have hecke_orbits={}).

Is this the only error you’re getting?

On May 15, 2016, at 20:10, Andrew Sutherland notifications@github.com wrote:

@sehlen https://github.com/sehlen So my automated test is failing because there is a form with Hecke orbit label 2.12.1.a in webnewforms, but this space has dimension 0. Should that record be there?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[AndrewVSutherland]

The only other problem was beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/23/38/1/a/, but I think that may just be an issue with the size of the coefficients, when I try it manually it works (but the display is slightly different, maybe you can double check it).

Otherwise it looks good!

[AndrewVSutherland]

Actually one other problem, take a look at http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/
which links to http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/1/ which just shows a blank page.

[sehlen]

On May 15, 2016, at 20:32, Andrew Sutherland notifications@github.com wrote:

The only other problem was beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/23/38/1/a/, but I think that may just be an issue with the size of the coefficients, when I try it manually it works (but the display is slightly different, maybe you can double check it).

What exactly fails for 23/38/1/a?

Otherwise it looks good!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[sehlen]

On May 15, 2016, at 20:35, Andrew Sutherland notifications@github.com wrote:

Actually one other problem, take a look at http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/ http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/
which links to http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/1/ http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/16/36/1/ which just shows a blank page.

Great find!

This one has not been computed to the end (strange to see an empty page, though, I will have to take a look tomorrow) and I overlooked it in the logs.
It is now done and displays fine.

The reason was something funny that I saw happening a few times before and didn’t know where it came from (now I do):
when we compute the twists, then the candidates have been recomputed (which they shouldn’t) and on top of that sometimes gave error messages, in this case because a record with too low precision was loaded/generated to test for the twist. Anyway, that’s in our modforms-db and not part of the lmfdb code.

Anyway, maybe @jwbober could also test his computations against this square before we make it completely available?
Do you have all of these?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[AndrewVSutherland]

I think 23/38/1/a is ok, it might have been failing in my test code just because the return page is so huge and the request might just be timing out. When I test it manually, other than taking a long time to render (which is understandable given the huge coefficients!) it seems fine.

[JohnCremona]

@sehlen and @fredstro have been absolute heroes recently and over the last 1-2 years writing and fixing a lot of things which were not their fault at all. All LMFDB developers really appreciate that, and are upset that the modular forms part of the website has been the one (essentially the only one) which has been criticised by anonymous bloggers. From now on (1) the more data we have in there the better (and we have lost of good ways to test its correctness, thanks to @AndrewVSutherland ) and (2) we need to respond to users' suggestions for additional search capabilities (as in #1362 ). Anyone can work on (2)!

[AndrewVSutherland]

@sehlen One other small thing: in webmodformsspace the space with label 2.11.3 is listed has having trivial character (which trips up my test). This also causes an error message on the (unreachable) page

http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/2/11/3/

I'm rerunning one final check on the pages (I will also check that the characters all match the labels). I'm not checking the coefficient data at all yet, that is a separate issue, I'm just making sure everything is internally consistent and that the pages load without crashing), at which point I think we can copy these over to www.lmfdb.org.

[AndrewVSutherland]

OK, everything in the box 2 <= w < = 40 and 1 <= N <= 24 with trivial character looks good.

@edgarcosta Can you move the collections modularforms2.webnewforms, modularforms2.webmodformspace, modularforms2.dimension_table and modularforms2.webeigenvalues from Warwick to the cloud? (temporarily rename the old ones and save them just in case).

[edgarcosta]

@sehlen
Could you please confirm that these are the collections that you want me to move the mongodb server serving lmfdb.org?

dimension_table
dimension_table.chunks
dimension_table.files

webmodformspace
webmodformspace.chunks
webmodformspace.files

webnewforms
webnewforms.chunks
webnewforms.files

webeigenvalues.chunks
webeigenvalues.files

[sehlen]

Confirmed.

On May 16, 2016, at 09:39, edgarcosta notifications@github.com wrote:

@sehlen https://github.com/sehlen
Could you please confirm that these are the collections that you want me to move the mongodb server serving lmfdb.org?

dimension_table
dimension_table.chunks
dimension_table.files

webmodformspace
webmodformspace.chunks
webmodformspace.files

webnewforms
webnewforms.chunks
webnewforms.files

webeigenvalues.chunks
webeigenvalues.files
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #1376 (comment)

[JohnCremona]

@sehlen and/or @fredstro It would help others take over from you (or at least contribute) to classical mfs if you could edit https://github.com/LMFDB/lmfdb-inventory/blob/master/db-modularforms2.md appropriately, including removing sections for redundant collections (and removing the redundant collections). I think the "new" mfs setup is stable enough for that? It is tedious but incredibly helpful, and I know hom much you two want to move on from this!

[sehlen]

@JohnCremona I know that this has to be updated. And that's part of what I will do this week (see also my plan at #1248).

[AndrewVSutherland]

I have implemented and tested an update script that will copy the 11 collections listed in #1376 (comment) from warick (beta) to the cloud (prod). This process takes an hour or two, but the code is implemented so that the changes will appear "instantaneously" at the end of the process via collection renaming, and it should be safe to run on the live system (we will still want to monitor it in real time; I do not suggest that we automate these incremental updates, there are too many things that could go wrong).

I suggest that we wait until we have addressed #1385 (which will make sure the new pages display in a reasonable fashion) and #1386 (which will change the database again) before applying it.

Until the update happens I will leave this issue open, but I am changing its description to make it specific to this update (if/when another update is needed, we can open a new issue to make sure the update happens).

[AndrewVSutherland]

@sehlen

It appears there are still some problems with the data, see:

http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/10/38/1/
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/14/36/1/
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/17/22/1/

The first two pages are blank and the second gives an error message. For the rest I can at least confirm that the dimension decomposition in agrees with Magma's output. I'm also going to check the coefficient fields.

[JohnCremona]

For me now they all look good. (08:48 BST Wednesday 18 May)

[sehlen]

Yep, I fixed them.
I also ran a couple of automated checks on the whole range and they should all be fine now.

[sehlen]

If this takes a while to update, it might be worth waiting a bit because there will be more updates including a larger code update from me.

[JohnCremona]

I am not sure if this Issue is the right place to report this, but having run Drew's new emf_test_pages several times today I can report that there is a failure:

...
Checking ModularForm/GL2/Q/holomorphic/2/10/1/a/
Label 2.11.3 does not match level=2, weight=11, character=1
Checking ModularForm/GL2/Q/holomorphic/2/11/3/ (0 Hecke orbits, total dimension 0)
Failed on ModularForm/GL2/Q/holomorphic/2/11/3/
...
Errors occurred for the following labels: [u'2.11.3', u'2.11.3']
FAIL

[sehlen]

Sorry, is the first one for 2/10/1/a an error? I cannot see what;'s wrong with that.
And 2/11/3 is not a valid input anyway. I guess there's a record that I should delete but it's certainly not reachable from the website.

[AndrewVSutherland]

@JohnCremona Yes this is the right place to report this and the case you found was already noted in #1376 (comment). It's harmless in the sense that there is no way for the user to see the problem (if they go to http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/2/10/1/a/ they will see the right thing, and 2.11.3 is not even a valid label), but these two records should be removed from the database (apparently @sehlen has not yet done this).

@sehlen, I agree it makes sense to wait for your updates, and I also want to verify all the coefficient fields (which none of the tests we have run so far do). I have computed these for all the spaces with level N <= 24 and weight 2 <= w <= 40 and trivial character in Magma, and we also have @jwbober's data, which includes defining polynomials for all these coefficient fields. I would be very happy to find that all three data sets (Magma, @jwbober, and modularforms2) agree on the coefficient field.

I also have computed the traces of the first 100 coefficients for all of these and have asked @jwbober to do the same, and would like to compare these as well (I expect that if the coefficient fields agree, then the coefficients are likely to be correct, but it does not hurt to check).

[AndrewVSutherland]

The new data (and the improved software form PR #1409) is now available on www.lmfdb.org:

It also includes the range 1 <= N <= 100 and 2 <= w <= 12: