Mf fixes #1465

sehlen · 2016-05-26T15:43:57Z

This is an alternative to #1453
which does not show the random modular form for now but includes the bugfixes (+1 more bugfix that does not affect the website at all).

…plate which I have been editing accidentally.

…rst save_to_db() call.

AndrewVSutherland · 2016-05-28T23:48:21Z

@sehlen It could be because I do not have a few of the additional sage packages the LMFDB uses (notably: git+git://github.com/jwbober/conrey-dirichlet-characters.git@master#egg=dirichlet_conrey) installed on my dev system at home (I haven't been able to get Sage 7.1 to rebuild properly with all the packages listed in requirements.txt). I'll be careful to double check on Atkin before reporting further errors.

sehlen · 2016-05-28T23:51:28Z

@AndrewVSutherland OK it almost has to be the conrey dirichlet characters although I would expect that you should see more errors.
When a form with non-trivial character is loaded from the db, a conrey dirichlet character is initialized and so errors are caught (and maybe raised again which we might not want to do, not sure) and give the general log message that you saw, I think.

On May 28, 2016, at 19:48, Andrew Sutherland notifications@github.com wrote:

@sehlen https://github.com/sehlen It could be because I do not have a few of the additional sage packages the LMFDB uses (notably: git+git://github.com/jwbober/conrey-dirichlet-characters.git@master#egg=dirichlet_conrey) installed on my dev system at home (I haven't been able to get Sage 7.1 to rebuild properly with all the packages listed in requirements.txt). I'll be careful to double check on Atkin before reporting further errors.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #1465 (comment), or mute the thread https://github.com/notifications/unsubscribe/ACW20bXADRo9GSJUiePDRJ2ROjcYZSfZks5qGNRGgaJpZM4InrYz.

AndrewVSutherland · 2016-05-29T02:22:33Z

Page http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/11/10/4/a/ (running on Atkin now) produces a server error caused by "AttributeError: 'unicode' object has no attribute 'absolute_degree'"

The pages http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/13/10/4/a/ and http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/23/6/2/a/ produce server errors caused by "TypeError: 'NoneType' object is not iterable"

The page http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/31/30/1/a/ produces a server error caused by "TypeError: unsupported operand type(s) for -: 'Undefined' and 'int'"

There are data problems (hecke orbit space dimensions not summing to the correct value) for the spaces 11.4.3, 11.6.3, 16.2.13, 16.2.13, 18.2.7, 20.2.3, 21.2.4, 21.2.5, 22.2.3, 32.4.21, 36.5.5. This isn't an issue for this PR (they already cause errors on www.lmfdb.org), just letting you know.

sehlen · 2016-05-29T02:40:07Z

For /ModularForm/GL2/Q/holomorphic/11/10/4/a: This space shouldn't be there (I'm not fixing this now as a more systematic update will run soon, I hope). Go to /ModularForm/GL2/Q/holomorphic/11/10/4/ and observe that it links to /ModularForm/GL2/Q/holomorphic/11/10/3.

The second one http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/13/10/4/a/ seems like a bug, I will check tomorrow.

The last one is a data issue and has maybe been observed before, not sure.
Observe that /ModularForm/GL2/Q/holomorphic/31/30/1/ tells you that "Unfortunately, we do not have this space in the database, yet.". But I will fix this in the code nevertheless so that the user does not see a server error.

AndrewVSutherland · 2016-05-29T02:47:14Z

@sehlen OK, thanks, just let me know when you make a new commit and I will retest. I have updated emf_test_pages.py to check all the spaces it can find in the database within our stated extent (any character) -- it hasn't finished yet but I'll let you know first thing tomorrow if it finds any more (I'll also merge the new emf_test_pages.py into the lmfdb master branch).

(http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/13/10/4/a/ and http://127.0.0.1:37777/ModularForm/GL2/Q/holomorphic/23/6/2/a/). _nv_coeff_index was not set for some reason. Moreover, make sure it's always set when savong to the db.

sehlen · 2016-05-29T12:58:48Z

All right - all problems should be solved with a combination of a code update and a small data update in the two cases where the "TypeError: 'NoneType' object is not iterable" error occurred.

Note that these forms do have "new" records with the properties we show when hiding the q-expansion stored in the db and they will be different on prod. As far as I can see, everything should work on prod but I have to admit, I replaced some data without making a backup to check with.

But since the modular_forms2 db certainly still contains many records that are identical to what we have on prod, this should be well tested already.

When you feel this is tested enough, it should go to beta, and after a little testing to prod, I think.
The dimension table can go to prod right away, I think.
Maybe you can make a backup so that we could switch back very fast (I made one on the warwick mongodb: dimension_table_old).

AndrewVSutherland · 2016-05-29T13:05:24Z

@sehlen Thanks for the update, I'll test it now. In terms of the data, will any of the changes you made cause problems for the software currently running on prod? If not I might be inclined to copy the 11 collections we copied before over to prod -- and yes I always save one backup that we can recover from very quickly (just a rename) if anything goes wrong.

Another question, which collections on prod are actually used by the software, is it just the 11 collections:

"dimension_table" "dimension_table.chunks" "dimension_table.files" "webmodformspace" "webmodformspace.chunks" "webmodformspace.files" "webnewforms" "webnewforms.chunks" "webnewforms.files" "webeigenvalues.chunks" "webeigenvalues.files"

or are there others? It would be nice to remove anything we don't need on prod (it would free up space to keep backup collections around).

AndrewVSutherland · 2016-05-29T13:55:31Z

@sehlen It looks good, I think we are ready to merge.

sehlen · 2016-05-29T14:08:21Z

@AndrewVSutherland Regarding the data:

The collections you list should indeed be the full list of collections that are currently used.
We might offer the modular symbols stored in Modular_symbols and Newform_factors for download at some point. But otherwise, I don't think anything else is used. @fredstro could you please confirm?
If we delete these from prod, ithey would still be backed-up by some backup running in Warwick as well as the replication to beta, right?
The data right now should at least not be worse than what we have on prod but I've been playing around with it a little bit - so there are some new records and some old ones (and I can't guarantee right now that there isn't a single record that I might have screwed up right now). When I make the larger update, I guess it's best to make a copy of the collections. But it should indeed be safe to copy it over to prod, although the difference is small.

sehlen · 2016-05-29T14:31:01Z

@AndrewVSutherland Probably it's indeed a good idea to copy the data.
Take the loading time of the following page (where the record has the new fields on beta running the code that uses it):

http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/
(right now I get a timeout)
vs
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/

However, this has not been updated systematically, yet.

fredstro · 2016-05-29T14:59:20Z

regarding the data. the only thing I'm not sure if (and I don't have access
to my computer now) is the dimension table. I'm not sure if this is the
correct dimension table. it seems that the one mentioned is using gridfs
and I think the one we use now only uses mongo records. Can you check?
@sehlen

On Sun, 29 May 2016 16:31 Stephan Ehlen, notifications@github.com wrote:

@AndrewVSutherland https://github.com/AndrewVSutherland Probably it's
indeed a good idea to copy the data.
Take the loading time of the following page (where the record has the new
fields on beta running the code that uses it):

http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/
(right now I get a timeout)
vs
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/

However, this has not been updated systematically, yet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1465 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ABhig-6ON11nKRDWEbRzWCG9f4KaaKsQks5qGaMmgaJpZM4InrYz
.

sehlen · 2016-05-29T15:11:19Z

Thanks for noticing, @fredstro. It is the correct dimension table (pretty
sure, I recreated it a few days ago and it is in use) but the .files and
.chunks don't exist in fact ;-)

On Sun, May 29, 2016, 10:59 Fredrik Strömberg notifications@github.com
wrote:

regarding the data. the only thing I'm not sure if (and I don't have access
to my computer now) is the dimension table. I'm not sure if this is the
correct dimension table. it seems that the one mentioned is using gridfs
and I think the one we use now only uses mongo records. Can you check?
@sehlen

On Sun, 29 May 2016 16:31 Stephan Ehlen, notifications@github.com wrote:

@AndrewVSutherland https://github.com/AndrewVSutherland Probably it's
indeed a good idea to copy the data.
Take the loading time of the following page (where the record has the new
fields on beta running the code that uses it):

http://www.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/
(right now I get a timeout)
vs
http://beta.lmfdb.org/ModularForm/GL2/Q/holomorphic/56/10/27/

However, this has not been updated systematically, yet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1465 (comment), or
mute
the thread
<
https://github.com/notifications/unsubscribe/ABhig-6ON11nKRDWEbRzWCG9f4KaaKsQks5qGaMmgaJpZM4InrYz

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1465 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ACW20cJgiFY76SvcMNdT-1sv-3b3FqAXks5qGanJgaJpZM4InrYz
.

AndrewVSutherland · 2016-05-29T15:53:08Z

@sehlen Great, I can trim the list down to 9 collections. I will update these 9 on prod later today.

AndrewVSutherland · 2016-05-29T16:04:01Z

On 2016-05-29 10:08, Stephan Ehlen wrote:

@AndrewVSutherland Regarding the data:

The collections you list should indeed be the full list of
collections that are currently used.
We might offer the modular symbols stored in Modular_symbols and
Newform_factors for download at some point. But otherwise, I don't
think anything else is used. @fredstro could you please confirm?
If we delete these from prod, ithey would still be backed-up by some
backup running in Warwick as well as the replication to beta, right?

Correct, beta (by which I mean the mongo db running on
lmfdb.warwick.ac.uk and used by beta.lmfdb.org) is backed up regularly,
and it is not synched to prod. Deleting collections from prod has no
impact on beta. The beta mongo db is also replicated in the cloud (but
note that deletions are replicated automatically as well, so this does
not protect you against accidentally deleting something you need).

But I might suggest that if you know you can get rid of any of the
collections on beta it would be worth doing so (they will still exist on
backup).

For example, are the collections WebNewForms.files and
WebNewForms.chunks relevant (there is no corresponding WebNewForms).

The data right now should at least not be worse than what we have
on prod because but I've been playing around with it a little bit - so
there are some new records and some old ones. When I make the larger
update, I guess it's best to make a copy of the collections. But it
should indeed be safe to copy it over to prod, although the difference
is small.

I think I might be happier having the data we tested on prod, so that it
matches what we tested the new software against (we can always revert to
the old data if need be, but reverting to the old software would
actually be a pain with our current setup).

When do you expect to have the larger update done?

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#1465 (comment)

AndrewVSutherland · 2016-05-29T17:55:40Z

@sehlen @fredstro @edgarcosta

I removed all but the following 12 collections from the production mongo db and everything still seems to work:

dimension_table
webmodformspace webmodformspace.chunks webmodformspace.files
webnewforms webnewforms.chunks webnewforms.files
webeigenvalues.chunks webeigenvalues.files
webchar webchar.chunks webchar.files

I'm hesitant to remove the last three because they are referenced in the code (in web_character.py).

sehlen · 2016-05-29T20:51:02Z

I haven't looked at this in a while but I believe that the webchar collections are in fact used but not necessary (i.e. the website also runs without it)
In theory they could be used more if we would insert records for all characters for which we have any modular forms spaces in the db. The code will work just fine without these collections but maybe a bit slower (in cases where a character now has a record in the db) - I haven't really tested it.

AndrewVSutherland · 2016-05-29T20:55:22Z

Thanks. I'll leave them in for the moment (the don't take up much space
in any case).

On 2016-05-29 16:51, Stephan Ehlen wrote:

I haven't looked at this in a while but I believe that the webchar
collections are in fact used but not necessary (i.e. the website also
runs without it)
In theory they could be used more if we would insert records for all
characters for which we have any modular forms spaces in the dab. The
code will work just fine without these collections but maybe a bit
slower, in case a character has a record in the db - I haven't really
tested it.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#1465 (comment)

fredstro · 2016-05-29T21:55:27Z

@sehlen at some point we should also see if we can use the webchars class
from the dirichlet character pages instead of having our own (or make them
use ours...)

On Sun, 29 May 2016 22:55 Andrew Sutherland, notifications@github.com
wrote:

Thanks. I'll leave them in for the moment (the don't take up much space
in any case).

On 2016-05-29 16:51, Stephan Ehlen wrote:

I haven't looked at this in a while but I believe that the webchar
collections are in fact used but not necessary (i.e. the website also
runs without it)
In theory they could be used more if we would insert records for all
characters for which we have any modular forms spaces in the dab. The
code will work just fine without these collections but maybe a bit
slower, in case a character has a record in the db - I haven't really
tested it.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#1465 (comment)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1465 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ABhigzaEPHusIRuQyE-fovHKGoPrxMoxks5qGf07gaJpZM4InrYz
.

sehlen added 30 commits May 20, 2016 22:55

add random modular form

4e28421

add forgotten import

5629a24

use function to connect to db

1468bd4

use function to connect to db

8127e67

use function to connect to db

238628b

missing import

8404c6a

correct call of url_for

8661a14

correct call of url_for

f24ae16

add random modular form to browse page

3fb8820

add random modular form to browse page

5a8dd30

Add random modular form to browse page and also remove unused old tem…

cc1518c

…plate which I have been editing accidentally.

correct wrong suggestion on brwose page

f2a1855

Merge remote-tracking branch 'lmfdb/master' into mf_random

94c05b1

Fix typo noticed by @AndrewVSutherland.

6eb9bc7

Fix a bug that prevented saving the lmfdb_label when set after the fi…

5d5b376

…rst save_to_db() call.

Merge branch 'bugfix_lmfdb_label'

350c08d

fix search when not using separate collection

2067a25

continue fix search when not using separate collection

94b4fbe

typo

1b468f3

do not update dynamic properties in find

f35902a

get only meta records for deleting

399a6ef

move setting prec to the right place

eea9406

fix typo

65e6392

fix very dangerous bug in delete_from_db()

2b6f866

Merge remote-tracking branch 'lmfdb/master'

37ad8f2

Merge branch 'master' into mf_random

1e7d1a6

This fixes LMFDB#1442 (comment).

8c633ae

added tests for the fixed cases

aee878b

added tests for the fixed cases

b1f0e7e

fix tests

ba4077e

sehlen added 8 commits May 29, 2016 08:17

return 0 for max_available_prec() if no file found

3a93819

Do not show full newform page if form is not in DB

4ca712f

Do not set WNF to None if it has not updated

be50c1d

do not pretend to be empty if not updated

bbf00d1

fix typo

1a0459c

only load the scripts if no error occured

649e9e8

formatting

078179a

fix a small typo

a3c016f

AndrewVSutherland merged commit a6758e2 into LMFDB:master May 29, 2016

sehlen deleted the mf_fixes branch May 29, 2016 14:19

AndrewVSutherland mentioned this pull request May 30, 2016

Browse box problem #510

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mf fixes #1465

Mf fixes #1465

sehlen commented May 26, 2016

AndrewVSutherland commented May 28, 2016

sehlen commented May 28, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016 •

edited

Loading

sehlen commented May 29, 2016

fredstro commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016 •

edited

Loading

AndrewVSutherland commented May 29, 2016

fredstro commented May 29, 2016

Mf fixes #1465

Mf fixes #1465

Conversation

sehlen commented May 26, 2016

AndrewVSutherland commented May 28, 2016

sehlen commented May 28, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016 • edited Loading

sehlen commented May 29, 2016

fredstro commented May 29, 2016

sehlen commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

AndrewVSutherland commented May 29, 2016

sehlen commented May 29, 2016 • edited Loading

AndrewVSutherland commented May 29, 2016

fredstro commented May 29, 2016

sehlen commented May 29, 2016 •

edited

Loading

sehlen commented May 29, 2016 •

edited

Loading