Decide if moving to MonetDB before Beta.

[benjeffery]

We're close to beta where we will be polishing and having public releases of Panoptes 2.0, so we'd like to have a hiatus on large feature work. We probably have time to switch to MonetDB as the last big change. Do we want to do this? I have rebased the MonetDB branch (https://github.com/cggh/panoptes/tree/monet_rebased) and will load up AG on it for some testing on an SSD based machine.

My personal opinion is that if no show-stoppers are found then we should go ahead. It would potentially let us avoid pre-calculating filter banks, opening the door to parameterised filterbanks, such as filtered variant density.

Comments welcome!

[leehart]

I think we should at least try. Roll on the AG data with SSD and see how it flies! :-)

[leehart]

https://www.monetdb.org/blog/embedded-pythonnumpy-monetdb

[benjeffery]

That functionality could be good for custom summarisers. I've also tried the python module that is vice versa - monet in the python process. It was super speedy, but I have no idea how it would work with a multiprocess web server.

[benjeffery]

My current list of concerns that need checking:

• DB server stability
• Small row/many columns performance
• Join performance for future related data features
• Import error handling
• Query cancel/timeout

[benjeffery]

Quick update:
I've set up a monetdb demo instance at http://104.199.5.137/

I am working this demo instance towards dynamic filterbanking - I've created a new query method on the server that supports SQL aggregations and returns ArrayBuffers that this will use.
We also need fundamental changes to the type system in panoptes as we previously stored all numeric data as floating point - for filterbanking queries we need to do integer arithmetic inside the database.

[benjeffery]

The instance at http://104.199.5.137/ has been updated - the genome browser is now completely DB driven. There are a few rough edges still to work on (e.g. query cancelling) but in general I am very happy with this approach so far.

[benjeffery]

PR #488 is ready! I'll take silence as a rousing endorsement of switching to Monet ;) Happy to hear any concerns though.

[alimanfoo]

JFDI :-)

On Thursday, 8 September 2016, Ben Jeffery notifications@github.com wrote:

PR #488 #488 is ready! I'll take
silence as a rousing endorsement of switching to Monet ;) Happy to hear any
concerns though.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#482 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QmavYPYNMVZUTICBwQCfj_zZzG6pks5qoCsBgaJpZM4Jl_il
.

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721