Canu Sorting ovlStore error

[HRitchie89]

Hi there,

I am running Canu on my MinION data and I'm having problems during the creation of the ovlStore. The bucketizing completes then when it moves onto Store I get the following error:

-- SORTING --

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Failed with 'Aborted'; backtrace (libbacktrace):
AS_UTL/AS_UTL_stackTrace.C::102 in _Z17AS_UTL_catchCrashiP7siginfoPv()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
../../gcc-4.8.2/libstdc++-v3/libsupc++/vterminate.cc::95 in _ZN9__gnu_cxx27__verbose_terminate_handlerEv()
../../gcc-4.8.2/libstdc++-v3/libsupc++/eh_terminate.cc::38 in _ZN10__cxxabiv111__terminateEPFvvE()
../../gcc-4.8.2/libstdc++-v3/libsupc++/eh_terminate.cc::48 in _ZSt9terminatev()
../../gcc-4.8.2/libstdc++-v3/libsupc++/eh_throw.cc::84 in __cxa_throw()
../../gcc-4.8.2/libstdc++-v3/libsupc++/new_op.cc::56 in _Znwm()
../../gcc-4.8.2/libstdc++-v3/libsupc++/new_opv.cc::32 in _Znam()
stores/ovOverlap.H::192 in _ZN9ovOverlap16allocateOverlapsEP7gkStorem()
stores/ovStoreBuild.C::584 in main()
(null)::0 in (null)()
(null)::0 in (null)()


I have increased the available memory for Canu so I am confident it is not a memory issue. 

--                     (tag)Threads
--            (tag)Memory         |
--        (tag)         |         |  algorithm
--        -------  ------  --------  -----------------------------
-- Grid:  meryl     64 GB    8 CPUs  (k-mer counting)
-- Grid:  cormhap   13 GB    8 CPUs  (overlap detection with mhap)
-- Grid:  obtovl     8 GB    8 CPUs  (overlap detection)
-- Grid:  utgovl     8 GB    8 CPUs  (overlap detection)
-- Grid:  cor       20 GB    4 CPUs  (read correction)
-- Grid:  ovb       64 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs      128 GB    1 CPU   (overlap store sorting)
-- Grid:  red        6 GB    4 CPUs  (read error detection)
-- Grid:  oea        2 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat       64 GB    8 CPUs  (contig construction)
-- Grid:  cns       48 GB    8 CPUs  (consensus)
-- Grid:  gfa        8 GB    8 CPUs  (GFA alignment and processing)

-- Finished on Fri Jan 12 18:52:35 2018 (36795 seconds) with 11934.062 GB free disk space
----------------------------------------
ERROR:
ERROR:  Failed with exit code 134.  (rc=34304)
ERROR:

ABORT:
ABORT: Canu 1.6
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT:   failed to create the overlap store.
ABORT:
ABORT: Disk space available:  11934.062 GB

I'm fairly stumped so any suggestions would be greatly appreciated.

[brianwalenz]

The error looks like it can't allocate the requested memory. It might be hitting shell limits on the size of a process allowed.

This step doesn't benefit from more memory -- algorithmically, it's slower with more memory. However, allowing too little memory for this stage results in other limits being hit.

Try reducing ovsMemory to 8g or 16g.

[HRitchie89]

Thanks for the reply.

Turns out it was a memory issue. I believe it's due to the way our cluster lets us allocate memory. I ended up having to request 1TB of memory before sorting would even begin.

My MinION data is RNA rather than DNA so the number of overlaps I have is understandably large. I think it might end up being a trade off between memory and how small I allow my overlap to be. (Currently it's 100bp)

[brianwalenz]

Can you elaborate? How big is the resulting ovlStore directory?

There are two memory sizes involved. One is to tell the algorithm how much data to process in one chunk, and the other is to tell the grid how much memory the process will use. Problems occur when the process gets bigger than the grid expects. To prevent that, and to keep it simple for users, a setting of ovsMemory=128 tells the grid to expect a job needing 128GB memory, but tells the job to process (for example) only 96GB of data in one chunk. Since this is a pretty simple algorithm, I'd like to understand why it failed for you.

[skoren]

This is assembly of RNA data? Canu isn't going to do anything reasonable past correction and I'm not sure anyone has tried correction on RNA data, only a few rna seq runs. See issue #641 for some suggestions to run given the very short overlap length. It might decrease your store size as well.

[HRitchie89]

brianwalenz, when I request memory for a program that program cannot use any more than I have requested. Like it was said in the issue #641 (highlighted by skoren) where XGB was requested but Canu actually used 10XGB I cannot do that. If I do not allocate an appropriate amount of memory to start with Canu won't run because it cannot request additional memory. In the end I had to request 1TB of mem for ovs and what Canu is actually using is "vmem=762.962G, maxvmem=762.962G". If I request any less than this when submitting the job it will not run. I do not have a completed ovlStore directory but the ovlStore.BUILDING is ~1.5TB.

skoren, it is for error correction and "assembly". As I am using MinION data generated from a cDNA sample I have a huge number of duplicate reads so essentially I just want to error correct these. As I don't have additional Illumina data I cannot assemble these reads any further so once they have been error corrected they are essentially as "assembled" as they are going to get. Thank you so much for pointing me in the right direction. My average transcript length is 800bp but I am retaining all sequences 200bp and above so I didn't want to set my overlap any more than 100bp for fear of losing my shorter transcripts.

[brianwalenz]

I'm not aware of anything in this algorithm that would need that much memory, unless you have exceptionally deep overlaps - as in billions of overlaps for a single read. How much of this data can you share? The *.counts files in 1-overlapper would be enough for me to figure out number of overlaps per read, but it'd be somewhat painful to fix the issue using just that.

[brianwalenz]

This has been fixed; see #758 for details.