memory usage

[ikher]

When I run circlator (circlator all) everything seems fine at the beginning, but after a couple of minutes running it crashes with this error message:

Traceback (most recent call last):
File "/home/ubuntu/.local/bin/circlator", line 58, in <module>
    exec('circlator.tasks.' + task + '.run()')
File "<string>", line 1, in <module>
File "/home/ubuntu/.local/lib/python3.5/site-packages/circlator/tasks/all.py", line 169, in run
    a.run()
File "/home/ubuntu/.local/lib/python3.5/site-packages/circlator/assemble.py", line 176, in run
    self.run_spades(stop_at_first_success=self.spades_use_first_success)
File "/home/ubuntu/.local/lib/python3.5/site-packages/circlator/assemble.py", line 145, in run_spades
    raise Error('Error running SPAdes. Output directories are:\n  ' + '\n  '.join(kmer_to_dir.values()) + '\nThe reason why should be in the spades.log file in each directory.')
circlator.assemble.Error: Error running SPAdes. Output directories are:
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.97.85z_h7yn
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.117.t9n_6clr
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.87.79_qms10
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.107.xsu1lo0e
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh
/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.127.vaus39y9
The reason why should be in the spades.log file in each directory.

Going through the SPAdes log files, as advised in the error message, there seems to be a memory issue...

Command line: /home/ubuntu/Desktop/scripts/circlator-1.5.1/build/SPAdes-3.7.1-Linux/bin/spades.py -s /home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/02.bam2reads.fasta -o /home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh -t 1 -k 77 --careful --only-assembler 

System information:
  SPAdes version: 3.7.1
  Python version: 2.7.12
  OS: Linux-4.4.0-83-generic-x86_64-with-Ubuntu-16.04-xenial

Output dir: /home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh
Mode: ONLY assembling (without read error correction)
Debug mode is turned OFF

Dataset parameters:
  Multi-cell mode (you should set '--sc' flag if input data was obtained with MDA (single-cell) technology or --meta flag if processing metagenomic dataset)
  Reads:
    Library number: 1, library type: single
      left reads: not specified
      right reads: not specified
      interlaced reads: not specified
      single reads: ['/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/02.bam2reads.fasta']
Assembly parameters:
  k: [77]
  Mismatch careful mode is turned ON
  Repeat resolution is enabled
  MismatchCorrector will be used
  Coverage cutoff is turned OFF
Other parameters:
  Dir for temp files: /home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh/tmp
  Threads: 1
  Memory limit (in Gb): 1


======= SPAdes pipeline started. Log can be found here: /home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh/spades.log


===== Assembling started.


== Running assembler: K77

  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.hpp          :  43)   Memory limit set to 1 Gb
  0:00:00.004     4M / 4M    INFO    General                 (main.cpp                  : 148)   Starting SPAdes, built from refs/heads/spades_3.7.1, git revision 0a208242974e028693083a617d54e8dbc9e2577b
  0:00:00.005     4M / 4M    INFO    General                 (main.cpp                  : 149)   Assembling dataset (/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh/dataset.info) with K=77
  0:00:00.005     4M / 4M    INFO    General                 (launch.hpp                :  30)   SPAdes started
  0:00:00.005     4M / 4M    INFO    General                 (launch.hpp                :  36)   Starting from stage: construction
  0:00:00.005     4M / 4M    INFO    General                 (launch.hpp                :  39)   Two-step RR enabled: 0
  0:00:00.005     4M / 4M    INFO    General                 (launch.hpp                :  58)   Will need read mapping, kmer mapper will be attached
  0:00:00.006     4M / 4M    INFO   StageManager             (stage.cpp                 : 126)   STAGE == Construction
  0:00:00.006     4M / 4M    INFO    General                 (read_converter.hpp        :  80)   Converting reads to binary format (takes a while)
  0:00:00.008     4M / 4M    INFO    General                 (read_converter.hpp        :  87)   Paired reads for library #0
  0:00:00.959    68M / 132M  INFO    General                 (binary_converter.hpp      : 159)   0 reads written
  0:00:00.960     4M / 132M  INFO    General                 (read_converter.hpp        :  98)   Single reads for library #0
  0:00:02.610   144M / 260M  INFO    General                 (binary_converter.hpp      : 159)   1064 reads written
  0:00:02.613     4M / 260M  INFO    General                 (graph_construction.hpp    : 130)   Constructing DeBruijn graph for k=77
  0:00:02.613     4M / 260M  INFO    General                 (kmer_splitters.hpp        : 173)   Splitting kmer instances into 1 buckets. This might take a while.
  0:00:02.613     4M / 260M  INFO    General                 (file_limit.hpp            :  30)   Open file limit set to 1024
  0:00:02.613     4M / 260M  INFO    General                 (kmer_splitters.hpp        : 191)   Memory available for splitting buffers: 0.332031 Gb
  0:00:02.613     4M / 260M  INFO    General                 (kmer_splitters.hpp        : 200)   Using cell size of 14854826
  0:00:03.773   380M / 756M  INFO    General                 (kmer_splitters.hpp        : 235)   Adding contigs from previous K
  0:00:03.774   380M / 756M  INFO    General                 (kmer_splitters.hpp        : 246)   Used 2128 reads. Maximum read length 15105
  0:00:03.774   380M / 756M  INFO    General                 (kmer_splitters.hpp        : 247)   Average read length 4950.47
  0:00:03.775     4M / 756M  INFO    General                 (kmer_index.hpp            : 348)   Starting k-mer counting.
=== Stack Trace ===
[0x404fda]
[0x44e00c]
[0x472983]
[0x474945]
[0x44cc13]
[0x4781d0]
[0x498caa]
[0x4a3669]
[0x4a3caa]
[0x4a3d83]
[0x4a4057]
[0x4806a3]
[0x44a473]
[0x402502]
[0x417646]
[0x803650]
[0x4001b9]
Verification of expression '(intptr_t)MappedRegion != -1L' failed in function 'MMappedReader::MMappedReader(const string&, bool, size_t, off_t, size_t)'. In file '/home/yasha/gitrep3/algorithmic-biology/assembler/src/include/io/mmapped_reader.hpp' on line 97. Message 'mmap(2) failed. Reason: Cannot allocate memory. Error code: 12'.
Verification of expression '(intptr_t)MappedRegion != -1L' failed in function 'MMappedReader::MMappedReader(const string&, bool, size_t, off_t, size_t)'. In file '/home/yasha/gitrep3/algorithmic-biology/assembler/src/include/io/mmapped_reader.hpp' on line 97. Message 'mmap(2) failed. Reason: Cannot allocate memory. Error code: 12'.
spades: /home/yasha/gitrep3/algorithmic-biology/assembler/src/include/io/mmapped_reader.hpp:97: MMappedReader::MMappedReader(const string&, bool, size_t, off_t, size_t): Assertion `(intptr_t)MappedRegion != -1L' failed.


== Error ==  system call for: "['/home/ubuntu/Desktop/scripts/circlator-1.5.1/build/SPAdes-3.7.1-Linux/bin/spades', '/home/ubuntu/Desktop/scripts/circlator-1.5.1/circlator/circlator_out/03.assemble.tmp.spades.77.u2x2lpqh/K77/configs/config.info']" finished abnormally, err code: -6

In case you have troubles running SPAdes, you can write to spades.support@bioinf.spbau.ru
Please provide us with params.txt and spades.log files from the output directory.

I work in a private company and my computing resources are quite limiting: I am not the administrator of my desktop PC, and I am forced to work in a virtual machine to work in a UNIX environment. The resources allocated to the virtual machine are 3Gb RAM and 2.2 Gb hard disk space free (the assembly and the corrected reads apart). In the circlator manuscript it is mentioned that the peak RAM usage was 490 Mb for a Salmonella genome but (I think) there is no mention to the hard disk storage memory required...
The bacterial genome I work with was sequenced with a PacBio RSII and assembled de novo with the RS_HGAP Assembly.2 pipeline. I got 10 contigs but I am not sure about the size of the genome (no reference is available for this species, and closest speceis vary between 4 and 6.5 Mb).
In summary, is 3 Gb RAM and 2.2 Gb hard disk space enough to run circlator to try and circularize 9-10 contigs from a baterial genome of 4-6 Mb?

[andrewjpage]

It does appear to be a lack of resources, however for less than $1 you can rent a machine with sufficient resources to perform your analysis from Amazon.

[splaisan]

I have the same issue although I have 84 threads and 512G RAM.
Circlator uses the first 250G and dies. How can I increase the spades memory limit? (SPAdes --memory is not recognised)
Also, my tmp is rather small, can I specify --tmp-dir for spades too (SPAdes --tmp-dir is not recognised)?
Thanks for your help