Partition sweep

[camillescott]

Open pull request for sparse graph coloring.

Has some old commits from bleeding-edge, but should be fine.

[mr-c]

@cswelcher Please merge in the latest from the master branch so that @ged-jenkins can test it out :-)

[mr-c]

(hmm.. Jenkins should have built this by now. Lets test the magic phrase)

please test this

[mr-c]

Grr.. the actual phrase is:

test this please

[mr-c]

test this please

[ged-jenkins]

Test PASSed.
Refer to this link for build results: http://ci.ged.msu.edu/job/khmer-multi-pullrequest/7/

[camillescott]

Instructions for Sweep

The script to use is scripts/sweep-reads-by-partition-buffer.py, which consumes a partitioned file and runs through a list of read files, outputting them to files by color. This version has a read buffering system built in which greatly speeds up the output process. Usage:

python sweep-reads-by-partition-buffered.py -k -x -i -r -b -e <aprx. partitions> -o <output_prefix> -m ...

Hashsize can be quite small; I often go with 1e9 (comes out to 5e8 bytes), as it just needs to be big enough to properly store the assembly graph, and preferably avoid unreasonable spurious edges when traversing from reads. Traversal range should be set high; -1 sets it to max (81 in the current khmer codebase). Buffer size is the total number of reads to keep in all buffers, which is dependent on memory -- 1000000 works quite well. Aprx partitions is a rough guess of the number of partitions (or exact, doesn't matter that much). Max buffers is the number of colors to buffer at any given time; 25000-50000 works well.

Notes on the buffering parameters: the program works by maintaining up to lists of reads in memory, the total of which can reach before files are written. When an individual buffer exceeds / reads, that buffer is written out to its file. If the total number of reads in the buffer exceeds or if the total number of buffers exceeds , all buffers are flushed to files. In practice, with the recommended values, all buffers get flushed at around % 200000 reads, increasing as increases. There is a tradeoff here between the number of buffers increasing query times and flushing all buffers often, though testing suggests that the cost of querying the data structure is minimal, and one is better off to choose -m liberally.

-i, the assembly file, should of course be partitioned, assembled contigs, and file1...fileN should be the reads to be swept. K length is up to user; one might want to choose this based on assembly K (Trinity uses 25).

So, most parameters are for the buffered output; this could be improved (and will be), but for now it works and works quite well. With max traversal range and the full, rather complex lamprey graph, the script chews through about 20m reads/hour, and all in one step. Yay!

[camillescott]

Things currently being worked on:

• Tests for sweep-reads-by-partition-buffered.py
• Traversal optimization
• Read pair awareness for sweep
• Better IO error handling in sweep script
• subclass all sparse labeling code
• Refactor names using 'coloring' to use 'labeling'

[camillescott]

Additional things:

• handle sweeping read short than k size
• general error handling in python glue

[camillescott]

Closing due to moving branch; see #24