State of streaming sequence IO in khmer

[SensibleSalmon]

Currently: There isn't any. Neither current screed (0.7) nor current read_parseres (not seqan) support streaming.

Soon: When screed 0.7.1/screed 1.rc1 is cut/merged in we will have uncompressed and bz2 streaming functionality across all scripts that use screed. gzip streaming was never backported to python 2 and they've said they won't be porting it later. We could go and port it over ourselves.

When seqan is merged in we still won't have read_parser streaming since their high-level interfaces don't provide any streaming stuff. We can look into using lower-level APIs or bothering the seqan folks to make it work in their high-level stuff; I don't believe there's currently a lean either way.

So:

• screed uncompressed fasta
• screed uncompressed fastq
• screed gzip fasta
• screed gzip fastq
• screed bzip fasta
• screed bzip fastq
• read_parser uncompressed fasta
• read_parser uncompressed fastq
• read_parser gzip fasta
• read_parser gzip fastq
• read_parser bzip2 fasta
• read_parser bzip2 fastq

Comments/thoughts?

Relevant to #633
Space check over-riding mentioned in #618 would be useful here (potentially?)

[ctb]

cc #393

[ctb]

Has anyone been in touch with seqan folk to see if they've got it on their roadmap?

[mr-c]

Thank you @bocajnotnef for the testing and writeup. A reminder: we have to add the FASTA / FASTQ dimension to the testing matrix.

@ctb Nope, no contact yet with SeqAn. I guess it is time I wrote a letter of introduction.

[SensibleSalmon]

I cover all the matrix cells in testing, I believe. I shall update the relevant comment.

[ctb]

Agreed re letter of intro.

[ctb]

Do the checkboxes above need to be updated, @mr-c and @bocajnotnef ?

[mr-c]

Checkboxes have been updated.

[ctb]

@bocajnotnef can you update #700 and close this (or explain to me why we shouldn't)? thanks!

[SensibleSalmon]

superseded by #700

[ctb]

@bocajnotnef was there anything to be updated on #700?

[SensibleSalmon]

AFAIK no, it was accurate.