Buffered reverse-complement studies #8045
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n-by-section, though we're still using a small buffer to read the bytes from stdin
|
@chapel-lang/perf-team : anyone want to review? |
mppf
approved these changes
Jan 2, 2018
| numLeft = fi.length(); | ||
| } | ||
|
|
||
| pragma "no copy return" |
There was a problem hiding this comment.
Can you put a comment saying that this returns a view into the buffer starting at low ?
| if avail.size > 0 { | ||
| const idx = _memchr(term, avail); | ||
| if idx >= 0 { | ||
| data.push_back(avail[..idx]); |
There was a problem hiding this comment.
For a while I was confused about this line. Could you add a comment here or at the function level, to indicate that it appends a big block to the array once it's available?
| proc main(args: [] string) { | ||
| const stdin = openfd(0); | ||
| var input = new buf(stdin, readSize); | ||
| var data : [1..0] uint(8); |
There was a problem hiding this comment.
It'd help to have some comments here. For one thing, you're trying to make data have enough storage to store all of the bytes in the file.
4 tasks
mppf
added a commit
that referenced
this pull request
Feb 1, 2018
Add channel.advancePastByte, use it to improve revcomp PR #8045 added some messy rev-comp studies that improve performance by improving the I/O pattern. What the performance comes down to is two things: 1. Copying large chunks of the input from the channel buffer in to the array to be used 2. Using memchr to identify the relevant chunks of the input I experimented with a version that used regexp format strings to replace the memchr call but that had unsatisfying performance. This PR adds channel.advancePastByte in order to enable the expression of the fast I/O pattern in revcomp easily. Now the revcomp version does the following: * "mark" (indicate to the I/O system not to drop the buffer as we might return) * identify the offset of the newline (end of the sequence description) * identify the offset of the > (start of the next sequence) * "revert" (go back to where we "marked") * read the data again in one go with readBytes I'm seeing a 10% speedup for this version beyond revcomp-buf.chpl, and it is much simpler. While there, I noticed that qio_channel_advance might not set up the buffer in some cases, so added code to do that. - [x] full local testing - [x] docs for advancePastByte - [x] update catch statement to specify type - [x] check for 32-bit issues ala #8116 Closes #8105. Reviewed by @benharsh - thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These versions are based off of Rust entries 2 and 3, and mimic Rust's
BufReaderstruct. One version reads entire sections at a time, and the other reads line-by-line. Both versions use a buffer to reduce the number of IO operations, and copy bytes into a given 1D array up until a particular character (> or \n) is found.The rust versions parallelize differently, but these entries are mainly to test an alternate approach to IO.
The line-by-line version is currently performing significantly worse, likely due to slicing overhead. PR #8022 added a test dedicated to measuring creation time for array views.
The 'entire sections at a time' version is competitive with the top C entry.