Add ChunkReader::get_bytes #2478
Conversation
| read | ||
| )); | ||
| } | ||
| let buffer = self.reader.get_bytes(front.offset as u64, page_len)?; |
There was a problem hiding this comment.
We can only do this when we have an offset index, as we need to know the size of the page to read. There is a question over whether we could just eagerly fetch the entire column chunk in the latter case, this needs some investigation. It would drastically simplify a lot of the code (it would eliminate FileSource)
Edit: Updated #1163 (comment)
There was a problem hiding this comment.
Cool ! like we call multi-times skip_rows in one page, this should eagerly fetch the entire column in memory.
There was a problem hiding this comment.
If a page is small enough like 1 mb, i guess there is no defect when using eagerly fetch the entire column. looking forward the investigation result!👍
|
Benchmark runs are scheduled for baseline = 6d0ea90 and contender = e80656f. e80656f is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Part of #2463
Rationale for this change
This allows for zero-copy slicing when reading from bytes with an offset index.
What changes are included in this PR?
Are there any user-facing changes?