use mmap to read files when possible

[StefanKarpinski]

It occurs to me that we could potentially mmap files that are mmappable when reading them, potentially improving performance quite drastically. We'd have to test it out to make sure it didn't have major issues, but it could be a big win.

One really great example of where this could be an enormous performance improvement would be when using a regex to scan a file. The normal approach is to real in a chunk at a time and then scan that with the regex. However, that has issues if you can't be sure whether the regex has to match within the chunk. For example, when reading line-by-line, matching a regex that can only match within a line, this is fine — as long as the lines don't get prohibitively long. If the scan is through the entire file, e.g. in the case where the regex is being used to split the file into chunks for consumption, then this can get tricky. Using the mmap approach, you can just map the file and let the regex engine go to work — the kernel will handle getting the data when the regex engine is ready for it!

It would be nice to be able to apply the same trick to streamed data, which cannot be mmapped. However, I think a related trick might work: memory protect the page after the last bit of valid, read data from the file and trap accesses to that memory. Then if the regex engine (or whatever is doing the reading) reads past the valid buffer, you can go ahead and read more data on demand. This allows a very similar trick to be done even with streamed input.

[StefanKarpinski]

@JeffBezanson: does this make any sense?

[JeffBezanson]

Yeah, we can make Arrays with any data pointer, so you can make strings or numeric arrays backed by binary files. Seems to make the most sense when the entire file corresponds to a single object, especially a random-access object. We could define a kind of i/o stream that wraps an Array{Uint8,1}, then an mmapped file can be accessed by both indexing and i/o operations.

[StefanKarpinski]

Cool. This is very 2.0, of course, but I think it could make it possible to do operations on files both efficiently and easily.

[ViralBShah]

This could be very cool if it could be done. I presume that this is done in R. Perhaps matlab only focusses on windows.

[StefanKarpinski]

I'm fairly certain this is not done in R. I don't know of any language that does this. For some reason, people seem to be allergic to mmap. I don't understand why because IMO it's one of the best syscalls every invented.

[JeffBezanson]

In commit d65e897, we have it.

[ViralBShah]

Can we close this one?

[timholy]

Good question. Stefan has some interesting ideas above that go above the current mmap implementation, so I'll let him comment.

[StefanKarpinski]

Let's leave it open as an idea bookmark, although I'm not sure this fits too well with our current I/O model. Would need a lot of deep surgery to make it work, I think.

[quinnj]

I think we can close this issue. We now have fairly accessible Mmap.mmap functionality, which returns an mmapped byte-array for any platform, and you can easily wrap it in an IOBuffer for IO operations. readdlm also utilizes this, as well as SharedArrays. I think we can open new issues for specific areas where we currently aren't leveraging mmap.