Optional use_memmap argument to SpyFile read methods

[tboggs]

Depending on an image file's band interleave, a memmap interface is not always faster than doing a direct read from the file. The various read_* methods of SpyFile subclasses should provide an optional use_memmap argument that can be set to False to avoid using the memmap. This argument should be defaulted to True or False based on performance for each read method of each of the SpyFile subclasses.

[tboggs]

Running this script indicates the following speedups using the memmap interface for various read methods:

method             BIL        BIP        BSQ    
------------------------------------------------
read_band          4.75      10.59       2.00   
read_bands         1.12      13.15       0.39   
read_pixel         6.07       1.12       3.91   
read_subregion     4.22       2.80       5.99   
read_subimage      3.01       0.02       1.77   
read_datum         2.15       1.39       1.77   

A few notes about the script:

• The script is very basic and does not attempt to clear any caches but gives a rough indication of where memmaps are faster than direct reads.
• It doesn't address the potential speedups that may occur when data are read repeatedly from a file.
• Results for read_datum are likely unreliable since these read times are already so small.

[tboggs]

Feedback provided by ohspite:

This was all done with your latest branch, in Ubuntu, and with a SSD (except
for the USB stuff at the end).

Whenever I say "clear the disk cache" below, I mean I executed

> sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

I think the first few output listings below are probably not useful for
benchmarks, but I'm including them so you can see the order I did things in
and so we can compare our two machines.

Here is what I get the first time I run the program (or after clearing the cache):

method             BIL        BIP        BSQ
------------------------------------------------
read_band         110.89     15.16       1.23
read_bands        23.74      17.45       0.51
read_pixel        181.48      1.10      100.55
read_subregion    16.49       1.48      17.88
read_subimage      0.23       0.01       0.48
read_datum        48.35       2.37       2.59

And here is second time I run the program:

method             BIL        BIP        BSQ
------------------------------------------------
read_band          2.29       9.00       1.18
read_bands         0.87      17.25       0.17
read_pixel         2.97       0.94       1.34
read_subregion     4.32       1.41       5.95
read_subimage      4.09       0.02       2.83
read_datum         1.66       2.10       2.05

This is fairly similar to your output, though there are some differences where
numbers flip from >1 to <1. It's much different than the first execution
though, because evidently a bunch of stuff is loaded into the disk cache at
this point. The results seem relatively stable if I run it more times without
clearing the cache, and even if I swap the order of execution of the tests (by
uncommenting the 1/True/False line that you left in around line 64).

But, if I swap the order of tests by uncommenting the 1/True/False line and
clear the cache before running it, this is what I get:

method             BIL        BIP        BSQ
------------------------------------------------
read_band          0.02       0.59       0.14
read_bands         0.04      17.26       0.14
read_pixel         0.71       0.96       0.01
read_subregion     0.87       1.42       0.18
read_subimage      0.69       0.02       0.47
read_datum        63.63       1.83       2.11

And from then on (if I don't clear the cache) I get:

method             BIL        BIP        BSQ
------------------------------------------------
read_band          2.27       9.30       1.05
read_bands         0.86      17.54       0.16
read_pixel         3.13       1.05       1.39
read_subregion     4.40       1.49       5.82
read_subimage      4.22       0.02       2.37
read_datum         1.65       4.45       2.00

The main thing that I get from all of this is that, for accurate benchmarks,
the disk cache probably needs to be cleared every time before a file is read.

As a tangent, the only thing I might try to infer from those last two outputs
is that maybe the memmap option is slightly better at repeatedly reading
from the disk cache, since it's typically faster once everything is in the
disk cache. I have no idea why that would be, but I don't know much about how
numpy's memmap is written.

All that to say, I added a line in timed() to clear the disk cache in
between file reads:

def timed(func):
    '''A timer for FileTimer read methods.'''
    import time
    import os
    def wrapped(*args):
        # I know subprocess.call is better, but I don't know how to do
        # stuff like this with it :)
        os.system("sync && echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null")
        # Replace file name with associated SpyFile object
        args = [args[0]] + [open_image(args[1])] + list(args[2:])
        t = time.time()
        func(*args)
        return time.time() - t
    return wrapped

After that change (clearing the cache before every run), these are the numbers
I get.

method             BIL        BIP        BSQ
------------------------------------------------
read_band          0.99       1.01       1.12
read_bands         0.94       1.96       0.68
read_pixel         1.18       0.64       0.41
read_subregion     1.00       1.23       1.17
read_subimage      0.72       0.03       0.54
read_datum         1.52       0.51       0.56

They are (qualitatively, at least) stable for repeated executions. Sometimes
read_subregion is .99, sometimes it's 1.00.

In case you're interested, I also ran the script from a directory on a USB
drive (connected via USB 2.0) so the BSQ/BIL/BIP files were on the USB drive.
These are the results:

method             BIL        BIP        BSQ
------------------------------------------------
[sudo] password for don:
read_band          0.97       1.00       0.99
read_bands         0.42       1.00       0.99
read_pixel         0.91       3.73       0.62
read_subregion     1.16       1.03       1.02
read_subimage      0.15       0.06       0.30
read_datum         4.64       3.64       0.40

In this case, there was a little more variation for repeated executions, so
here's the output from run # 2:

method             BIL        BIP        BSQ
------------------------------------------------
read_band          0.97       1.00       0.98
read_bands         0.43       1.00       0.95
read_pixel         1.09       4.11       0.62
read_subregion     1.16       1.00       1.03
read_subimage      0.15       0.06       0.30
read_datum         4.50       3.49       0.95

[tboggs]

Based on the timing numbers above and the likelihood of making repeat calls on each method, here are the defaults I'll use for the use_memmap argument:

method             BIL        BIP        BSQ
------------------------------------------------
read_band          True       True       True
read_bands         True       True       False
read_pixel         True       True       True
read_subregion     True       True       True
read_subimage      False      False      False
read_datum         True       True       True

read_pixel is questionable but it is the method most likely to be called repeatedly (e.g., when training a classifier on an image that doesn't fit into memory) so I'll leave that one at True.