Benchmarking

[AhmetCanSolak]

We need to set up a benchmarking infra which we can run on different contexts and get help to make more educated decisions related to the performance. I'm inclined to use asv: https://github.com/airspeed-velocity/asv, but first wanted to hear more opinions. I want to hear your opinions on both benchmarking frameworks and what aspects of iohub you wish to see in benchmarks.

[JoOkuma]

I think benchmarking compression algorithms might be interesting. However, there are already several benchmarks about this on the web; the only additional information we would get it's if some algorithms might perform better in images, which might yield different performances in fluorescent vs. brightfield images.

I have mixed feelings about benchmarking chunk size, I think it's more processing step/dataset dependent, and a good benchmark will be very difficult to integrate into the CI, because it would require different storages and applications.

I'm happy with asv, but I don't know other benchmarking frameworks.

[ziw-liu]

I haven't been exposed to frameworks other than asv either.

There was some manual benchmarking results done in the time of waveorder.io regarding different Blosc algorithms (speed and ratio) on a single 500 MB dataset, which informed the choice of the default compressor that's inherited by iohub:

https://github.com/czbiohub/iohub/blob/2a48cf597612e72716ea1e6dc9281845e052d6c1/iohub/ngff.py#L386-L393

It's interesting to see how sparsity and patterns of BF, fluorescence, and mixed images will affect the results, and we can potentially recommend different compression schemes for different datasets.

---

By @camFoltz:

Compression level 1:

Compression level 9:

[mattersoflight]

Are you thinking of benchmarks that should be run during CI to catch performance gains or drops? If yes, I'd suggest timing the write and read operations for a 1GB random array so we can evaluate how using different dependencies (zarr-python vs tensorstore) affects the performance when using a single process or multiple processes.

Separately, we do need to know io performance as a function of chunk size and as a function of compression for our HPC infrastructure, specifically when using ESS or scratch space from the compute nodes. This is needed to make sound choices for how to run different pipelines. These benchmarks need not (should not) run on the CI servers. Also useful to evaluate the speed and compression ratios for different modalities of data. That also need not run during CI.

[JoOkuma]

Got it, @mattersoflight . I was thinking about a benchmark with CI.
I think you're right with your suggestion of non-CI benchmarks.