Support for sparse frames without background (XPCS among other)

[kif]

@jonwright can you provide one of your sparse file generated at ID11 to check if it is compatible
@marcocamma same for you at ID10.

[jonwright]

I don't think we are compatible? We usually store a 3D image stack now:

• nnz[nframes] = active pixels on each frame
• row[npixels] = slow pixel direction
• col[npixels] = fast pixel direction
• intensity[npixels] = signal
• optional: labels[npixels] = the peak this pixel is assigned to. Rarely used.

To select the pixels for one frame you need to make the cumsum of nnz (ImageD11.sparseframe.nnz_to_pointer) and slice the [npixels] arrays.

An example for mapping to a lima file: /data/id11/jon/sparse_example/Si_Nscope_calib/scan0001/eiger_0000_sparse.h5
The generating notebook for that file is in /data/id11/jon/sparse_example/demo_ImageD11_sparse.ipynb
These can be shared, I already sent to @t20100 for the compression experts to look at.

Over in ImageD11 (ping @jadball) we should locate some test examples for the many different cases where we have XRDCT data (180 vs 360 degrees for a series of 1D fscan, finterlaced 1D, single fscan2d, single f2scan). It might be worth collecting new data to do that in a clean way.

There was a single frame format that does not have the nnz item but I don't think I used it for a long time now

Some places in ImageD11 that read or write sparse:

ImageD11.sparseframe: SparseScan.__init__ reads a hdf5 file
ImageD11.sparseframe.nnz_to_pointer
ImageD11.sinograms.assemble_label: getsparse( ImageD11.sinograms.dataset, sparsefilenumber )
ImageD11.sinograms.assemble_label: harvest_masterfile ( ... )
ImageD11.sinograms.lima_segmenter:segment_lima  (writes)

ImageD11.sparseframe: sparseframe.to_hdf_group( frame, group )   # single frame
ImageD11.sparseframe: from_hdf_group( group ) # single frame

If you want to generate some noise - it usually needs to be a Poisson distribution with the low count rate. Could be interesting to run sparsify bragg on that silicon data above to see what happens?