Handling Arbitrary (>n, where n is not necessarily hardcoded) Numbers of Channels

[ilan-gold]

Right now we only support 6 channels and while I think this is fine for a variety of reasons, I am opening this issue for anyone who is interested in investigating/creating a fix for this since it could be fun.

Right now we are blocked from doing this on one shader via an array of textures by #95 (this also would only fix the issue for WebGL2).

In theory we could also set up shader-building code that adds lines of code to the current shaders for channel7, channel8 etc. by just injecting code into our shaders (since the code is fairly repetitive) for the sampling, compositing, uniforms etc.

A final option would be using separate XRLayer layers for each channel. I think this is not the way to go for a few reasons:

1. I believe this would require separate shader passes for each layer (i.e each channel) which could (and probably would) affect performance.
2. We need to be able to blend channels, which would need to be controlled through a WebGL setting (thus preventing us from doing alpha-composited multi-modal overlays as the setting is global).
3. I don't know how we do viridis magma etc. type colormapping. In this case blending would not work since we sum up all the intensities and then apply the colormap as opposed to blending the application of the colormap.
4. We would have to uproot large swaths of our API.

I am not planning on doing any of these things in the near future but if anyone is curious, I figured I'd put my thoughts into Github issues to give a jumping off point. I think the real way to solve this problem is probably the second option since the first one is only valid for WebGL2, and I think it would be easier than the luma.gl fix.

If I have any of this wrong, feel free to comment!

[andreasg123]

As a preliminary step, would it be possible to support nine channels for now? I don't have an example image but I conducted a search on how many colors are possible. 8-Plex, 9-Color Field of View Slide Staining is described in the linked document.

[ilan-gold]

@andreasg123 If I understand you right, I think you are right that the number of channels in an image can be greater than 6. However the number of channels can grow even larger, up to ~40 (in my experience) like here or here. I think the reason we didn't go above 6 was that
a) it's hard to distinguish different stains/colors when the number of channels is ~>6
b) OMERO has a limit of 6 in (some of) their viewers so it seemed reasonable
If you want we can consider going above that. Does the max number of 9 still stand? Another issue that I may not have mentioned is that more channels is more data, so for multiscale images or z-stack images, changing the view (zoom/pan or z-stack respectively) takes longer with more channels, although I suppose that's not a reason not to bump the limit.

[andreasg123]

I agree that it is uncommon to have more than 6 channels. However, if adding a few more channels would not be much work as you described above, I think that 9 channels would be a good limit for the near future.

If a solution can be found that would allow for an unlimited number of channels without a performance impact for few channels, that would be preferable. In the meantime, a limit slightly higher than 6 would be nice.

[ilan-gold]

Ok @andreasg123 that can be done.

[ilan-gold]

Apparently, deck.gl offers hooks to temporarily change blending parameters per draw call: visgl/deck.gl#5450 (comment)
See: https://github.com/visgl/deck.gl/blob/b1a068120e53220892616b0977de4d27c49913b2/modules/core/src/lib/layer.js#L742-L751
This needs serious consideration. The likely thorn in our side appears to be the additive colormapping. I'm not sure how to/if we can blend the RGB outputs of the colormaps properly since it internally relies on clipping the summed up intensities before applying the colormap.

[manzt]

Closing to track this issue in #687