Add support for loading multiple single channel TIFFs

[rj3d]

User story

Most of my images take the form of folders containing multiple standard TIFFs that each contain one channel. It would be nice to be able to visualize these images with Viv and Avivator.

Preferred solution

Add support for loading multiple single channel TIFFs to Viv and Avivator.

[ilan-gold]

Is this a standardized format? Looking at the PR, there is a CSV but who makes that? I see you comment here that it "only supports flat tiffs" but who makes that decision? I don't think we want to be implicitly maintaining a custom format. @manzt any thoughts here? It seems like a use-case that has come up before (flat/individual non OME-TIFF) but we have never really addressed it since we have correctly assessed, given that @rj3d made a PR, that people can write custom loaders.

[rj3d]

Hey! I'm the maintainer of a tool called Mantis Viewer. I met with you and the rest of your group a few months ago and discussed using Vitessce to replace the Mantis Viewer UI. We talked about adding a flat tiff folder loader to Viv so that Vitessce could support loading a folder of flat tiffs as well. I think using a folder of flat tiffs is fairly common in the biological imaging analysis world. Mantis has ~50 users in at least five different scientific groups, and most of the users analyze their data as a folder of flat tiffs. Ideally I wouldn't have to maintain a forked Viv and a forked Vitessce, so I'm open to any changes that you want (that still support multiple flat tiffs) to get this accepted.

The CSV is not a standardized format, and I'm open to changing or replacing that part. The loader has two ways to get files. For local files the user can just drag and drop a bunch of tiffs to the viewer. For remote files, ideally a user would be able to pass in a URL of a directory, but browser JS cant list all of the files in a web directory. I added the CSV file so that users could pass multiple URLs to be opened, but there are other ways this could be accomplished. Maybe a comma separated list of URLs? I'm very open to any suggestions you might have.

[ilan-gold]

Right yes! Ok! I thought the name looked familiar! @keller-mark and I are in the process of updating Vitessce right now to allow for registering custom loaders and much more (mostly Mark, I am just slowly reviewing the flood of PR's). @keller-mark are the loaders exposed yet as a custom format?

[ilan-gold]

So, I'd say, fear not, because your code will get used somewhere and you can stop maintaining two forks!

[keller-mark]

My two cents would be that ideally:

• the Viv core would contain loaders for:
  • individual (non-OME-) TIFFs
  • multi-page (non-OME-) TIFFs
  • BigTIFF

since these are standard (or at least widely used) formats.

Once those loaders are exposed by Viv, it should be very simple for external code to extend / compose the loader for individual TIFFs (for instance in order support a folder of TIFFs based on a custom CSV format).

Any additional metadata required by Viv for rendering (i.e., any properties that are not in the TIFF standard) should be passed to Viv loaders using the standard OME-XML JSON format used by other loaders.

As far as then using a custom Viv loader in Vitessce, yes a plugin file type could be used. The OmeZarrLoader class could be a starting point for a developer who wanted to implement a plugin file type for image files.

[rj3d]

@ilan-gold awesome, glad to hear! Based on @keller-mark's comment, where do you think this loader should live?

I think my PR could be used as, or easily modified to be, an individual TIFF loader. For local files it already acts as one. A user can drag and drop one (or multiple) flat tiffs and they will load. Currently remote TIFFs have to be loaded from a CSV specifying the URLs, but I would be happy to replace that. I included the CSV format because I couldn't figure out how to load multiple TIFFs from a URL. It would be easy to add loading single remote TIFFs, but I could use suggestions on how a user could specify to load multiple remote TIFFs.

We have some users who also use Mantis to visualize multi-page, non-OME tiffs. I was planning to write a loader for these as well if you both open to it.

[ilan-gold]

I think what @keller-mark is saying is that you get to keep the loader in your repo, unfork viv/vitessce, and then just "register" the loader as a plugin file type for Vitessce: http://vitessce.io/docs/dev-plugins/#plugin-file-types

So you'd probably have some sort of TiffCSVSource that you implement that returns the Viv loaders, similar to what Mark linked to in the OmeZarrLoader, which returns a loader and metadata.

[keller-mark]

Right, but I was also suggesting that a Viv loader for standard (non-OME) TIFFs could live in the Viv core. What are your thoughts on that @ilan-gold?

Once that was in place, then a plugin file type for the CSV-based TIFFs would look like:

import { loadTiff } from '@hms-dbmi/viv'; // Non-OME-TIFF loader in Viv core

function loadMultipleTiffsFromCSV(csvData) {
  const myTiffsFromCSV = csvData.map(row => loadTiff(row.url));
  // ...
}

class MultipleTiffsFromCSVLoader extends AbstractTwoStepLoader {
  // ...
}

registerPluginFileType('raster.csv-tiff', 'raster', MultipleTiffsFromCSVLoader);

[ilan-gold]

Ah sorry yes @keller-mark you're right. I think we would be willing to accept a PR for just generic TIFFs but I am not sure what level of abstraction would fit into our repo here. Perhaps just the TiffFolderPixelSource into Viv? Anything above that I think would involve metadata judging by the PR - maybe we would take load since it doesn't involve the CSV but I would leave this to Trevor to decide.

[keller-mark]

Yes I was suggesting the level of abstraction of a core loader that supports individual TIFF files, and then leave it up to external users to implement anything more custom (e.g., for custom folder/CSV structures)

[rj3d]

@manzt what do you think? If you're open to the idea of an individual TIFF loader than I will modify my PR to remove anything that provides metadata outside of the TIFF.

[ilan-gold]

@manzt do you want to weigh in here? I would be willing to accept a more generic tiff loader for loading from a set of files, but I want to get your opinion here. Seems like a generic enough use-case. Perhaps we could make the axis of selection arbitrary (like z-stack instead of c).

[manzt]

Hi all - thanks for your patience. I am on vacation at the moment and was waiting to reply until I'd had a closer look at this thread. I'll be back soon (this weekend) but also don't want to block anything.

[manzt]

Ok, just had a look at the PR and a read through of this thread. Thanks for your patience!

I think it would be generally useful to have a reusable interface for TIFF-based images in Viv's core, which is why I tried to make the TiffPixelSource reusable and decoupled from OME-ness.

TiffPixelSource background

All TIFF images (whether within the same binary file or separate files) are 2D planes, so the idea of the TiffPixelSource was to create an object that:

• reflects a multi-dimensional "stack" of related TIFF planes
• provides a more intuitive API to access specific planes based on a labeled selection for the "stack"

ATiffPixelSource may theoretically combine TIFF planes from one or many files; its just we have only implemented support for assembling TiffPixelSources for OME-TIFFs. In order for 2D planes to be grouped in a TiffPixelSource, they must be the same dtype and shape.

Note multiresolution images are thus represented as an Array<TiffPixelSource> where each item in the array reflects a downsampled resolution.

GeoTIFF-based Indexers

The key is that TiffPixelSource does not manage any GeoTIFF objects directly. Instead, the constructor requires a function that translates a selection (e.g., { z: 0, c: 2, t: 100 }) to a Promise<GeoTIFFImage>.

viv/src/loaders/tiff/pixel-source.ts

Line 25 in 000b9b5

indexer: (sel: PixelSourceSelection<S>) => Promise<GeoTIFFImage>,

We call this special function an Indexer and have two separate implementations depending on the (OME-)TIFF type.

viv/src/loaders/tiff/lib/indexers.ts

Lines 10 to 18 in 000b9b5

	/*
	* An "indexer" for a GeoTIFF-based source is a function that takes a
	* "selection" (e.g. { z, t, c }) and returns a Promise for the GeoTIFFImage
	* object corresponding to that selection.
	*
	* For OME-TIFF images, the "selection" object is the same regardless of
	* the format version. However, modern version of Bioformats have a different
	* memory layout for pyramidal resolutions. Thus, we have two different "indexers"
	* depending on which format version is detected.

Implementing a multi-file TIFF indexer

In a sense we already do support what @keller-mark suggested (since these are supported by GeoTIFF).

the Viv core would contain loaders for:
individual (non-OME-) TIFFs
multi-page (non-OME-) TIFFs
BigTIFF

It's just an Indexer is required to provide semantics over what "dimensions" are, and this requires parsing some additional metadata. So in your case, a load function would initialize a custom multi-file Indexer to accommodate this layout described in the CSV.

A benefit here is that GeoTIFF objects can be instantiated on first access from within the async indexer.

Summary

My hope that we can figure out a solution that reuses the TiffPixelSource to accommodate your use case, rather than creating a new PixelSource for a variant of a format we already support. The primary issue, as identified by @keller-mark, is that some metadata is required to provide semantics to non-xy dimensions of image stacks, and this metadata can be formal (OME-XML) or informal (e.g., your CSV).

Perhaps (a more generally solution) we can think of exposing a custom loadTiffStack API (or similar) that makes this notion of "stacking" tiffs and "indexing" using a selection explicit:

loadTiffStack([
  { source: "http://example.com/c1.tiff", selection: { c: 0 } },
  { source: "http://example.com/c2.tiff", selection: { c: 1 } },
  { source: "http://example.com/c2.tiff", selection: { c: 2 } },
]);

And that way users can parse their own metadata source and reuse the same indexer, etc.

let pixelSource = parseCsv(url)
  .then(urls => urls.map((url, c) => ({ source, selection: { c } })))
  .then(loadTiffStack);

[rj3d]

@manzt, thanks very much for the detailed explanation!

I think I understand, but I want to run what I'm thinking by you to make sure. I want to add the functionality in my TIFF folder loader to the TIFF loader and use the TiffPixelSource instead of my new TiffFolderPixelSource.

To accomplish this, I'm thinking I would modify the TIFF folder indexer I wrote to work with the TiffPixelSource, create a new file in the tiff loader folder called tiff-stack.js (or similar) to handle indexer and metadata creation, and then exposing the new TIFF stack functionality through the tiff loader's index.

Let me know if my understanding and proposed application is correct or if there's anything I should change. If it sounds good to you, I'll make the changes to my PR and report back when I'm done.

[rj3d]

@manzt @keller-mark @ilan-gold what do you all think of my understanding/proposal?

[manzt]

@rj3d This makes sense to me. This feature would make the most sense to me under src/loaders/tiff/tiff-stack.js, which exports loadTiffSack (or similar as you suggested). Users could then import:

import { loadOmeTiff, loadTiffStack, loadOmeZarr } from '@hms-dbmi/viv';

Ideally we could generalize a way to "stack" single-channel tiffs together with an indexer. My hope is to decouple the metadata describing this "virtual" stack from Viv (e.g,. the CSV), and that we can work towards an loadTiffStack API which declares how one or many files are combined. Happy to review whatever you come up with, and thank you!

[rj3d]

@manzt great, thanks! I'll update the PR and ping you when it's done.

[rj3d]

@manzt @ilan-gold @keller-mark just updated PR #610 with the requested restructure. Let me know what you think and if there are any other changes I should make!

[manzt]

Ok great! I'm at a conference this week but should be able to have a look when it's over.

[rj3d]

Sounds good, thanks!

[pace-ls]

Hi there! I got this stacked flat TIFF solution running in my environment, so thank you!

Is there a possibility of easily supporting multiple progressive OME-TIFFs? That's the target I have now. Even better would be to support progressive TIFFs (TIFFs with multiple subresolutions) as well as OME-TIFFs. Can you suggest how I could merge the logic between this and the default Viv behavior which does support a single OME-TIFF, so that we can have multiple OME-TIFFs with progressively-loaded or zoom-loaded subresolutions?

[ilan-gold]

@pace-ls From what I can tell, your main idea here would be to allow for folders of pyramidal tiffs? That seems reasonable.

[pace-ls]

Not necessarily pyramidal, but just multiple single-channel OME-TIFFs that would display as if it were a single OME-TIFF. Like this file for example: https://avivator.gehlenborglab.org/?image_url=https://viv-demo.storage.googleapis.com/LuCa-7color_3x3component_data.ome.tif. It has 4 channels on page load, another 4 in the dropdown, a total of 8 channels in one file. What if instead they were 8 files, each with a single channel, imported like image_url={file1},{file2},{file3},...?

[ilan-gold]

I don't think this is something we want to support in Avivator at the moment since the contribution in #610 was meant to be general i.e you provide the indexing so that {file1},{file2},{file3} could be configured to be a z-stack instead of channels. Now, one option for OME-TIFF would be to allow for multi-file OME-TIFF. There is a field in the OMEXML that allows users to spread different slices of Z/C/T across different files. If you would like to contribute that to the core, we would take that. Then you could just do image_url={file1} where {file1} is the root of the multi-file OME-TIFF. Although to be honest, I'm not too experienced with this, if you wanted to contribute this, we would probably take it since there is nothign custom about OMEXML data https://docs.openmicroscopy.org/ome-model/6.0.1/ome-tiff/data.html#multi-file-ome-tiff-filesets. The issue in #610 is that we didn't want to commit some customized tiff folder implementation to our core. In fact, building a multi-file OMETIFF reader should be pretty simple using #610, but perhaps I am wrong.

[ilan-gold]

Someone else asked about this as well in #580 so seems like it's something we'd be happy to support.