Enhancements to the directory tree

[danielballan]

• By default, strip file extension from node name. That is, a.tif becomes a. Rationale: the file extension is not part of the "name". (Windows hides it by default.) Including it makes it confusing to do things like http://.../thing.xlsx?format=csv. This does create the possibility of name collisions, but if you have example.tiff and example.json next to each other, they are likely related --- part of the same "data" --- and need a special Adapter to deal with. Strip suffixes from filenames. #49
• Add a configurable function that, given a full path to a directory, returns a metadata dict. (The default will always return an empty dict.)
• Add a configurable function that, given a full path to a directory, returns a mimetype or None. If it returns None, that means, "Crawl into this subdirectory." If it returns a mimetype, that means, "I recognize this as a {TIFF sequence, Zarr directory, TileDB directory} with internal structure than an Adapter shoudl manage. Do not crawl any further. We might want to make this a list of functions to make it easy to extend default behavior. Support cases where files are not one-to-one with nodes #62

Developed in a phone conversation with @tacaswell

[J-avery32]

Add a configurable function that, given a full path to a directory, returns a metadata dict.

Currently is there no way to add metadata for a node without re-writing the adapter and referencing it in a yml config?

For example, I have a lot of directories, and each has some tiff files and one json file. I would like to use the json file for that directory's metadata.

[danielballan]

Correct. I think what you want here is similar to what we added in #259 — a way to extend the existing behavior without having to rewrite a bunch of stuff.

[danielballan]

I think the feature you need is subdirectory_handler. Does each directory map to a single (3-dimensional) array, comprised off all the TIFFs stacked up, with a dictionary of metadata attached? If so, it should be fit your use case as is. See

tiled/tiled/adapters/tiff.py

Line 178 in 34ede8a

def subdirectory_handler(path):

and the (rough) documentation

https://blueskyproject.io/tiled/explanations/scaling-down.html

If the structure you need is more complex than that, we should talk about how to extend subdirectory_handler to enable it.

[danielballan]

@J-avery32 Does subdirectory_handler address your use case?

[J-avery32]

I'm not sure. How am I supposed to hook this up to read the json file in that directory? Or are you suggesting that I rewrite the Tiff adapter? Even if I re wrote the tiff adapter I don't see any way to give my new subdirectory_handler to tiled to use. Also I will be offline until Friday.

[J-avery32]

Ah! Never mind I think it does fit my use case. I will try it out and let you know.

[J-avery32]

How am I supposed to pass subdirectory_handler to tiled? Is there a hook in the yml for that or am I supposed to create a new DirectoryAdapter: https://blueskyproject.io/tiled/reference/generated/tiled.adapters.files.DirectoryAdapter.html?highlight=subdirectory_handler

[jmaruland]

You can create a config file that call a method which will load the right adapter. An example of a config file could be:

trees:
  - tree: files
    path: /
    args:
      directory: files/
      subdirectory_handler: module:subdirectory_handler_method

To run the config file with tiled you can use (https://blueskyproject.io/tiled/how-to/configuration.html):
tiled serve config my_config_file.yml

[J-avery32]

I think it might be what I want. Originally I had each tiff file be its own separate node but hadn't considered joining it into a single 3 dimensional array. However, there are some csv files in these directories as well which are important. If I were to turn each directory into a 3 dimensional array then it seems that these csv files would disappear.

If there was a way to avoid that issue with the CSVs then this will work.

[danielballan]

@J-avery32 In #511 there are some changes that may be of interest:

1. The feature whereby Tiled automatically updates when the file system changes is now opt in. You need to add --watch to tiled serve directory ... to engage it.
2. The aforementioned subdirectory_handler is gone, in favor of an approach that is both easier to explain and more powerful. I think this is the way to handle "TIFF with associated JSON". Your use case may be a good way to test that and motivate a new documentation page to illustrate it. Can you expand a little on what your directories of files look like?

[J-avery32]

I'll get back to you on this, but it will take me a bit to go back through my old stuff and remember what I was trying to do.

[danielballan]

I'm going to close this now, as I think the use case of "node backed by heterogeneous files" is now addressed nicely.

Docs on how this works at a low level (in the database) are here: https://blueskyproject.io/tiled/explanations/catalog.html#data-source-asset-relation