Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancements to the directory tree #48

Closed
2 of 3 tasks
danielballan opened this issue Aug 4, 2021 · 12 comments
Closed
2 of 3 tasks

Enhancements to the directory tree #48

danielballan opened this issue Aug 4, 2021 · 12 comments

Comments

@danielballan
Copy link
Member

danielballan commented Aug 4, 2021

  • By default, strip file extension from node name. That is, a.tif becomes a. Rationale: the file extension is not part of the "name". (Windows hides it by default.) Including it makes it confusing to do things like http://.../thing.xlsx?format=csv. This does create the possibility of name collisions, but if you have example.tiff and example.json next to each other, they are likely related --- part of the same "data" --- and need a special Adapter to deal with. Strip suffixes from filenames. #49
  • Add a configurable function that, given a full path to a directory, returns a metadata dict. (The default will always return an empty dict.)
  • Add a configurable function that, given a full path to a directory, returns a mimetype or None. If it returns None, that means, "Crawl into this subdirectory." If it returns a mimetype, that means, "I recognize this as a {TIFF sequence, Zarr directory, TileDB directory} with internal structure than an Adapter shoudl manage. Do not crawl any further. We might want to make this a list of functions to make it easy to extend default behavior. Support cases where files are not one-to-one with nodes #62

Developed in a phone conversation with @tacaswell

@J-avery32
Copy link

J-avery32 commented Jul 27, 2022

Add a configurable function that, given a full path to a directory, returns a metadata dict.

Currently is there no way to add metadata for a node without re-writing the adapter and referencing it in a yml config?

For example, I have a lot of directories, and each has some tiff files and one json file. I would like to use the json file for that directory's metadata.

@danielballan
Copy link
Member Author

Correct. I think what you want here is similar to what we added in #259 — a way to extend the existing behavior without having to rewrite a bunch of stuff.

@danielballan
Copy link
Member Author

I think the feature you need is subdirectory_handler. Does each directory map to a single (3-dimensional) array, comprised off all the TIFFs stacked up, with a dictionary of metadata attached? If so, it should be fit your use case as is. See

def subdirectory_handler(path):

and the (rough) documentation

https://blueskyproject.io/tiled/explanations/scaling-down.html

If the structure you need is more complex than that, we should talk about how to extend subdirectory_handler to enable it.

@danielballan
Copy link
Member Author

@J-avery32 Does subdirectory_handler address your use case?

@J-avery32
Copy link

I'm not sure. How am I supposed to hook this up to read the json file in that directory? Or are you suggesting that I rewrite the Tiff adapter? Even if I re wrote the tiff adapter I don't see any way to give my new subdirectory_handler to tiled to use. Also I will be offline until Friday.

@J-avery32
Copy link

Ah! Never mind I think it does fit my use case. I will try it out and let you know.

@J-avery32
Copy link

How am I supposed to pass subdirectory_handler to tiled? Is there a hook in the yml for that or am I supposed to create a new DirectoryAdapter: https://blueskyproject.io/tiled/reference/generated/tiled.adapters.files.DirectoryAdapter.html?highlight=subdirectory_handler

@jmaruland
Copy link
Collaborator

jmaruland commented Aug 9, 2022

You can create a config file that call a method which will load the right adapter. An example of a config file could be:

trees:
  - tree: files
    path: /
    args:
      directory: files/
      subdirectory_handler: module:subdirectory_handler_method

To run the config file with tiled you can use (https://blueskyproject.io/tiled/how-to/configuration.html):
tiled serve config my_config_file.yml

@J-avery32
Copy link

J-avery32 commented Aug 12, 2022

I think it might be what I want. Originally I had each tiff file be its own separate node but hadn't considered joining it into a single 3 dimensional array. However, there are some csv files in these directories as well which are important. If I were to turn each directory into a 3 dimensional array then it seems that these csv files would disappear.

If there was a way to avoid that issue with the CSVs then this will work.

@danielballan
Copy link
Member Author

@J-avery32 In #511 there are some changes that may be of interest:

  1. The feature whereby Tiled automatically updates when the file system changes is now opt in. You need to add --watch to tiled serve directory ... to engage it.
  2. The aforementioned subdirectory_handler is gone, in favor of an approach that is both easier to explain and more powerful. I think this is the way to handle "TIFF with associated JSON". Your use case may be a good way to test that and motivate a new documentation page to illustrate it. Can you expand a little on what your directories of files look like?

@J-avery32
Copy link

I'll get back to you on this, but it will take me a bit to go back through my old stuff and remember what I was trying to do.

@danielballan
Copy link
Member Author

I'm going to close this now, as I think the use case of "node backed by heterogeneous files" is now addressed nicely.

Docs on how this works at a low level (in the database) are here: https://blueskyproject.io/tiled/explanations/catalog.html#data-source-asset-relation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants