Questions regarding dandi organize

[tuanpham96]

Hi, I have a few questions regarding the usage and outcomes of dandi organize.

Q1: Is it possible to use dandi organize without any dandiset initialized?

Users would have to download a random dandiset.yaml to be able to run it even for -f dry mode to see how the results might look.

Maybe there's a way to also initialize dandiset.yaml in a dry/test mode so dandi organize can be operational?

Q2: From my understanding dandi organize changes the file names and folder structures of the original data directory.

• (a) Is there a way to turn that off?
• (b) Or can you point me to the resources explaining the rationale behind this?
• (c) Can you give me some examples of other data repositories that also display similar behaviors?

The reason that I'm asking is because analysis notebooks/code repositories, when published. would usually already assume some data directory organization and naming conventions, for example sometimes the names or formats of folders/files hint at what types of experiments/tasks/areas of recordings.

• (d) Is it possible to define the file names in a more flexible manner? For example, certain metadata (e.g. the brain areas of recordings or the tasks the animals perform in an experiment) may be within a specific data or metadata modules, how would one include that in the name?

The available field names to add seem limited (below) in terms of what researchers might want to annotate to find files quickly without having to parse the files for such metadata.

Click to expand on --required-field

--required-field <field>
Force a given field to be included in the organized filename of any file for which it is nonempty. Can be specified multiple times.
The valid field names are:
- subject_id (already required by default)
- session_id
- tissue_sample_id
- slice_id
- cell_id
- probe_ids
- obj_id
- modalities (already required by default)
- extension (already required by default)

Q3: Follow up on Q2, regarding reproducibility:

• (a) If one were to publish a paper with an accompanying code repository (or many) and dandiset (or many), anyone who wishes to reproduce or play with the notebooks and data, including the original authors, would have to re-parse the paths of these files.
  • Is that correct?
  • Would such process of post-hoc path-parsing potentially error prone and might affect reproducibility?
  • Comment: Sometimes a dandiset may contain a large number of files and a file may take quite some time to even get the metadata out, depending on what's inside it.
• (b) If one were to collect the data while doing analysis, and had to update both the notebooks and dandisets, how would the re-parsing be more streamlined for users? Maybe a database of some sort (e.g. a simple CSV file for re-mapping file paths or contain relevant metadata)

Thanks!

[satra]

1 - I don't think this is possible at the moment, although i think the only thing it cares about is that a file named dandiset.yaml exists. the reason we ask people to create and download a dandiset, is for the subsequent upload step. a local organization i believe can be hacked with just creating an empty dandiset.yaml file in a directory.

2(b) the intent of dandi organize is to move away from ad hoc organization within a lab to a more consistent naming pattern similar to BIDS

2(d) at present this is not possible. however, as with bids, we welcome suggestions to improve but remain systematic and backwards compatible about useful keywords in the filename.

3 - this is exactly why we suggest building analysis notebooks/scripts on the organized data. this allows consistent and perhaps easier adaptability across dandisets. hence acquire, organize, upload, analyze. this is what the openscope project is now doing.

3(post) - at future scales of data, we expect a query will be on the knowledge graph we are creating (not public yet), rather than on the filesystem. the objects will simply be iterators in memory similar to how the current python dandi library works.