Add files

[pwalczysko]

Inspired by the comments on https://github.com/IDR/SubmissionWorkflow/pull/27#issuecomment-691962849
started to add the following

• .travis.yml (from https://raw.githubusercontent.com/IDR/idr0054-segura-tonsilhyperion/1d17d297f5f62af5400ec373264eac9561456d11/.travis.yml)
• bulk.yml (from https://raw.githubusercontent.com/IDR/idr0081-georgi-adenovirus/master/bulk.yml)

But on the second look, it seems that I am stopped by lack of knowledge.

The bulk.yml does not seem to be bringing much, as it is not listing any concrete experiments.
Not sure whether or not the .travis.yml is valid either.

Also, did not find any studies which would contain the present files in this template, namely idr0000-experimentB-assays.txt and the idr0000-experimentB-processed.txt (I am aware that there are some assays.txt files, but not inside the particular repos)

This seems to suggest that this template diverged so far from reality of the repos nowadays that it might need a complete rethink ?

cc @sbesson @manics @joshmoore

[pwalczysko]

Should I maybe close this and turn into an issue ?

[manics]

Let's discuss it at the next IDR meeting?

[sbesson]

Summary from this morning's meeting. There are two separate goals at stake here:

• maintaining a template repository in sync with our current layout recommendations for the metadata repositories containing the import/annotation/rendering files and linked from https://github.com/IDR/idr-metadata
• holding the metadata templates for the submitters

After reviewing a few options, the proposed middle ground is to add the extra files to this repository (bulk YML files, rendering files etc) on the one hand and work on creating a reduced archive containing the metadata files only (probably as a GitHub release artifact) that we can point submitters to.

In the mid-term, @francesw mentioned the submission pages at https://idr.openmicroscopy.org/about/ could be updated to consume/link to these metadata files directly.

[pwalczysko]

Adding the various files to this repo, I am finiding that the top level bulk.yml file is typically above a repository, i.e. if I am inside a repo such as idr0030-sero-yap, then the bulk.yml would be at the level of ../. But in this present idr0000 repo, the example bulk.yml will necessarily be inside the repo - or maybe I should not be adding it at all in order not to create a confusion ?

cc @sbesson @manics

[sbesson]

My position is that a top-level bulk file should never point outside its repository. https://github.com/IDR/idr0030-sero-yap is actually a repository that I filtered from the large idr-metadata but didn't have the time to cleanup esp. in terms of the bulk inclusion.

My general rule for all new study repository:

• if there is one experiment/screen, no need for the top-level bulk.yml complexity, a single bulk YML file inside the screen or experiment is sufficient -see https://github.com/IDR/idr0021-lawo-pericentriolarmaterial/blob/master/experimentA/idr0021-experimentA-bulk.yml
• if multiple screens/experiments re-use the same configuration, then common arguments can be refactored into a top-level YML file - see https://github.com/IDR/idr0078-mattiazziusaj-endocyticcomp/blob/master/bulk.yml

[pwalczysko]

Added here some examples from various repos.

Not sure if I should unify the naming to name all files as idr0000... An argument for it would be to keep it unified in this repo. An argument against would be to keep knowledge of the original source study.

This also leads to another question: several concepts are represented here (complex and simple bulk.yml, side-by-side, companion files, processed files, motley assortment of scripts...) which are sometimes mutually exclusive.
Should we have them here as suggested by the commits above and then explain "everything" in the SubmissionWorkflow doc ?

[manics]

I don't think we should be adding anything that's not generic- so scripts should stay in the original repo unless they're generic enough to apply to multiple repositories, in which case they should probably go into https://github.com/IDR/idr-utils. Same applies for XLS or CSV files, if they're not useful as a template for the submitter and they're not generic enough to apply to multiple studies I don't think they should be in here.

[pwalczysko]

thanks @manics , removed in 33bd53c. I am wondering about the experimentB which has (this is from previous, not changed here) the .txt files which are important for submitters ? Shouldnt these txt files be rather in experimentA ? Is it enough to move them there and rename them ...experimentA.txt ?

[manics]

There should be just screenA and experimentB

[pwalczysko]

What is the rationale for not having experimentA but having experimentB instead ?

For a data loader, a template without an experimentA (the default, most used case) makes little sense ?

[manics]

Probably just because A is used for screenA. If you don't think it's confusing we could rename experimentB to experimentA.

[pwalczysko]

Probably just because A is used for screenA. If you don't think it's confusing we could rename experimentB to experimentA.

I do not see any usage of a singular letter A, so obviously did not get the original logic. For me, the original logic would suggest that the "numeration (letterization ?)" of the experiments should start with B which is untrue...

maybe there are some steps here which are escaping me

[sbesson]

Pretty sure the intent of the A, B suffix is supposed to capture some order in between experiments and/or screens. All published IDR studies are still mono-type i.e. screen-only or experiment-only so it will be experimentA/experimentB or screenA/screenB

As we are starting to receive mixed studies, the decision was to keep using this suffix i.e. screenA/experimentB/... or experimentA/screenB/.. are both valid