-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: Proposal for model-<modelname>
directory
#1280
Conversation
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## master #1280 +/- ##
=======================================
Coverage 87.90% 87.90%
=======================================
Files 14 14
Lines 1273 1273
=======================================
Hits 1119 1119
Misses 154 154 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
See bids-standard/bids-bep016#50 for relevant context. Specifically bids-standard/bids-bep016#50 (comment) |
I haven't really integrated this whole proposal yet, but I want to jot down some quick notes about the direction we took in FitLins, which is at least superficially quite different. Not sure if it can apply here at all... In FitLins, we have hierarchical models (e.g., run-level -> subject-level -> dataset-level), such that you could potentially have multiple dataset-level models that build off the results in a common subject-level model. Instead of nesting models more deeply inside modalities, I've been pulling them to the top:
I imagined the |
@arokem If you want to fix up the markdown, use |
Thanks for the quick comments here @effigies! Are these proposals really in conflict? I can imagine having a |
|
the npx remark command doesn't seem to work |
I ran it. Apparently it doesn't fix everything, though... |
I was wondering how this fits with the computational model BEP (if it's related/needs to)? |
As per that BEP: BIDS Computational Model files store the mathematical and computational descriptions of Instead, we want this PR to gauge the interest/position of the BIDS community in the possibility that we add the new There are some deeply related aspects of this PR that we intendedly left for the future:
Those are critical elements that affect this PR, so it's good to have them in mind (and it's a likely possibility that a majority of people push towards having one or both issues covered here too, should it be going towards an accepted decision). It'd be great to have critical reviews from @rwblair and @tsalo, and anyone else involved in the BIDS schema feat. Does this complicate things on that front? Historically, this PR comes from bids-standard/bids-bep016#50. It does not itself fully resolve the inheritance issue described there, but it basically evolves from Option 3 there, which indirectly resolves the problem to the extent it is necessary within BEP016. |
model-<modelname>
directory model-<modelname>
directory
@arokem @Lestropie I have allowed myself to remove the WIP label from the title. I have made some edits to use macros and make the text more consistent with the rest of the spec. For a render of this PR at this point, go to https://output.circle-artifacts.com/output/job/ecd7481c-1064-41ad-a51b-e362395503e4/artifacts/0/dev_docs/01-introduction.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These will be json files, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few initial thoughts:
- The schema doesn't have mechanism for the additional folder layer you're proposing, so we'd need to extend it. I think it would be helpful to have a general concept associated with this level beyond "models". If this is a layer the community wants to use for models, then presumably it will be applicable to other types of derivatives.
- I think you need a
models
suffix inschema/objects/suffixes.yaml
, as well as rules for it- perhaps those could go inschema/rules/tabular_metadata.yaml
.
Following up on bids-standard#1280, this PR differentiates between model parameters and model-derived parameters. This distinction arises from considerations described in more detail in BEP16. For example, the distinction between the 6 elements of the tensor that are fit in the process of DTI, and secondary calculations of FA and MD from the eigenvalues of the tensor.
Following up on bids-standard#1280, this PR differentiates between model parameters and model-derived parameters. This distinction arises from considerations described in more detail in BEP16. For example, the distinction between the 6 elements of the tensor that are fit in the process of DTI, and secondary calculations of FA and MD from the eigenvalues of the tensor.
Following up on bids-standard#1280, this PR differentiates between model parameters and model-derived parameters. This distinction arises from considerations described in more detail in BEP16. For example, the distinction between the 6 elements of the tensor that are fit in the process of DTI, and secondary calculations of FA and MD from the eigenvalues of the tensor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the suffix model
or models
?
@@ -669,6 +669,17 @@ meg: | |||
description: | | |||
Unprocessed MEG data stored in the native file format of the MEG instrument | |||
with which the data was collected. | |||
models: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be model
in singular?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think so, the file is called _models.tsv
.
As opposed to that, there will be a new model
entry in this file in the context of #1282.
"sub-<label>_models.tsv": "", | ||
"sub-<label>_models.json": "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, in singular?
"sub-<label>_models.tsv": "", | |
"sub-<label>_models.json": "" | |
"sub-<label>_model.tsv": "", | |
"sub-<label>_model.json": "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(I don't think so, see above)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OIC. I got confused. The description makes it sound like there is one models.tsv
file for the dataset. Here, we introduce the notion that different subjects (And different datatypes?) can have their own sub-<label>_models.tsv
file (and <datatype>_models.tsv
is also allowable?). We might want to describe this before providing an example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, when I added the sub-<>_
prefix I actually disallowed a /models.tsv
implicitly. I'll send some changes to make this explicit.
Dear all, by chance I stumbled on this BEP, and as I'm working on various BIDS-related software, I wanted to put my 5 cents. My concern is sub-folders like I would propose to put models into separate data-type folder together with the sidecar json and summury tsv file:
This way the model data is introduced exactly the same way as any other data file in the dataset, simplifying parsing.
With convention that For the group level models
Cheers, |
I don't think there's anything in this PR that is not supported by the current specifications. It's true that the BIDS Validator may require some extra heuristic, but that's more a problem of being able to formalize BIDS rules into a schema that can be used directly by the validator. Since this only affects BIDS-Derivatives (and that's clearly stated at a couple of places in this PR) I don't think this would break any existing software either, basically because there's no specification to break at this point. Finally, I'll also mention that this
I think these suggestions fit better in #1282. Can you check that out? Thanks a lot for the input, happy to clarify further if the argument is not sufficiently convincing. |
Indeed, the derivatives in BIDS are not regulated. Imagine you want a script that compare different models inside the same dataset. In opposite, you need to do recursive search for the folders containing I'm speaking less from human and more from machine perspective.
My point was about |
How this proposal deviates from BIDS style specifically?
BIDS should remain agnostic to the decisions made by the software developer. This idea comes from the fact that scientists use folders typically to cluster the outputs of important models in their analysis. They typically group by subject first, and that's why
Human: goes to the >>> layout.get_models(datatype="func")
[ list of all model identifiers under datatype func ] |
Co-authored-by: Taylor Salo <tsalo006@fiu.edu>
This is the relevant article in the open-access Stanford Encyclopedia of Philosophy.
Co-authored-by: Ariel Rokem <arokem@gmail.com>
2c7ebd4
to
104abd5
Compare
For the time being, I have rebased this on top of current master to facilitate more discussion of this proposal. @tsalo : regarding your previous comment, should I add a new file called "models.yaml" in here and try to follow the example of the existing files in that folder? |
Update on the status of this PR: I brought this for discussion with the steering group. It met with some skepticism, particularly as to how this would apply to other modalities that we have not considered in our proposal here. I suggest that we close this PR, and that we try to deal with the issues that this PR proposes to address through the introduction of suffixes, rather than this rather extensive change to the spec and to the schema. I guess that different models can be also thought of as different pipelines, and thus separated from each other at that level for the purpose of ease of navigation. Closing this PR would open up the path to make progress on the connectivity derivatives BEPs through the addition of suffixes that described model-fit parameters (i.e., |
As an auxilliary to work being done on BEP16 (DWI derivatives), we are proposing a change to the common derivatives specification to allow the creation of a sub-directory into which model parameters will be saved, together with a
models.tsv
metadata file to describe all the different models that are fit within a pipeline.EDIT BY @oesteban
We arrived to this proposal as a way of partially address the issues identified by @Lestropie (see bids-standard/bids-bep016#50), building from his proposal n. 3.