[ENH]: Add support for subject(s) file for junifer run

[synchon]

Are you requiring a new dataset or marker?

• I understand this is not a marker or dataset request

Which feature do you want to include?

I think in most data processing projects it is desirable to be able to select specific subsets of all subjects. Sometimes one just wants to get all available data, but sometimes one may want only specific subsets (like for example only unrelated subjects or subjects matched on some other variable). In other cases one may want to just preprocess a specific subset for some initial exploration or testing before getting the full sample.

How do you imagine this integrated in junifer?

Add a subject parameter to in-built data grabbers. Ideally in the full pipeline one can add a list of subjects to the yaml directly, or for example by providing a "subject.txt" file that lists all desired subjects. Something like that.

Do you have a sample code that implements this outside of junifer?

No response

Anything else to say?

No response

[fraimondo]

This was thought already:

junifer/junifer/api/cli.py

Line 54 in ace98da

# TODO: If len == 1, check if its a file, then parse elements from file

The only thing we need to define is the file format.

• CSV?
• TSV?
• both depending on the extension?
• With headers?
• without headers?

Main issue is elements as tuples.

For example, using HCP1200 datagrabber, we can set in the datagrabber task="REST1" and then the iterator of the datagrabber will go through all the element tuples, but only for the specified task:

(sub-000001, REST1, LR)
(sub-000001, REST1, RL)
(sub-000002, REST1, LR)
(sub-000002, REST1, RL)
...

Setting the elements on the CLI or the YAML is just a way of bypassing the iterator. So in this case, the .CSV file with the elements to process should contain all the keys of the element ("subject", "task", "phase_encoding").

Possible solutions:

• Modify the datagrabber so we can also get an iterator using a list, in which some of the keys of the element can be left out. Something like a filter function:

def filter(self, selection) -> Iterator:

then these lines in the run function:

junifer/junifer/api/functions.py

Lines 165 to 172 in ace98da

	with datagrabber_object:
	if elements is not None:
	for t_element in elements:
	mc.fit(datagrabber_object[t_element])
	else:
	for t_element in datagrabber_object:
	mc.fit(datagrabber_object[t_element])

would change for something like:

 with datagrabber_object: 
     if elements is not None: 
         for t_element in datagrabber_object.filter(elements): 
             mc.fit(datagrabber_object[t_element]) 
     else: 
         for t_element in datagrabber_object: 
             mc.fit(datagrabber_object[t_element]) 

• Ask the user to provide the list all elements with all keys in it.

[LeSasse]

What you say makes sense, but I really only mean like setting one parameter of the element, i.e. mostly for subjects, to a list. The rest of the element can still be constructed. So for example, as a user I dont want to construct all elements, but just provide a list of subjects, for example of the HCP in a txt file. Junifer can then construct the elements, for example by using the default values for all other parameters (i.e. [LR, RL] and [REST1..., LAST_TASK]), without me having to make an iterator/file over all elements myself. The reasoning is that subject lists can be quite long and are therefore more difficult to set in the YAML file than other parameters. Having said that, making a csv file of all elements is also easy enough, but in my opinion puts more burden on the user (also in terms of data discovery).

[synchon]

Passing only the subject IDs makes sense to me as this will make it simple for users and also enable one to filter what the DataGrabber iterates over in combination with other parameters that one DataGrabber provides, for example,types, tasks and so on. From the implementation perspective, we already have a good way as @fraimondo shows.

[fraimondo]

Then let's move on that way.

[github-actions]

PR Preview Action v1.4.4
Preview removed because the pull request was closed.
2023-10-10 13:10 UTC

[codecov]

Codecov Report

Merging #182 (8666562) into main (22ca06f) will increase coverage by 0.01%.
Report is 3 commits behind head on main.
The diff coverage is 96.42%.

@@            Coverage Diff             @@
##             main     #182      +/-   ##
==========================================
+ Coverage   93.03%   93.04%   +0.01%     
==========================================
  Files          84       84              
  Lines        3718     3739      +21     
  Branches      724      733       +9     
==========================================
+ Hits         3459     3479      +20     
  Misses        161      161              
- Partials       98       99       +1     

Flag	Coverage Δ
docs	100.00% <ø> (ø)
junifer	93.04% <96.42%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
junifer/api/cli.py	71.28% <100.00%> (+2.13%)	⬆️
junifer/api/functions.py	96.23% <100.00%> (ø)
junifer/datagrabber/base.py	98.36% <93.75%> (-1.64%)	⬇️