[EEG-BIDS] Add support for events.json file & HED Tags #769

jesscall · 2022-03-09T18:16:30Z

Add support to bids_import for `*events.json` file & HED Tags

Scope

This PR refactors how the bids_import handles event files. It adds support for an event.json file in which HED tags can be stored. The PR edits the fetching and inserting of event files and adds an archiving feature similar to the annotation files.

Information

Please reference this documentation for more info on HED Tags in BIDS

Testing

The SQL patches from [EEG] Database Architecture for HED Tags Loris#8036 should be sourced before attempting to test these changes.
Place the sample *events.json file at the root of the RB BIDS directory.
Attempt to test the changes in this PR
Try putting the *events.json file in a subject directory (and renaming) and make sure the import still runs as expected.

…ving of tsv & json

jesscall · 2022-03-15T16:43:06Z

@cmadjar I made the database_lib additions - I haven't had a chance to re-test the import however. I'll try to get to this by end of week

python/lib/database_lib/physiologicaleventparameter.py

jesscall · 2022-05-13T14:16:21Z

Bug:

The events.json is being stored at the individual candidate level. Should be stored only once at the BIDS root level... @cmadjar I haven't had the chance to debug this yet. Any ideas off the top of your head how I can achieve this since you are more familiar with the code?

Also to note:
There will be some more changes coming to this PR to account for HED tag assembly.

jesscall · 2022-06-02T18:52:34Z

This PR now supports assembly of events' HED tags using the HEDtools web service.

python/lib/physiological.py

cmadjar

@jesscall thank you for working on this!!

I tested the following:

no events.json anywhere => 👍
task-faceO_events.json at the root directory of the BIDS dataset => for some reason, the content of physiological_event_parameter_category does not populate Description and HED fields

12	2	boundary	NULL	NULL
13	2	checker-left	NULL	NULL
14	2	checker-right	NULL	NULL
15	2	e-254	NULL	NULL
16	2	e-255	NULL	NULL
17	2	face-inverted	NULL	NULL
18	2	face-upright	NULL	NULL
19	2	house-inverted	NULL	NULL
20	2	house-upright	NULL	NULL
21	2	press-left	NULL	NULL
22	2	press-right	NULL	NULL

I would have expected to see Description="boundary event" and HED="Experiment-procedure" for LevelName="boundary"

sub-OTT501_task-faceO_events.json in the subject directory => tables physiological_event_parameter and physiological_event_parameter_category do not get populated for some reason (physiological_event_file, physiological_event_archive and physiological_task_event get populated though)

File used for testing: the file you sent me on Slack last Friday.

python/lib/database_lib/physiologicaleventfile.py

cmadjar · 2022-06-13T15:29:55Z

python/lib/database_lib/physiologicaleventfile.py

+
+        return event_paths
+
+    def grep_event_file_id_from_event_path(self, event_file_path):


Is event_file_path supposed to be the TSV file?

Also, what would happen if an event file is common to all physiological files? Maybe the wrong eventFileID could be returned?

Example:

EventFileID PhysiologicalFileID FileType FilePath 1 1 tsv <BIDS_ROOT_DIR>/task-faceO_events.tsv 2 1 json <BIDS_ROOT_DIR>/task-faceO_events.json 3 2 tsv <BIDS_ROOT_DIR>/task-faceO_events.tsv 4 2 json <BIDS_ROOT_DIR>/task-faceO_events.json

looking at the return of the function, it looks like it would return EventFileID = 1 (for TSV) or 2 (for JSON) which are linked to PhysiologicalFileID 1 even though this is PhysiologicalFileID 2 that is under process.

yes it takes in the event_tsv file

ok. But what happens in the example above where the same TSV file links to two different PhysiologicalFileID? Would it just return the first PhysiologicalFileID encountered? If so, sounds like a possible major bug, no?

Where do you see the case where two tsv point to the same physioFile?

when the event.tsv file is identical for all EEG files and therefore put directly under root. I believe this is possible in BIDS.

The event.json file can be shared by all subjects. I don't think the events.tsv can ? This means that ALL recordings have the same events/ stimulus? Seems unlikely, but I could be wrong. Do you have an example of this in the spec somewhere?

From the BIDS specs: https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/05-task-events.html

It is also possible to have a single events.tsv file describing events for all participants and runs (see [Inheritance Principle](https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#the-inheritance-principle)).

Fixed by passing in physioFileID as a param in the query

python/lib/database_lib/physiologicaleventparameter.py

python/lib/eeg.py

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

python/lib/physiological.py

jesscall · 2022-06-14T15:25:30Z

@cmadjar I believe this commit af956f7 will address the issue with physiological_event_parameter_category_level saving

python/lib/eeg.py

cmadjar

@jesscall see replies to your comments.

Also, once the LORIS side has been changed to link the file types to ImagingFileTypes, a little reminder to modify the code here to reflect on the DB structure

cmadjar · 2022-06-17T19:00:23Z

python/lib/database_lib/physiologicaleventfile.py

+
+        return event_paths
+
+    def grep_event_file_id_from_event_path(self, event_file_path):


ok. But what happens in the example above where the same TSV file links to two different PhysiologicalFileID? Would it just return the first PhysiologicalFileID encountered? If so, sounds like a possible major bug, no?

python/lib/eeg.py

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

python/lib/eeg.py

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

python/lib/database_lib/physiologicaleventfile.py

cmadjar

LGTM 👍 Waiting for aces/Loris#8036 to be merged on the LORIS to merge this PR.

* refactoring how bids_import handles event files: fetch, insert, archiving of tsv & json * bugfixes! * add event tables to database_lib * lint * missed one * ? * some more fixes after testing import with db libs * hed-assemble events * lint * more lint edits * rm prints * rm print * fixed bug with storing events.json and naming of archive * Apply suggestions from code review Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Fix values for save to parameter_category_level * lint * trying to fix flake8 * flake 8? Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Update python/lib/eeg.py Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * fixing grwp eventFileID function * fix EOL error * keep making syntax errors :( Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

jesscall added 2 commits March 8, 2022 15:25

refactoring how bids_import handles event files: fetch, insert, archi…

cfbe859

…ving of tsv & json

bugfixes!

be6c745

jesscall added A-BIDS Area: BIDS. Issues and pull requests related to BIDS and the BIDS import pipeline A-EEG Area: EEG. Issues and pull requests related to EEG / iEEG processing labels Mar 9, 2022

jesscall added 3 commits March 15, 2022 12:34

add event tables to database_lib

4cf62c6

lint

58b77f3

missed one

514320f

?

4574f71

jesscall commented Mar 15, 2022

View reviewed changes

python/lib/database_lib/physiologicaleventparameter.py Outdated Show resolved Hide resolved

cmadjar added this to the 25.0.0 milestone Mar 16, 2022

jesscall added 2 commits March 22, 2022 14:15

some more fixes after testing import with db libs

022707b

conflicts

d76288f

jesscall mentioned this pull request May 2, 2022

[EEG] Database Architecture for HED Tags aces/Loris#8036

Merged

hed-assemble events

45128f0

jesscall added 3 commits June 2, 2022 14:54

lint

3285b93

more lint edits

ee680a6

rm prints

9c5622c

jesscall commented Jun 2, 2022

View reviewed changes

python/lib/physiological.py Outdated Show resolved Hide resolved

jesscall and others added 2 commits June 2, 2022 15:46

rm print

08b9b7e

fixed bug with storing events.json and naming of archive

073250e

cmadjar requested changes Jun 13, 2022

View reviewed changes

Apply suggestions from code review

5b654bf

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

jesscall commented Jun 14, 2022

View reviewed changes

python/lib/physiological.py Outdated Show resolved Hide resolved

Fix values for save to parameter_category_level

af956f7

jesscall commented Jun 14, 2022

View reviewed changes

python/lib/eeg.py Outdated Show resolved Hide resolved

lint

601ef9b

jesscall commented Jun 14, 2022

View reviewed changes

python/lib/eeg.py Outdated Show resolved Hide resolved

trying to fix flake8

45db229

jesscall commented Jun 14, 2022

View reviewed changes

python/lib/eeg.py Outdated Show resolved Hide resolved

cmadjar requested changes Jun 17, 2022

View reviewed changes

flake 8?

fe81761

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

cmadjar reviewed Jun 20, 2022

View reviewed changes

python/lib/eeg.py Outdated Show resolved Hide resolved

jesscall and others added 2 commits June 20, 2022 13:43

Update python/lib/eeg.py

199e702

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

fixing grwp eventFileID function

6214ffe

jesscall commented Jun 22, 2022

View reviewed changes

python/lib/database_lib/physiologicaleventfile.py Outdated Show resolved Hide resolved

fix EOL error

174b47b

jesscall commented Jun 22, 2022

View reviewed changes

python/lib/database_lib/physiologicaleventfile.py Outdated Show resolved Hide resolved

keep making syntax errors :(

36475ee

cmadjar approved these changes Jun 28, 2022

View reviewed changes

cmadjar added the Blocked Merge it and you die label Jun 28, 2022

cmadjar merged commit c1bbcbd into aces:main Jun 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EEG-BIDS] Add support for events.json file & HED Tags #769

[EEG-BIDS] Add support for events.json file & HED Tags #769

jesscall commented Mar 9, 2022

jesscall commented Mar 15, 2022

jesscall commented May 13, 2022

jesscall commented Jun 2, 2022

cmadjar left a comment

cmadjar Jun 13, 2022

jesscall Jun 14, 2022

cmadjar Jun 17, 2022

jesscall Jun 20, 2022

cmadjar Jun 20, 2022

jesscall Jun 20, 2022

cmadjar Jun 21, 2022

jesscall Jun 22, 2022

jesscall commented Jun 14, 2022

cmadjar left a comment

cmadjar Jun 17, 2022

cmadjar left a comment


		return event_paths

		def grep_event_file_id_from_event_path(self, event_file_path):

[EEG-BIDS] Add support for events.json file & HED Tags #769

[EEG-BIDS] Add support for events.json file & HED Tags #769

Conversation

jesscall commented Mar 9, 2022

Add support to bids_import for *events.json file & HED Tags

Scope

Information

Testing

jesscall commented Mar 15, 2022

jesscall commented May 13, 2022

jesscall commented Jun 2, 2022

cmadjar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesscall commented Jun 14, 2022

cmadjar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmadjar left a comment

Choose a reason for hiding this comment

Add support to bids_import for `*events.json` file & HED Tags