[ENH] Add support for the BIDS template coordinate frames

[alexrockhill]

Related to mne-tools/mne-bids#961 and mne-tools/mne-bids#962.

Both EEG and iEEG data can be stored in any of the template coordinate frames in BIDS https://bids-specification.readthedocs.io/en/stable/99-appendices/08-coordinate-systems.html#eeg-specific-coordinate-systems. If one of these coordinate frames is used, MNE-Python could have an internal representation for it in transforms.py. The question is what would we need? What comes to mind are 1) a mapping in FIF for roundtrip save/load, 2) the fiducials in this coordinate frame to transform it to other coordinate frames and 3) optionally, the template MR image itself for examples (and potentially also the freesurfer recon but that's going to be quite a bit of data).

Once this is done, a montage object could have apply_trans used and be in that coordinate frame to save to BIDS using mne_bids.write_raw_bids and then could also be read out using mne_bids.read_raw_bids and it would have an official named integer representation and work just like the other coordinate frames.

I think the hardest part would be figuring out the representation in the FIF standard. Do you know if this would be easy or very hard @larsoner or @agramfort?

[larsoner]

Both EEG and iEEG data can be stored in any of the template coordinate frames in BIDS

At least for MEG data, our policy when reading CTF and BTi data -- which have their own coordinate frames -- is to convert these to Neuromag. Then we only need to worry about the HEAD coordinate frame in the end. (IIRC we actually do store some stuff about the original CTF coordinate frame, but I'm not sure we need to generalize that to other systems.) So my inclination would be not to support writing these to FIF, and instead always use our standard coordinate frames.

Then transforms.py can contain the knowledge/functions of how to go from MNE-Python's standard coordinate frames to any others of interest, without needing to support actually writing those to disk (the instances and FIF files on disk continue to use standard coordinate frames). It makes everything much easier at the coding end if we know things are always stored in head, device, or MRI (FreeSurfer surface RAS) space, regardless of where they started at the native acquisition (or BIDS) end.

[alexrockhill]

Okay so then maybe all we need to do is go and collect the fiducials from all those templates so that they can be transformed to head. And then maybe the T1 or downsampled T1 for each could be added to some dataset so that we could add an example to check that coordinates from a BIDS dataset make sense in MNE. Maybe it's own MNE templates data repository? I'd have to look to see if there's already one location where all the templates are stored (and maybe open an issue on BIDS).

I think this would mostly be a PR to mne-bids but it would probably be nice to store the coordinate frames names and transforms to head in MNE-Python.

[larsoner]

Okay so then maybe all we need to do is go and collect the fiducials from all those templates so that they can be transformed to head

The fiducials only allow you to determine a rigid transformation. It's likely that you'll probably want a full affine (at least scaling, maybe also shear) to get from one space to another, assuming you're talking about things like different MRI templates (MNI versus Talairach, etc.)

[alexrockhill]

I thought you could get scaling (at least directionally uniform) with the fiducials.

Anyway, I'm not sure how to compute that, that's all done by Freesurfer from individual T1 to Talairach right?

We have symmetric diffeomorphic mapping but that would be a lot of data to store. I guess it could go in the template data repository...

[larsoner]

I thought you could get scaling (at least directionally uniform) with the fiducials.

Only in two out of three directions (X and Y), which IMO is worse than giving you zero out of three via a rigid. The get_ras_neuromag_t only gives a rigid based on the mapping of three points to three other points, which I think is the right behavior. I don't think just fiducials is enough here unless you know you don't need any scale or shear...

We have symmetric diffeomorphic mapping but that would be a lot of data to store. I guess it could go in the template data repository...

I would check first if there is a standard way of transforming for the (MRI?) space of interest to standard MNI space. My guess is that people live / it's acceptable to use some simple affine transformation for this. And storing 4x4 matrices is trivial and much more permanent / translatable across other platforms / future proof compared to the SDR stuff. I'd really like to avoid having to use SDR if possible.

[larsoner]

... to give a concrete example and sticking to MRI template spaces, I would expect that at least some of these:

https://bids-specification.readthedocs.io/en/stable/99-appendices/08-coordinate-systems.html#standard-template-identifiers

Have some standard affine 4x4 matrix you can use to get to MNI305 space either in the literature, or defined in the code of some software package somewhere. Or at least we can say, we only support the subset of them that do, and just hard-code those affines. If we really need the others, maybe we can use FLIRT or dipy or ANTSpy or whatever to compute the affine from that space to MNI305, whichever gives the best goodness of fit.

[alexrockhill]

Thinking about it, they could just have the fiducials so that they can be transformed to "head" and then saved. Then you just have to worry about the head->template mri transform. Template to template transforms would be optional; I assume that's not really the goal anyway, you probably just want to work in the template you got the data in so that you don't run into transform of a transform problems.

[larsoner]

Thinking about it, they could just have the fiducials so that they can be transformed to "head" and then saved

It sounds like this requires no changes at the MNE-Python end -- just require at the MNE-BIDS end that they have fiducials properly defined/marked in whatever space their electrodes live in.

For convenience, if you want, you could even automagically add them for different templates (like we already have for fsaverage in MNI305 space) at the MNE-BIDS end if they are missing so that people don't have to manually do this.

[alexrockhill]

I think the original issue was that they wanted the montage coordinate frame to say it was in the template space so I think we'd just need named integers for the coordinate frame and then raise an error if you try and save the montage with that coordinate frame maybe?

I agree the magic transforms would be nice but I think you want it to magically work when you save to disk and load but not to magically transform your montage into different coordinates when you think they're in a template mri space.

[larsoner]

I think the original issue was that they wanted the montage coordinate frame to say it was in the template space so I think we'd just need named integers for the coordinate frame and then raise an error if you try and save the montage with that coordinate frame maybe? ... I agree the magic transforms would be nice but I think you want it to magically work when you save to disk and load but not to magically transform your montage into different coordinates when you think they're in a template mri space.

Are we talking about montage.save here? We can just leave that as UNKNOWN, and no transformations are performed. No need to enumerate every possibility.

Are we talking about raw.set_montage(...).save(...)? In MNE-Python when you do raw.set_montage, it will transform to head (or at least try to do so, and warn if it cannot IIRC). If you want to be able to get back to your original coordinate frame, you should use montage.get_native_head_t and keep track of it somewhere. This has been the more or less established way of doing things for a while.

For some datasets like fsaverage, we're "lucky" I suppose in that fsaverage is MNI, so you can just do trans='fsaverage', and this means you are in MNI space and MRI surface RAS for the fsaverage subject. In principle we could decide to have additional helper function get_mri_template_head_trans('MNI152NLin6Sym') that a) knows the fiducial locations for MNI152NLin6Sym, b) is used by MNE-BIDS to set the fiducial locations, and c) uses these to get you the same transformation that a MNI152NLin6Sym montage.get_native_head_t would get you. But I don't see where we end up needing new constants for that coordinate frame, since info will always be in Neuromag head coordinates.

Maybe I still don't quite follow... maybe discussing on Friday would be easier.

[alexrockhill]

Happy to discuss Friday, that all sounds good and the montage.save part answered my question. I think the only thing is to check that if all of the dig points in a raw object have coordinate frame 'MNI152NLin6Sym', that doesn't mess anything up. I thought the coordinate frame had to be an integer but maybe not.

[larsoner]

I think the only thing is to check that if all of the dig points in a raw object have coordinate frame 'MNI152NLin6Sym', that doesn't mess anything up. I thought the coordinate frame had to be an integer but maybe not.

I think you missed / I wasn't clear enough about this point above:

Are we talking about raw.set_montage(...).save(...)? In MNE-Python when you do raw.set_montage, it will transform to head (or at least try to do so, and warn if it cannot IIRC).

So I'm not sure what you mean by "raw object have the coordinate frame 'MNI152NLin6Sym'". When you do raw.set_montage(montage), the electrode locations in the raw object will and should be in head coordinates regardless of what coordinate frame they are in for the montage, at least according to our standard procedures. The fact that these positions in the head coordinate frame can be transformed to some other coordinate frame (like MNI152NLin6Sym) has been lost at that point.

[alexrockhill]

The transform only happens if there are fiducials though and because we added apply_trans, the montage can be moved around to different coordinate frames. I was asking about what happens when the montage is transformed back from head to the template space to be used in things like label montage contacts etc. in the ieeg tutorial.

[larsoner]

The transform only happens if there are fiducials though

... true, but they ideally should be present. If they aren't, I think we are right to warn, as many/most viz functions will do the wrong thing / misrepresent the data, and we are right to call them UNKNOWN because this is more or less an unsupported way of using MNE. (Calling them HEAD is misleading, though it might be what set_montage actually does when setting info['chs'][ii]['coord_frame'].) This is why we warn when using montages this way, and we shouldn't advertise this as a way to use MNE.

the montage can be moved around to different coordinate frames

Ahh so you're talking about transforming the montage, not raw...

I was asking about what happens when the montage is transformed back from head to the template space to be used in things like label montage contacts etc. in the ieeg tutorial.

...I think the best way to make this work is to ensure that you can properly get to and from the HEAD coordinate frame. In other words:

1. Make sure your montage has LPA/Nasion/RPA (maybe adding things to MNE or MNE-BIDS to make this easy for spaces other than mni_tal / MNI305 / fsaverage)
2. MNE transforms to head in raw.set_montage
3. You can always get the transform from head to your native space using montage.get_native_head_t

[alexrockhill]

Ok, now I think we're on the same page. The only question to add to MNE is: can the montage coordinate frame just be a string or does it have to be a signed int supported by MNE?

[larsoner]

It has to be int. Hence I would just leave it as UNKNOWN, but containing LPA/RPA/Nasion is enough to get to/from it and HEAD. The fact that it's 'MNI152NLin6Sym' or any other space can just be in the tutorial narrative.

[alexrockhill]

Okay, good discussion, let's have it all be MNE-BIDS-level tutorial, I agree.

[larsoner]

Hopefully you can convince others that this limited support for coordinate frames is ultimately a good thing and not laziness on our part :)