soundloc_bids.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "42a123af-a52d-4746-ba7e-0a848277ecb9",
   "metadata": {},
   "source": [
    "# BIDSifying the Sound Localization Dataset\n",
    "\n",
    "This notebook will provide a walkthrough of my steps based on consultation with the researcher.\n",
    "\n",
    "Some notes:\n",
    "\n",
    "1. This notebook will be located inside of the project's `code` folder.\n",
    "\n",
    "2. All source files will be placed in the `sourcedata` folder as per BIDS convention.\n",
    "\n",
    "3. Not all steps will generalize to other studies one-to-one, however the general workflow certainly will "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "07c46621-5f4b-4e2e-a216-20f331685ae1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "../sourcedata/CB1_T2_013.bdf  ../sourcedata/CB1_T4_013.bdf\n",
      "../sourcedata/CB1_T3_013.bdf  ../sourcedata/CB1_T5_013.bdf\n"
     ]
    }
   ],
   "source": [
    "!ls ../sourcedata/*013.bdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d836fed0-e43e-41fc-afae-e1568a2613e9",
   "metadata": {},
   "source": [
    "As per the researcher, each file is named as `$COUNTER_BALANCE_$TASK_$SUBJECT.bdf`\n",
    "\n",
    "All four are to be merged together into one file for analysis.\n",
    "\n",
    "Following the \"[04. Convert EEG data to BIDS format](https://mne.tools/mne-bids/stable/auto_examples/convert_eeg_to_bids.html)\" tutorial from MNE-BIDS, I will generate a list of subjects to initialize. However, each item in the list will be a group of the four files to merge together.\n",
    "\n",
    "A researcher can generate this however they choose (e.g. Excel) but I will provide my method below. Note that it depends on the file naming pattern outlined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "19b24bf0-e864-4bb8-a33b-2bbb8d2baaed",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('../sourcedata/CB1_T2_013.bdf',\n",
       "  '../sourcedata/CB1_T3_013.bdf',\n",
       "  '../sourcedata/CB1_T4_013.bdf',\n",
       "  '../sourcedata/CB1_T5_013.bdf'),\n",
       " ('../sourcedata/CB2_T2_014.bdf',\n",
       "  '../sourcedata/CB2_T3_014.bdf',\n",
       "  '../sourcedata/CB2_T4_014.bdf',\n",
       "  '../sourcedata/CB2_T5_014.bdf'),\n",
       " ('../sourcedata/CB3_T2_015.bdf',\n",
       "  '../sourcedata/CB3_T3_015.bdf',\n",
       "  '../sourcedata/CB3_T4_015.bdf',\n",
       "  '../sourcedata/CB3_T5_015.bdf')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob, re # filepath patterns, and regular expressions\n",
    "\n",
    "# This looks complicated, but is just sorting the files based on\n",
    "# the subject ID at the end of the file name\n",
    "sorted_files = sorted(glob.glob('../sourcedata/*.bdf'),\n",
    "                      key=lambda x: re.search('\\d{3}(?=\\.bdf)', x).group())\n",
    "\n",
    "# Takes the sorted list and groups every four items together with no overlaps\n",
    "merge_list = [tuple(sorted_files[i:i+4]) for i in range(0, len(sorted_files), 4)]\n",
    "\n",
    "# Let's see the first three subjects to merge together\n",
    "merge_list[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07b7697d-3f5c-4cda-8c89-5e4e49ed43c2",
   "metadata": {},
   "source": [
    "Now that we have this list, we should be able to merge all files in the group, and then write the output in bids format. We will need to extract the subject ID from the file name again, but it doesn't present a barrier.\n",
    "\n",
    "The function we will be using for merging is `concatenate_raws` and its documentation can be found [here](https://mne.tools/stable/generated/mne.concatenate_raws.html).\n",
    "\n",
    "The following cell will define some required BIDS variables for output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c129a924-45d2-4395-b805-b827999e86cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Required by mne_bids, and is the parent directory as\n",
    "# we are in the \"code\" folder of a BIDS project:\n",
    "bids_root = '..'\n",
    "\n",
    "# Researcher chosen:\n",
    "task_name = 'soundLoc'\n",
    "\n",
    "# Power line frequency as required by BIDS\n",
    "power_line = 60"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd40152-6dcc-414a-b0b7-0d1f4b900094",
   "metadata": {},
   "source": [
    "Now we are onto the actual code required to start BIDSifying each subject.\n",
    "\n",
    "This includes: loading, concatenating, and saving. Most of the lines of code below are taken from the previously mentioned MNE-BIDS tutorial. The only new code is iterating over the list of groups, and extracting the subject ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e5b03685-e8d4-43a6-9388-ec72b9b898b4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB1_T2_013.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB1_T3_013.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB1_T4_013.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB1_T5_013.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Writing '../participants.tsv'...\n",
      "Writing '../participants.json'...\n",
      "The provided raw data contains annotations, but you did not pass an \"event_id\" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass \"event_id\".\n",
      "Used Annotations descriptions: ['BAD boundary', 'EDGE boundary']\n",
      "Writing '../sub-013/eeg/sub-013_task-soundLoc_events.tsv'...\n",
      "Writing '../dataset_description.json'...\n",
      "Writing '../sub-013/eeg/sub-013_task-soundLoc_eeg.json'...\n",
      "Writing '../sub-013/eeg/sub-013_task-soundLoc_channels.tsv'...\n",
      "Copying data files to sub-013_task-soundLoc_eeg.edf\n",
      "Reading 0 ... 842751  =      0.000 ...  1645.998 secs...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_294624/1724751933.py:20: RuntimeWarning: Converting data files to EDF format\n",
      "  mne_bids.write_raw_bids(raw, bids_path, overwrite=True, format='EDF', allow_preload=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing '../sub-013/sub-013_scans.tsv'...\n",
      "Wrote ../sub-013/sub-013_scans.tsv entry with eeg/sub-013_task-soundLoc_eeg.edf.\n",
      "Done subject: 013\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB2_T2_014.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB2_T3_014.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB2_T4_014.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB2_T5_014.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Writing '../participants.tsv'...\n",
      "Writing '../participants.json'...\n",
      "The provided raw data contains annotations, but you did not pass an \"event_id\" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass \"event_id\".\n",
      "Used Annotations descriptions: ['BAD boundary', 'EDGE boundary']\n",
      "Writing '../sub-014/eeg/sub-014_task-soundLoc_events.tsv'...\n",
      "Writing '../dataset_description.json'...\n",
      "Writing '../sub-014/eeg/sub-014_task-soundLoc_eeg.json'...\n",
      "Writing '../sub-014/eeg/sub-014_task-soundLoc_channels.tsv'...\n",
      "Copying data files to sub-014_task-soundLoc_eeg.edf\n",
      "Reading 0 ... 931839  =      0.000 ...  1819.998 secs...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_294624/1724751933.py:20: RuntimeWarning: Converting data files to EDF format\n",
      "  mne_bids.write_raw_bids(raw, bids_path, overwrite=True, format='EDF', allow_preload=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing '../sub-014/sub-014_scans.tsv'...\n",
      "Wrote ../sub-014/sub-014_scans.tsv entry with eeg/sub-014_task-soundLoc_eeg.edf.\n",
      "Done subject: 014\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB3_T2_015.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB3_T3_015.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB3_T4_015.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Extracting EDF parameters from /media/tyler/TylerIsGreat/eegnet/SoundLoc/sourcedata/CB3_T5_015.bdf...\n",
      "BDF file detected\n",
      "Setting channel info structure...\n",
      "Creating raw.info structure...\n",
      "Writing '../participants.tsv'...\n",
      "Writing '../participants.json'...\n",
      "The provided raw data contains annotations, but you did not pass an \"event_id\" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass \"event_id\".\n",
      "Used Annotations descriptions: ['BAD boundary', 'EDGE boundary']\n",
      "Writing '../sub-015/eeg/sub-015_task-soundLoc_events.tsv'...\n",
      "Writing '../dataset_description.json'...\n",
      "Writing '../sub-015/eeg/sub-015_task-soundLoc_eeg.json'...\n",
      "Writing '../sub-015/eeg/sub-015_task-soundLoc_channels.tsv'...\n",
      "Copying data files to sub-015_task-soundLoc_eeg.edf\n",
      "Reading 0 ... 918527  =      0.000 ...  1793.998 secs...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_294624/1724751933.py:20: RuntimeWarning: Converting data files to EDF format\n",
      "  mne_bids.write_raw_bids(raw, bids_path, overwrite=True, format='EDF', allow_preload=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing '../sub-015/sub-015_scans.tsv'...\n",
      "Wrote ../sub-015/sub-015_scans.tsv entry with eeg/sub-015_task-soundLoc_eeg.edf.\n",
      "Done subject: 015\n",
      "Done all subjects!\n"
     ]
    }
   ],
   "source": [
    "import mne, mne_bids # mne for reading bdf, mne_bids for bids\n",
    "\n",
    "# Iterate over each group:\n",
    "for task_group in merge_list[:3]: # For now only do the first three subjects\n",
    "    # Take the first item in the group and grab the subject ID from it\n",
    "    subject_id = re.search('\\d{3}(?=\\.bdf)', task_group[0]).group()\n",
    "    \n",
    "    # Load the first file\n",
    "    raw = mne.io.read_raw_bdf(task_group[0])\n",
    "    # Iterate over the rest of the files in the group...\n",
    "    for other_tasks in task_group[1:]:\n",
    "        # And append them to the first \"raw\" variable\n",
    "        mne.io.concatenate_raws([raw, mne.io.read_raw_bdf(other_tasks)])\n",
    "    \n",
    "    # We now have a merged subject and can do the following:\n",
    "    raw.info['line_freq'] = power_line\n",
    "    \n",
    "    # Write merged subject to BIDS as per tutorial\n",
    "    bids_path = mne_bids.BIDSPath(subject=subject_id, task=task_name, root=bids_root)\n",
    "    mne_bids.write_raw_bids(raw, bids_path, overwrite=True, format='EDF', allow_preload=True)\n",
    "    \n",
    "    print(f'Done subject: {subject_id}')\n",
    "\n",
    "print('Done all subjects!')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}