templates/4b_normalize_image_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8e012798-3632-4107-8d5a-5f47e7671a9d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Image Normalization\n",
    "\n",
    "This notebook will walk you through the process of normalizing your image data.\n",
    "\n",
    "Changes in detector sensitivity over a run can result in different image intensities, even when there are no actual difference in biological signal. To correct for this, we use the median pulse height (MPH) to measure detector sensitivity. We then combine this estimate of sensitivity with an instrument tuning curve to determine the normalization coefficient for each FOV.\n",
    "\n",
    "Before running through the notebook, make sure you've already compensated your data with rosetta using `4_compensate_image_data.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c9751084-7855-4005-b152-55895ff40823",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from toffy import normalize\n",
    "from toffy.panel_utils import load_panel\n",
    "from alpineer.io_utils import list_folders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbe49e0b-34c2-4f86-b786-403c24b2f678",
   "metadata": {},
   "source": [
    "### You'll first need to specify the location of the relevant files to enable image normalization\n",
    " - `run_name` should contain the exact name of the MIBI run to extract from\n",
    " - `panel_path` should point to a panel csv specifying the targets on your panel (see [panel format](https://github.com/angelolab/toffy#panel-format) for more information)\n",
    " - `tuning_curve_file` should point to a tuning curve contained in `toffy/tuning_curves` (`avg_norm_func_450.json`, `avg_norm_func_1300.json`, or `avg_norm_func_2600.json`)\n",
    " - `autogain`: if the Ionpath autogain setting was used for this run\n",
    "\n",
    "`avg_norm_func_2600.json` is the curve that should be used when running the MIBI with Ionpath's default settings. `avg_norm_func_450.json` and `avg_norm_func_1300.json` are curves generated for Angelo Lab-specific parameters. Use the power supply settings to determine which curve is most applicable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f48127b8-84da-4763-9c44-01a5bde4c4ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of the run\n",
    "run_name = '20220101_run_to_be_processed'\n",
    "\n",
    "# Path to user panel\n",
    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'\n",
    "\n",
    "# Name of tuning curve file\n",
    "tuning_curve_file = 'avg_norm_func_2600.json'\n",
    "\n",
    "# Autogain setting of run\n",
    "autogain = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc348f78",
   "metadata": {},
   "source": [
    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f96a31-79cb-46d7-a720-aa013bc0116b",
   "metadata": {},
   "outputs": [],
   "source": [
    "panel = load_panel(panel_path)\n",
    "\n",
    "# These paths should point to the folders containing each step of the processing pipeline\n",
    "bin_base_dir = 'D:\\\\Data'\n",
    "rosetta_base_dir = 'D:\\\\Rosetta_Compensated_Images'\n",
    "normalized_base_dir = 'D:\\\\Normalized_Images'\n",
    "mph_run_dir = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', run_name, 'fov_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62142cdc-fb78-4c21-8cb2-77b3fb60d399",
   "metadata": {},
   "source": [
    "### Within the defined directories, we'll specify the relevant folders based on the run_name provided above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e323824-38a1-4c67-8088-a1725ecbc910",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify sub-folder for rosetta images\n",
    "img_sub_folder = 'rescaled'\n",
    "\n",
    "# create directory to hold normalized images\n",
    "normalized_run_dir = os.path.join(normalized_base_dir, run_name)\n",
    "if not os.path.exists(normalized_run_dir):\n",
    "    os.makedirs(normalized_run_dir)\n",
    "    \n",
    "# create directory to hold associated processing files\n",
    "if not os.path.exists(mph_run_dir):\n",
    "    os.makedirs(mph_run_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26a674b2-e6f1-4886-b93f-0a14f24e86db",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the voltage change across FOVs if autogain set\n",
    "# otherwise, verify the voltage across all FOVs is constant\n",
    "if autogain:\n",
    "    normalize.plot_detector_voltage(\n",
    "        run_folder=os.path.join(bin_base_dir, run_name),\n",
    "        mph_run_dir=os.path.dirname(mph_run_dir)\n",
    "    )\n",
    "else:\n",
    "    normalize.check_detector_voltage(os.path.join(bin_base_dir, run_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "748a2e7c-ecb0-421a-8192-d4bfaf889f37",
   "metadata": {},
   "source": [
    "### Then, we'll loop over each FOV, generating the necessary normalization files if they weren't already created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfe67691-1676-4c3a-93fc-491e3abd6a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all FOVs\n",
    "fovs = list_folders(os.path.join(rosetta_base_dir, run_name), substrs='fov-')\n",
    "\n",
    "# loop over each FOV\n",
    "for fov in fovs:\n",
    "    # generate mph values\n",
    "    mph_file_path = os.path.join(mph_run_dir, fov + '_pulse_heights.csv')\n",
    "    \n",
    "    # base MPH normalization\n",
    "    if not os.path.exists(mph_file_path):\n",
    "        normalize.write_mph_per_mass(base_dir=os.path.join(bin_base_dir, run_name), output_dir=mph_run_dir, \n",
    "                                     fov=fov, masses=panel['Mass'].values, start_offset=0.3, stop_offset=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e60fc2eb-9072-4ffe-a302-7ddb175bf4e7",
   "metadata": {},
   "source": [
    "###  Finally, we'll normalize the images, and save them to the output folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f49779-5e77-4017-8e68-50afce126883",
   "metadata": {},
   "outputs": [],
   "source": [
    "normalize.normalize_image_data(img_dir=os.path.join(rosetta_base_dir, run_name), norm_dir=normalized_run_dir, pulse_height_dir=mph_run_dir,\n",
    "                               panel_info=panel, img_sub_folder=img_sub_folder,\n",
    "                               norm_func_path=os.path.join('..', 'tuning_curves', tuning_curve_file),\n",
    "                               autogain=autogain)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "9e60005e90321f713102c51a1871f113d268f3287359a77b641fffde68437f71"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}