head-and-mouth/Head and Mouth Cancer Datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bf7462b8",
   "metadata": {},
   "source": [
    "# Head and Mouth Cancer Datasets\n",
    "\n",
    "This Jupyter Notebook builds a dataset relating to head and mouth cancers. This category was chosen because it is easy to define on the Proteomic Data Commons and the Imaging Data Commons, but will need to be manually aggregated on the Genomic Data Commons.\n",
    "\n",
    "## Dataset definition\n",
    "\n",
    "This dataset is defined as:\n",
    "* On the Proteomic Data Commons: all cases whose [`primary_site` is set to `Head and Neck`](https://pdc.cancer.gov/pdc/browse/filters/primary_site:Head+and+Neck)\n",
    "* On the Imaging Data Commons: all cases whose [Primary Site Location is `Head-Neck`](https://portal.imaging.datacommons.cancer.gov/explore/?filters_for_load=%5B%7B%22filters%22:%5B%7B%22id%22:%22128%22,%22values%22:%5B%22Head-Neck%22%5D%7D%5D%7D%5D)\n",
    "    * This isn't currently included, since I don't think the IDC API is publicly accessible yet.\n",
    "* On the Genomics Data Commons: [all cases](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22base%20of%20tongue%22%2C%22floor%20of%20mouth%22%2C%22gum%22%2C%22hypopharynx%22%2C%22larynx%22%2C%22lip%22%2C%22nasal%20cavity%20and%20middle%20ear%22%2C%22nasopharynx%22%2C%22oropharynx%22%2C%22other%20and%20ill-defined%20sites%20in%20lip%2C%20oral%20cavity%20and%20pharynx%22%2C%22other%20and%20unspecified%20major%20salivary%20glands%22%2C%22other%20and%20unspecified%20parts%20of%20mouth%22%2C%22other%20and%20unspecified%20parts%20of%20tongue%22%2C%22palate%22%2C%22tonsil%22%5D%7D%7D%5D%7D) whose `primary_site` is set to one of:\n",
    "    * [`Other and unspecified parts of major salivary glands`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22other%20and%20unspecified%20major%20salivary%20glands%22%5D%7D%7D%5D%7D)\n",
    "    * [`Other and ill-defined sites in lip, oral cavity and pharynx`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22other%20and%20ill-defined%20sites%20in%20lip%2C%20oral%20cavity%20and%20pharynx%22%5D%7D%7D%5D%7D)\n",
    "    * [`Oropharynx`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22oropharynx%22%5D%7D%7D%5D%7D)\n",
    "    * [`Larynx`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22larynx%22%5D%7D%7D%5D%7D)\n",
    "    * [`Other and unspecified parts of tongue`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22other%20and%20unspecified%20parts%20of%20tongue%22%5D%7D%7D%5D%7D)\n",
    "    * [`Nasopharynx`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22nasopharynx%22%5D%7D%7D%5D%7D)\n",
    "    * [`Floor of mouth`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22floor%20of%20mouth%22%5D%7D%7D%5D%7D)\n",
    "    * [`Tonsil`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22tonsil%22%5D%7D%7D%5D%7D)\n",
    "    * [`Other and unspecified parts of mouth`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22other%20and%20unspecified%20parts%20of%20mouth%22%5D%7D%7D%5D%7D)\n",
    "    * [`Nasal cavity and middle ear`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22nasal%20cavity%20and%20middle%20ear%22%5D%7D%7D%5D%7D)\n",
    "    * [`Hypopharynx`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22hypopharynx%22%5D%7D%7D%5D%7D)\n",
    "    * [`Base of tongue`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22base%20of%20tongue%22%5D%7D%7D%5D%7D)\n",
    "    * [`Gum`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22gum%22%5D%7D%7D%5D%7D)\n",
    "    * [`Lip`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22lip%22%5D%7D%7D%5D%7D)\n",
    "    * [`Palate`](https://portal.gdc.cancer.gov/exploration?filters=%7B%22op%22%3A%22and%22%2C%22content%22%3A%5B%7B%22op%22%3A%22in%22%2C%22content%22%3A%7B%22field%22%3A%22cases.primary_site%22%2C%22value%22%3A%5B%22palate%22%5D%7D%7D%5D%7D)\n",
    "\n",
    "Note that we specifically don't include the brain, eye, trachea or esophagus -- please let us know if you think they should be included in this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e03ee8c",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "16f9f4cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# Install pandas\n",
    "#!{sys.executable} -m pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "69c75165",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load required packages.\n",
    "import requests\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6501297f",
   "metadata": {},
   "source": [
    "## Download PDC data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9d50c64b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 257 case IDs, of which 148 are unique.\n"
     ]
    }
   ],
   "source": [
    "pdc_graphql_endpoint = \"https://pdc.cancer.gov/graphql\"\n",
    "\n",
    "# Step 1. Get a list of all the case IDs relevant to us.\n",
    "primary_site = \"Head and Neck\"\n",
    "primary_site_query = \"\"\"{ uiCase(primary_site: \"%s\") { case_id } }\"\"\" % (primary_site)\n",
    "\n",
    "response = requests.get(pdc_graphql_endpoint, params={\"query\": primary_site_query})\n",
    "result = json.loads(response.content)\n",
    "\n",
    "case_ids = list(map(lambda case: case[\"case_id\"], result[\"data\"][\"uiCase\"]))\n",
    "unique_case_ids = np.unique(case_ids)\n",
    "print(f\"We have {len(case_ids)} case IDs, of which {len(unique_case_ids)} are unique.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6ca85977",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded case 1 of 148: 0232701d-6d00-440c-af6c-5899fbbf4142\n",
      "Downloaded case 2 of 148: 0e943de7-c277-48f2-8fa9-b2e836b03c2c\n",
      "Downloaded case 3 of 148: 1104505a-9890-49ce-8d7d-7a8070261324\n",
      "Downloaded case 4 of 148: 195cd133-0d53-402d-b31c-3d4fe0481858\n",
      "Downloaded case 5 of 148: 1df726a4-8520-4474-8c00-d238a7384be1\n",
      "Downloaded case 6 of 148: 2b4204ab-87af-4a79-913f-fabdcb2d02b4\n",
      "Downloaded case 7 of 148: 3195cf36-998a-4f48-a7f1-a84f2b9fdcc5\n",
      "Downloaded case 8 of 148: 37ea85c2-5e02-4d7c-846a-7cd40ef1912d\n",
      "Downloaded case 9 of 148: 3f8aba34-9229-4701-bd5d-869b3705d4a6\n",
      "Downloaded case 10 of 148: 4633e71f-5fbe-49b0-9721-9769182a08eb\n",
      "Downloaded case 11 of 148: 482dd5ba-9c50-4ee3-a3f2-3bb0ad6487dc\n",
      "Downloaded case 12 of 148: 4c848c55-2c7e-4d32-9a17-b2531a18a19a\n",
      "Downloaded case 13 of 148: 58355fa4-e2ad-434b-a915-acbd09abbb77\n",
      "Downloaded case 14 of 148: 59448f1d-6be8-466f-bb1a-1a3aef693d4f\n",
      "Downloaded case 15 of 148: 5e99d311-24f2-4ee5-abfe-c9d40771e8e3\n",
      "Downloaded case 16 of 148: 6e4c8397-715c-4dc5-8782-2437ff207241\n",
      "Downloaded case 17 of 148: 71ebb445-ec2c-4ee1-aeaa-9cf30bb3bfff\n",
      "Downloaded case 18 of 148: 7c02730b-5240-41ae-88b3-3f4e4514aa21\n",
      "Downloaded case 19 of 148: 7e16a4fa-3063-4ead-95e5-922f2cce6fac\n",
      "Downloaded case 20 of 148: 82b7bff1-63d8-47f3-8520-bee0c1b6b14e\n",
      "Downloaded case 21 of 148: 8369a540-474e-4b9b-8f2b-f96153ac8bfe\n",
      "Downloaded case 22 of 148: 8782e6f7-d072-4107-8688-2289d001557a\n",
      "Downloaded case 23 of 148: 8ac42bc3-472d-42e8-8455-09528a9977ea\n",
      "Downloaded case 24 of 148: 9548db2a-702d-46a0-8d02-bfdc497bdca8\n",
      "Downloaded case 25 of 148: 96fdbd9a-c9a5-4db9-8e0d-6ea200f975db\n",
      "Downloaded case 26 of 148: 9bad78c0-6cb9-45bf-85e8-86cffe0b96fc\n",
      "Downloaded case 27 of 148: b2a945e3-e5cb-47d3-b74c-8bbf03fd33f2\n",
      "Downloaded case 28 of 148: b5988f8a-f71d-43f8-9474-51a8974e0bf7\n",
      "Downloaded case 29 of 148: b9c7c8c2-271c-4eae-87f2-19a89b1ac58f\n",
      "Downloaded case 30 of 148: c7a90e98-6f13-4480-8f69-74eea9e6efa6\n",
      "Downloaded case 31 of 148: ca087e3a-9e96-48e3-a28d-10bfaa39bcf1\n",
      "Downloaded case 32 of 148: ced7d55d-fcb5-4b4a-8ba2-aaca18b958d7\n",
      "Downloaded case 33 of 148: d5b43f05-8489-44b4-8b3c-81a428f17fbd\n",
      "Downloaded case 34 of 148: df4e907e-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 35 of 148: df4e95d2-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 36 of 148: df4e970b-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 37 of 148: df4e9966-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 38 of 148: df4e9a58-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 39 of 148: df4e9b42-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 40 of 148: df4e9d3d-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 41 of 148: df4e9e22-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 42 of 148: df4e9ef8-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 43 of 148: df4ea101-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 44 of 148: df4ea1e7-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 45 of 148: df4ea3f5-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 46 of 148: df4ea4dc-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 47 of 148: df4ea7d8-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 48 of 148: df4ea9fe-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 49 of 148: df4eaafe-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 50 of 148: df4eac21-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 51 of 148: df4eaeb4-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 52 of 148: df4eaf8f-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 53 of 148: df4eb063-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 54 of 148: df4eb20d-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 55 of 148: df4eb2e7-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 56 of 148: df4eb3f1-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 57 of 148: df4eb5b7-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 58 of 148: df4eb697-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 59 of 148: df4eb76a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 60 of 148: df4eb92f-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 61 of 148: df4eba07-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 62 of 148: df4ebbd7-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 63 of 148: df4ebcaf-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 64 of 148: df4ebebb-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 65 of 148: df4ebfa3-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 66 of 148: df4ec190-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 67 of 148: df4ec26a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 68 of 148: df4ec459-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 69 of 148: df4ec539-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 70 of 148: df4ec739-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 71 of 148: df4ec812-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 72 of 148: df4eca22-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 73 of 148: df4ecafb-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 74 of 148: df4ecd30-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 75 of 148: df4ece54-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 76 of 148: df4ecf32-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 77 of 148: df4ed0d5-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 78 of 148: df4ed1b2-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 79 of 148: df4ed282-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 80 of 148: df4ed42c-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 81 of 148: df4ed504-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 82 of 148: df4ed5d3-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 83 of 148: df4ed786-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 84 of 148: df4ed85e-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 85 of 148: df4eda1a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 86 of 148: df4edaf6-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 87 of 148: df4edcd7-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 88 of 148: df4eddbb-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 89 of 148: df4edf88-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 90 of 148: df4ee063-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 91 of 148: df4ee240-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 92 of 148: df4ee338-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 93 of 148: df4ee534-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 94 of 148: df4ee623-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 95 of 148: df4ee821-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 96 of 148: df4ee8fe-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 97 of 148: df4eeb29-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 98 of 148: df4eec12-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 99 of 148: df4eee28-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 100 of 148: df4eef1c-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 101 of 148: df4ef039-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 102 of 148: df4ef10e-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 103 of 148: df4ef2ab-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 104 of 148: df4ef386-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 105 of 148: df4ef458-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 106 of 148: df4ef6b1-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 107 of 148: df4ef7ac-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 108 of 148: df4ef87b-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 109 of 148: df4efa74-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 110 of 148: df4efb5a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 111 of 148: df4efd1a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 112 of 148: df4efdf1-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 113 of 148: df4effbc-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 114 of 148: df4f0094-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 115 of 148: df4f0263-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 116 of 148: df4f0338-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 117 of 148: df4f0514-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 118 of 148: df4f05ee-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 119 of 148: df4f07d3-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 120 of 148: df4f08ad-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 121 of 148: df4f0a9f-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 122 of 148: df4f0b7a-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 123 of 148: df4f0d7f-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 124 of 148: df4f0e55-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 125 of 148: df4f1067-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 126 of 148: df4f1155-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 127 of 148: df4f1268-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 128 of 148: df4f1344-8f98-11ea-b1fd-0aad30af8a83\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded case 129 of 148: df4f14e8-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 130 of 148: df4f15be-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 131 of 148: df4f1689-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 132 of 148: df4f1830-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 133 of 148: df4f1907-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 134 of 148: df4f19d3-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 135 of 148: df4f1b8c-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 136 of 148: df4f1c60-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 137 of 148: df4f1d2d-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 138 of 148: df4f1ef8-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 139 of 148: df4f1fff-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 140 of 148: df4f20dd-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 141 of 148: df4f22b3-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 142 of 148: df4f238b-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 143 of 148: df4f2567-8f98-11ea-b1fd-0aad30af8a83\n",
      "Downloaded case 144 of 148: df6bef95-c233-4b10-b321-36ef4e79b5d4\n",
      "Downloaded case 145 of 148: e11e9155-4ac6-43dc-b8e5-1be822cd2dab\n",
      "Downloaded case 146 of 148: ea7c9fbd-8353-4f3c-9fea-2fba79140536\n",
      "Downloaded case 147 of 148: f581075d-1b69-4812-9fe4-2bde4aad8bf2\n",
      "Downloaded case 148 of 148: ff5a631f-93b9-486e-beb9-d16737db8c9e\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "148"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 2. Get all the cases relevant to us.\n",
    "\n",
    "# I got this query from https://pdc.cancer.gov/data-dictionary/publicapi-documentation/#!/Case/case\n",
    "case_query = \"\"\"{\n",
    "    case (\n",
    "        case_id: \"%s\"\n",
    "        acceptDUA: true\n",
    "    ) {\n",
    "        case_id case_submitter_id project_submitter_id days_to_lost_to_followup disease_type\n",
    "        index_date lost_to_followup primary_site\n",
    "        externalReferences {\n",
    "            external_reference_id\n",
    "            reference_resource_shortname reference_resource_name reference_entity_location\n",
    "        }\n",
    "        demographics {\n",
    "            demographic_id ethnicity gender demographic_submitter_id race cause_of_death days_to_birth\n",
    "            days_to_death vital_status year_of_birth year_of_death\n",
    "        }\n",
    "        samples {\n",
    "            sample_id sample_submitter_id sample_type sample_type_id gdc_sample_id gdc_project_id\n",
    "            biospecimen_anatomic_site composition current_weight days_to_collection days_to_sample_procurement\n",
    "            diagnosis_pathologically_confirmed freezing_method initial_weight intermediate_dimension is_ffpe\n",
    "            longest_dimension method_of_sample_procurement oct_embedded pathology_report_uuid preservation_method\n",
    "            sample_type_id shortest_dimension time_between_clamping_and_freezing time_between_excision_and_freezing\n",
    "            tissue_type tumor_code tumor_code_id tumor_descriptor\n",
    "            aliquots {\n",
    "                aliquot_id aliquot_submitter_id analyte_type\n",
    "                aliquot_run_metadata {\n",
    "                    aliquot_run_metadata_id\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "        diagnoses {\n",
    "            diagnosis_id tissue_or_organ_of_origin age_at_diagnosis primary_diagnosis tumor_grade tumor_stage\n",
    "            diagnosis_submitter_id classification_of_tumor days_to_last_follow_up days_to_last_known_disease_status\n",
    "            days_to_recurrence last_known_disease_status morphology progression_or_recurrence\n",
    "            site_of_resection_or_biopsy prior_malignancy ajcc_clinical_m ajcc_clinical_n ajcc_clinical_stage\n",
    "            ajcc_clinical_t ajcc_pathologic_m ajcc_pathologic_n ajcc_pathologic_stage ajcc_pathologic_t\n",
    "            ann_arbor_b_symptoms ann_arbor_clinical_stage ann_arbor_extranodal_involvement ann_arbor_pathologic_stage\n",
    "            best_overall_response burkitt_lymphoma_clinical_variant circumferential_resection_margin\n",
    "            colon_polyps_history days_to_best_overall_response days_to_diagnosis days_to_hiv_diagnosis\n",
    "            days_to_new_event figo_stage hiv_positive hpv_positive_type hpv_status iss_stage laterality\n",
    "            ldh_level_at_diagnosis ldh_normal_range_upper lymph_nodes_positive lymphatic_invasion_present\n",
    "            method_of_diagnosis new_event_anatomic_site new_event_type overall_survival perineural_invasion_present\n",
    "            prior_treatment progression_free_survival progression_free_survival_event residual_disease\n",
    "            vascular_invasion_present year_of_diagnosis icd_10_code synchronous_malignancy\n",
    "            tumor_largest_dimension_diameter\n",
    "        }\n",
    "    }\n",
    "}\"\"\"\n",
    "\n",
    "cases = []\n",
    "for num, case_id in enumerate(unique_case_ids):\n",
    "    response = requests.get(\n",
    "        pdc_graphql_endpoint, params={\"query\": case_query % (case_id)}\n",
    "    )\n",
    "    result = json.loads(response.content)\n",
    "    case = result[\"data\"][\"case\"]\n",
    "    if case and len(case) == 1:\n",
    "        cases.append(case[0])\n",
    "        print(\"Downloaded case %d of %d: %s\" % (num + 1, len(unique_case_ids), case_id))\n",
    "    else:\n",
    "        print(\n",
    "            \"Could not download case %d of %d: %s\"\n",
    "            % (num + 1, len(unique_case_ids), case_id)\n",
    "        )\n",
    "\n",
    "len(cases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b4799c34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before we examine it, let's save this as a file.\n",
    "\n",
    "with open(\"pdc-head-and-mouth.json\", \"w\") as file:\n",
    "    json.dump(cases, file, indent=2, sort_keys=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a22f4f83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((148, 12),\n",
       " ['case_id',\n",
       "  'case_submitter_id',\n",
       "  'project_submitter_id',\n",
       "  'days_to_lost_to_followup',\n",
       "  'disease_type',\n",
       "  'index_date',\n",
       "  'lost_to_followup',\n",
       "  'primary_site',\n",
       "  'externalReferences',\n",
       "  'demographics',\n",
       "  'samples',\n",
       "  'diagnoses'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df_pdc_cases = pd.DataFrame(cases)\n",
    "df_pdc_cases = pd.json_normalize(cases)\n",
    "df_pdc_cases.shape, list(df_pdc_cases.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "046a94b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>case_id</th>\n",
       "      <th>case_submitter_id</th>\n",
       "      <th>project_submitter_id</th>\n",
       "      <th>days_to_lost_to_followup</th>\n",
       "      <th>disease_type</th>\n",
       "      <th>index_date</th>\n",
       "      <th>lost_to_followup</th>\n",
       "      <th>primary_site</th>\n",
       "      <th>externalReferences</th>\n",
       "      <th>demographics</th>\n",
       "      <th>samples</th>\n",
       "      <th>diagnoses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0232701d-6d00-440c-af6c-5899fbbf4142</td>\n",
       "      <td>OSCC_13</td>\n",
       "      <td>Oral Squamous Cell Carcinoma - Chang Gung Univ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Oral Squamous Cell Carcinoma</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': 'c4f66cfa-2d1d-4d58-b7b6-8...</td>\n",
       "      <td>[{'sample_id': 'd58e2a88-8b0c-4cc4-bb1a-e7734a...</td>\n",
       "      <td>[{'diagnosis_id': '90972454-6af2-4704-bb16-c00...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0e943de7-c277-48f2-8fa9-b2e836b03c2c</td>\n",
       "      <td>OSCC_25</td>\n",
       "      <td>Oral Squamous Cell Carcinoma - Chang Gung Univ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Oral Squamous Cell Carcinoma</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': 'de8038df-cf33-46c2-a9eb-8...</td>\n",
       "      <td>[{'sample_id': 'b6ecae8d-08a9-44e3-ac70-51331b...</td>\n",
       "      <td>[{'diagnosis_id': 'ae7a3f6c-173c-44d3-8a3e-381...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1104505a-9890-49ce-8d7d-7a8070261324</td>\n",
       "      <td>OSCC_23</td>\n",
       "      <td>Oral Squamous Cell Carcinoma - Chang Gung Univ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Oral Squamous Cell Carcinoma</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': 'efef05c6-84a9-4353-95d0-2...</td>\n",
       "      <td>[{'sample_id': '4dc515ba-6525-4619-8d57-520cc2...</td>\n",
       "      <td>[{'diagnosis_id': '4c1cade2-18a6-4a2e-8264-949...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>195cd133-0d53-402d-b31c-3d4fe0481858</td>\n",
       "      <td>OSCC_37</td>\n",
       "      <td>Oral Squamous Cell Carcinoma - Chang Gung Univ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Oral Squamous Cell Carcinoma</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': 'd978bb14-8056-4f56-ac5b-b...</td>\n",
       "      <td>[{'sample_id': '5836c294-187d-4a17-96da-e1ee30...</td>\n",
       "      <td>[{'diagnosis_id': 'e993cc6a-3774-4ef6-9f6a-77e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1df726a4-8520-4474-8c00-d238a7384be1</td>\n",
       "      <td>OSCC_06</td>\n",
       "      <td>Oral Squamous Cell Carcinoma - Chang Gung Univ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Oral Squamous Cell Carcinoma</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': '5b24c6ed-ff68-4d1a-8aad-4...</td>\n",
       "      <td>[{'sample_id': '37d52145-52cc-40f9-8d6e-e13f3d...</td>\n",
       "      <td>[{'diagnosis_id': '60422ff9-30a2-41da-ac80-783...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                case_id case_submitter_id  \\\n",
       "0  0232701d-6d00-440c-af6c-5899fbbf4142           OSCC_13   \n",
       "1  0e943de7-c277-48f2-8fa9-b2e836b03c2c           OSCC_25   \n",
       "2  1104505a-9890-49ce-8d7d-7a8070261324           OSCC_23   \n",
       "3  195cd133-0d53-402d-b31c-3d4fe0481858           OSCC_37   \n",
       "4  1df726a4-8520-4474-8c00-d238a7384be1           OSCC_06   \n",
       "\n",
       "                                project_submitter_id days_to_lost_to_followup  \\\n",
       "0  Oral Squamous Cell Carcinoma - Chang Gung Univ...                     None   \n",
       "1  Oral Squamous Cell Carcinoma - Chang Gung Univ...                     None   \n",
       "2  Oral Squamous Cell Carcinoma - Chang Gung Univ...                     None   \n",
       "3  Oral Squamous Cell Carcinoma - Chang Gung Univ...                     None   \n",
       "4  Oral Squamous Cell Carcinoma - Chang Gung Univ...                     None   \n",
       "\n",
       "                   disease_type index_date lost_to_followup   primary_site  \\\n",
       "0  Oral Squamous Cell Carcinoma       None             None  Head and Neck   \n",
       "1  Oral Squamous Cell Carcinoma       None             None  Head and Neck   \n",
       "2  Oral Squamous Cell Carcinoma       None             None  Head and Neck   \n",
       "3  Oral Squamous Cell Carcinoma       None             None  Head and Neck   \n",
       "4  Oral Squamous Cell Carcinoma       None             None  Head and Neck   \n",
       "\n",
       "  externalReferences                                       demographics  \\\n",
       "0                 []  [{'demographic_id': 'c4f66cfa-2d1d-4d58-b7b6-8...   \n",
       "1                 []  [{'demographic_id': 'de8038df-cf33-46c2-a9eb-8...   \n",
       "2                 []  [{'demographic_id': 'efef05c6-84a9-4353-95d0-2...   \n",
       "3                 []  [{'demographic_id': 'd978bb14-8056-4f56-ac5b-b...   \n",
       "4                 []  [{'demographic_id': '5b24c6ed-ff68-4d1a-8aad-4...   \n",
       "\n",
       "                                             samples  \\\n",
       "0  [{'sample_id': 'd58e2a88-8b0c-4cc4-bb1a-e7734a...   \n",
       "1  [{'sample_id': 'b6ecae8d-08a9-44e3-ac70-51331b...   \n",
       "2  [{'sample_id': '4dc515ba-6525-4619-8d57-520cc2...   \n",
       "3  [{'sample_id': '5836c294-187d-4a17-96da-e1ee30...   \n",
       "4  [{'sample_id': '37d52145-52cc-40f9-8d6e-e13f3d...   \n",
       "\n",
       "                                           diagnoses  \n",
       "0  [{'diagnosis_id': '90972454-6af2-4704-bb16-c00...  \n",
       "1  [{'diagnosis_id': 'ae7a3f6c-173c-44d3-8a3e-381...  \n",
       "2  [{'diagnosis_id': '4c1cade2-18a6-4a2e-8264-949...  \n",
       "3  [{'diagnosis_id': 'e993cc6a-3774-4ef6-9f6a-77e...  \n",
       "4  [{'diagnosis_id': '60422ff9-30a2-41da-ac80-783...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pdc_cases.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a12dec24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>case_id</th>\n",
       "      <th>case_submitter_id</th>\n",
       "      <th>project_submitter_id</th>\n",
       "      <th>days_to_lost_to_followup</th>\n",
       "      <th>disease_type</th>\n",
       "      <th>index_date</th>\n",
       "      <th>lost_to_followup</th>\n",
       "      <th>primary_site</th>\n",
       "      <th>externalReferences</th>\n",
       "      <th>demographics</th>\n",
       "      <th>samples</th>\n",
       "      <th>diagnoses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>0</td>\n",
       "      <td>148</td>\n",
       "      <td>110</td>\n",
       "      <td>110</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>111</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "      <td>148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>0232701d-6d00-440c-af6c-5899fbbf4142</td>\n",
       "      <td>OSCC_13</td>\n",
       "      <td>CPTAC3-Discovery</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Head and Neck Squamous Cell Carcinoma</td>\n",
       "      <td>Diagnosis</td>\n",
       "      <td>No</td>\n",
       "      <td>Head and Neck</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'demographic_id': 'c4f66cfa-2d1d-4d58-b7b6-8...</td>\n",
       "      <td>[{'sample_id': 'd58e2a88-8b0c-4cc4-bb1a-e7734a...</td>\n",
       "      <td>[{'diagnosis_id': '90972454-6af2-4704-bb16-c00...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>110</td>\n",
       "      <td>NaN</td>\n",
       "      <td>110</td>\n",
       "      <td>110</td>\n",
       "      <td>72</td>\n",
       "      <td>148</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     case_id case_submitter_id  \\\n",
       "count                                    148               148   \n",
       "unique                                   148               148   \n",
       "top     0232701d-6d00-440c-af6c-5899fbbf4142           OSCC_13   \n",
       "freq                                       1                 1   \n",
       "\n",
       "       project_submitter_id days_to_lost_to_followup  \\\n",
       "count                   148                        0   \n",
       "unique                    2                        0   \n",
       "top        CPTAC3-Discovery                      NaN   \n",
       "freq                    110                      NaN   \n",
       "\n",
       "                                 disease_type index_date lost_to_followup  \\\n",
       "count                                     148        110              110   \n",
       "unique                                      2          1                3   \n",
       "top     Head and Neck Squamous Cell Carcinoma  Diagnosis               No   \n",
       "freq                                      110        110               72   \n",
       "\n",
       "         primary_site externalReferences  \\\n",
       "count             148                148   \n",
       "unique              1                111   \n",
       "top     Head and Neck                 []   \n",
       "freq              148                 38   \n",
       "\n",
       "                                             demographics  \\\n",
       "count                                                 148   \n",
       "unique                                                148   \n",
       "top     [{'demographic_id': 'c4f66cfa-2d1d-4d58-b7b6-8...   \n",
       "freq                                                    1   \n",
       "\n",
       "                                                  samples  \\\n",
       "count                                                 148   \n",
       "unique                                                148   \n",
       "top     [{'sample_id': 'd58e2a88-8b0c-4cc4-bb1a-e7734a...   \n",
       "freq                                                    1   \n",
       "\n",
       "                                                diagnoses  \n",
       "count                                                 148  \n",
       "unique                                                148  \n",
       "top     [{'diagnosis_id': '90972454-6af2-4704-bb16-c00...  \n",
       "freq                                                    1  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pdc_cases.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5974df",
   "metadata": {},
   "source": [
    "## Download GDC data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f05589db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warnings: {}\n"
     ]
    }
   ],
   "source": [
    "# Search by cases.primary_site.\n",
    "cases_endpt = \"https://api.gdc.cancer.gov/cases\"\n",
    "\n",
    "field_groups = [\"diagnoses\", \"samples\", \"demographic\"]\n",
    "field_groups = \",\".join(field_groups)\n",
    "\n",
    "filters = {\n",
    "    \"op\": \"in\",\n",
    "    \"content\": {\n",
    "        \"field\": \"cases.primary_site\",\n",
    "        \"value\": [\n",
    "            \"baseoftongue\",\n",
    "            \"floorofmouth\",\n",
    "            \"gum\",\n",
    "            \"hypopharynx\",\n",
    "            \"larynx\",\n",
    "            \"lip\",\n",
    "            \"nasalcavityandmiddleear\",\n",
    "            \"nasopharynx\",\n",
    "            \"oropharynx\",\n",
    "            \"otherandill-definedsitesinlip,oralcavityandpharynx\",\n",
    "            \"otherandunspecifiedmajorsalivaryglands\",\n",
    "            \"otherandunspecifiedpartsofmouth\",\n",
    "            \"otherandunspecifiedpartsoftongue\",\n",
    "            \"palate\",\n",
    "            \"tonsil\",\n",
    "        ],\n",
    "    },\n",
    "}\n",
    "\n",
    "params = {\n",
    "    \"filters\": json.dumps(filters),\n",
    "    \"expand\": field_groups,\n",
    "    \"format\": \"JSON\",\n",
    "    \"size\": 1000,\n",
    "}\n",
    "\n",
    "response = requests.get(cases_endpt, params=params)\n",
    "result = json.loads(response.content)\n",
    "\n",
    "print(f\"Warnings: {result['warnings']}\")\n",
    "\n",
    "gdc_entries = result[\"data\"][\"hits\"]\n",
    "\n",
    "with open(\"gdc-head-and-mouth.json\", \"w\") as file:\n",
    "    json.dump(gdc_entries, file, indent=2, sort_keys=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56db7c8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((560, 45),\n",
       " ['id',\n",
       "  'lost_to_followup',\n",
       "  'days_to_lost_to_followup',\n",
       "  'disease_type',\n",
       "  'submitter_id',\n",
       "  'submitter_aliquot_ids',\n",
       "  'aliquot_ids',\n",
       "  'diagnoses',\n",
       "  'diagnosis_ids',\n",
       "  'created_datetime',\n",
       "  'sample_ids',\n",
       "  'samples',\n",
       "  'submitter_sample_ids',\n",
       "  'primary_site',\n",
       "  'submitter_diagnosis_ids',\n",
       "  'updated_datetime',\n",
       "  'case_id',\n",
       "  'index_date',\n",
       "  'state',\n",
       "  'demographic.cause_of_death',\n",
       "  'demographic.race',\n",
       "  'demographic.vital_status',\n",
       "  'demographic.ethnicity',\n",
       "  'demographic.gender',\n",
       "  'demographic.age_at_index',\n",
       "  'demographic.submitter_id',\n",
       "  'demographic.days_to_birth',\n",
       "  'demographic.created_datetime',\n",
       "  'demographic.year_of_birth',\n",
       "  'demographic.cause_of_death_source',\n",
       "  'demographic.premature_at_birth',\n",
       "  'demographic.weeks_gestation_at_birth',\n",
       "  'demographic.demographic_id',\n",
       "  'demographic.age_is_obfuscated',\n",
       "  'demographic.updated_datetime',\n",
       "  'demographic.occupation_duration_years',\n",
       "  'demographic.days_to_death',\n",
       "  'demographic.state',\n",
       "  'demographic.year_of_death',\n",
       "  'slide_ids',\n",
       "  'submitter_slide_ids',\n",
       "  'analyte_ids',\n",
       "  'submitter_analyte_ids',\n",
       "  'portion_ids',\n",
       "  'submitter_portion_ids'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pd_gdc_entries = pd.DataFrame(gdc_entries)\n",
    "df_gdc_entries = pd.json_normalize(gdc_entries)\n",
    "df_gdc_entries.shape, list(df_gdc_entries.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fa7fdee9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>lost_to_followup</th>\n",
       "      <th>days_to_lost_to_followup</th>\n",
       "      <th>disease_type</th>\n",
       "      <th>submitter_id</th>\n",
       "      <th>submitter_aliquot_ids</th>\n",
       "      <th>aliquot_ids</th>\n",
       "      <th>diagnoses</th>\n",
       "      <th>diagnosis_ids</th>\n",
       "      <th>created_datetime</th>\n",
       "      <th>...</th>\n",
       "      <th>demographic.occupation_duration_years</th>\n",
       "      <th>demographic.days_to_death</th>\n",
       "      <th>demographic.state</th>\n",
       "      <th>demographic.year_of_death</th>\n",
       "      <th>slide_ids</th>\n",
       "      <th>submitter_slide_ids</th>\n",
       "      <th>analyte_ids</th>\n",
       "      <th>submitter_analyte_ids</th>\n",
       "      <th>portion_ids</th>\n",
       "      <th>submitter_portion_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a203ac35-914f-4f4d-816c-2af124257500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Squamous Cell Neoplasms</td>\n",
       "      <td>GENIE-DFCI-011620</td>\n",
       "      <td>[GENIE-DFCI-011620-10763_aliquot]</td>\n",
       "      <td>[3d4995b8-5b04-46f2-8d37-7e0b9f9b1b1a]</td>\n",
       "      <td>[{'irs_stage': None, 'iss_stage': None, 'ajcc_...</td>\n",
       "      <td>[a4f6276a-b3cc-45f9-9fb8-30edd56ad4ea]</td>\n",
       "      <td>2018-09-13T13:41:51.057497-05:00</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26d5f693-dfbc-44ec-a073-49a59a3f09a0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Squamous Cell Neoplasms</td>\n",
       "      <td>GENIE-DFCI-050738</td>\n",
       "      <td>[GENIE-DFCI-050738-234120_aliquot]</td>\n",
       "      <td>[57d18da1-d1b9-40b0-8ee6-0f94fd9f7575]</td>\n",
       "      <td>[{'irs_stage': None, 'iss_stage': None, 'ajcc_...</td>\n",
       "      <td>[2e4427b7-e557-49c7-85ef-39a02c4a441c]</td>\n",
       "      <td>2019-06-03T12:43:36.681258-05:00</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d7c7ecbd-7495-4d29-8bb6-78797f5a47eb</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Squamous Cell Neoplasms</td>\n",
       "      <td>GENIE-DFCI-004072</td>\n",
       "      <td>[GENIE-DFCI-004072-413_aliquot]</td>\n",
       "      <td>[95066691-03ea-422a-bb4c-ba09e9cbd7ab]</td>\n",
       "      <td>[{'irs_stage': None, 'iss_stage': None, 'ajcc_...</td>\n",
       "      <td>[85d63c6e-a6c1-47a5-92a9-216719073a4a]</td>\n",
       "      <td>2018-09-13T13:44:12.915115-05:00</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33fa625e-852e-49ef-8134-6ea46edb5183</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Squamous Cell Neoplasms</td>\n",
       "      <td>GENIE-GRCC-a8pxs0u6</td>\n",
       "      <td>[GENIE-GRCC-a8pxs0u6-sample-a_aliquot]</td>\n",
       "      <td>[b17a8d8a-395e-4d42-bfb0-7e829e1d4a8b]</td>\n",
       "      <td>[{'irs_stage': None, 'iss_stage': None, 'ajcc_...</td>\n",
       "      <td>[d0678ad6-4a4a-4ec8-9e5f-352c1347efc2]</td>\n",
       "      <td>2019-06-04T18:08:22.482657-05:00</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4d49b9f5-09a0-49de-84f7-0d3441c214f6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Squamous Cell Neoplasms</td>\n",
       "      <td>GENIE-GRCC-2b4655c3</td>\n",
       "      <td>[GENIE-GRCC-2b4655c3-sample-a_aliquot]</td>\n",
       "      <td>[a0f16f51-94eb-4a6b-beb6-9159fa6acc4b]</td>\n",
       "      <td>[{'irs_stage': None, 'iss_stage': None, 'ajcc_...</td>\n",
       "      <td>[2707217b-5a4b-404b-97de-921526581d45]</td>\n",
       "      <td>2018-10-02T17:53:10.070290-05:00</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  lost_to_followup  \\\n",
       "0  a203ac35-914f-4f4d-816c-2af124257500               NaN   \n",
       "1  26d5f693-dfbc-44ec-a073-49a59a3f09a0               NaN   \n",
       "2  d7c7ecbd-7495-4d29-8bb6-78797f5a47eb               NaN   \n",
       "3  33fa625e-852e-49ef-8134-6ea46edb5183               NaN   \n",
       "4  4d49b9f5-09a0-49de-84f7-0d3441c214f6               NaN   \n",
       "\n",
       "   days_to_lost_to_followup             disease_type         submitter_id  \\\n",
       "0                       NaN  Squamous Cell Neoplasms    GENIE-DFCI-011620   \n",
       "1                       NaN  Squamous Cell Neoplasms    GENIE-DFCI-050738   \n",
       "2                       NaN  Squamous Cell Neoplasms    GENIE-DFCI-004072   \n",
       "3                       NaN  Squamous Cell Neoplasms  GENIE-GRCC-a8pxs0u6   \n",
       "4                       NaN  Squamous Cell Neoplasms  GENIE-GRCC-2b4655c3   \n",
       "\n",
       "                    submitter_aliquot_ids  \\\n",
       "0       [GENIE-DFCI-011620-10763_aliquot]   \n",
       "1      [GENIE-DFCI-050738-234120_aliquot]   \n",
       "2         [GENIE-DFCI-004072-413_aliquot]   \n",
       "3  [GENIE-GRCC-a8pxs0u6-sample-a_aliquot]   \n",
       "4  [GENIE-GRCC-2b4655c3-sample-a_aliquot]   \n",
       "\n",
       "                              aliquot_ids  \\\n",
       "0  [3d4995b8-5b04-46f2-8d37-7e0b9f9b1b1a]   \n",
       "1  [57d18da1-d1b9-40b0-8ee6-0f94fd9f7575]   \n",
       "2  [95066691-03ea-422a-bb4c-ba09e9cbd7ab]   \n",
       "3  [b17a8d8a-395e-4d42-bfb0-7e829e1d4a8b]   \n",
       "4  [a0f16f51-94eb-4a6b-beb6-9159fa6acc4b]   \n",
       "\n",
       "                                           diagnoses  \\\n",
       "0  [{'irs_stage': None, 'iss_stage': None, 'ajcc_...   \n",
       "1  [{'irs_stage': None, 'iss_stage': None, 'ajcc_...   \n",
       "2  [{'irs_stage': None, 'iss_stage': None, 'ajcc_...   \n",
       "3  [{'irs_stage': None, 'iss_stage': None, 'ajcc_...   \n",
       "4  [{'irs_stage': None, 'iss_stage': None, 'ajcc_...   \n",
       "\n",
       "                            diagnosis_ids                  created_datetime  \\\n",
       "0  [a4f6276a-b3cc-45f9-9fb8-30edd56ad4ea]  2018-09-13T13:41:51.057497-05:00   \n",
       "1  [2e4427b7-e557-49c7-85ef-39a02c4a441c]  2019-06-03T12:43:36.681258-05:00   \n",
       "2  [85d63c6e-a6c1-47a5-92a9-216719073a4a]  2018-09-13T13:44:12.915115-05:00   \n",
       "3  [d0678ad6-4a4a-4ec8-9e5f-352c1347efc2]  2019-06-04T18:08:22.482657-05:00   \n",
       "4  [2707217b-5a4b-404b-97de-921526581d45]  2018-10-02T17:53:10.070290-05:00   \n",
       "\n",
       "   ... demographic.occupation_duration_years demographic.days_to_death  \\\n",
       "0  ...                                   NaN                       NaN   \n",
       "1  ...                                   NaN                       NaN   \n",
       "2  ...                                   NaN                       NaN   \n",
       "3  ...                                   NaN                       NaN   \n",
       "4  ...                                   NaN                       NaN   \n",
       "\n",
       "  demographic.state demographic.year_of_death slide_ids submitter_slide_ids  \\\n",
       "0          released                       NaN       NaN                 NaN   \n",
       "1          released                       NaN       NaN                 NaN   \n",
       "2          released                       NaN       NaN                 NaN   \n",
       "3          released                       NaN       NaN                 NaN   \n",
       "4          released                       NaN       NaN                 NaN   \n",
       "\n",
       "  analyte_ids submitter_analyte_ids portion_ids  submitter_portion_ids  \n",
       "0         NaN                   NaN         NaN                    NaN  \n",
       "1         NaN                   NaN         NaN                    NaN  \n",
       "2         NaN                   NaN         NaN                    NaN  \n",
       "3         NaN                   NaN         NaN                    NaN  \n",
       "4         NaN                   NaN         NaN                    NaN  \n",
       "\n",
       "[5 rows x 45 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gdc_entries.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e6f25d5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lost_to_followup</th>\n",
       "      <th>days_to_lost_to_followup</th>\n",
       "      <th>demographic.cause_of_death</th>\n",
       "      <th>demographic.age_at_index</th>\n",
       "      <th>demographic.days_to_birth</th>\n",
       "      <th>demographic.year_of_birth</th>\n",
       "      <th>demographic.cause_of_death_source</th>\n",
       "      <th>demographic.premature_at_birth</th>\n",
       "      <th>demographic.weeks_gestation_at_birth</th>\n",
       "      <th>demographic.age_is_obfuscated</th>\n",
       "      <th>demographic.occupation_duration_years</th>\n",
       "      <th>demographic.days_to_death</th>\n",
       "      <th>demographic.year_of_death</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>499.000000</td>\n",
       "      <td>201.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>77.000000</td>\n",
       "      <td>54.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12998.078156</td>\n",
       "      <td>-21889.791583</td>\n",
       "      <td>1948.029851</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>824.909091</td>\n",
       "      <td>2005.314815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11059.558005</td>\n",
       "      <td>3837.513498</td>\n",
       "      <td>11.702953</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>964.107048</td>\n",
       "      <td>5.012528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>-32871.000000</td>\n",
       "      <td>1916.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1996.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>-24471.000000</td>\n",
       "      <td>1940.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>330.000000</td>\n",
       "      <td>2001.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18627.000000</td>\n",
       "      <td>-22182.000000</td>\n",
       "      <td>1950.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>480.000000</td>\n",
       "      <td>2006.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22645.000000</td>\n",
       "      <td>-19470.000000</td>\n",
       "      <td>1956.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>941.000000</td>\n",
       "      <td>2009.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30681.000000</td>\n",
       "      <td>-7670.000000</td>\n",
       "      <td>1973.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6417.000000</td>\n",
       "      <td>2013.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       lost_to_followup  days_to_lost_to_followup  demographic.cause_of_death  \\\n",
       "count               0.0                       0.0                         0.0   \n",
       "mean                NaN                       NaN                         NaN   \n",
       "std                 NaN                       NaN                         NaN   \n",
       "min                 NaN                       NaN                         NaN   \n",
       "25%                 NaN                       NaN                         NaN   \n",
       "50%                 NaN                       NaN                         NaN   \n",
       "75%                 NaN                       NaN                         NaN   \n",
       "max                 NaN                       NaN                         NaN   \n",
       "\n",
       "       demographic.age_at_index  demographic.days_to_birth  \\\n",
       "count                499.000000                 499.000000   \n",
       "mean               12998.078156              -21889.791583   \n",
       "std                11059.558005                3837.513498   \n",
       "min                   35.000000              -32871.000000   \n",
       "25%                   62.500000              -24471.000000   \n",
       "50%                18627.000000              -22182.000000   \n",
       "75%                22645.000000              -19470.000000   \n",
       "max                30681.000000               -7670.000000   \n",
       "\n",
       "       demographic.year_of_birth  demographic.cause_of_death_source  \\\n",
       "count                 201.000000                                0.0   \n",
       "mean                 1948.029851                                NaN   \n",
       "std                    11.702953                                NaN   \n",
       "min                  1916.000000                                NaN   \n",
       "25%                  1940.000000                                NaN   \n",
       "50%                  1950.000000                                NaN   \n",
       "75%                  1956.000000                                NaN   \n",
       "max                  1973.000000                                NaN   \n",
       "\n",
       "       demographic.premature_at_birth  demographic.weeks_gestation_at_birth  \\\n",
       "count                             0.0                                   0.0   \n",
       "mean                              NaN                                   NaN   \n",
       "std                               NaN                                   NaN   \n",
       "min                               NaN                                   NaN   \n",
       "25%                               NaN                                   NaN   \n",
       "50%                               NaN                                   NaN   \n",
       "75%                               NaN                                   NaN   \n",
       "max                               NaN                                   NaN   \n",
       "\n",
       "       demographic.age_is_obfuscated  demographic.occupation_duration_years  \\\n",
       "count                            0.0                                    0.0   \n",
       "mean                             NaN                                    NaN   \n",
       "std                              NaN                                    NaN   \n",
       "min                              NaN                                    NaN   \n",
       "25%                              NaN                                    NaN   \n",
       "50%                              NaN                                    NaN   \n",
       "75%                              NaN                                    NaN   \n",
       "max                              NaN                                    NaN   \n",
       "\n",
       "       demographic.days_to_death  demographic.year_of_death  \n",
       "count                  77.000000                  54.000000  \n",
       "mean                  824.909091                2005.314815  \n",
       "std                   964.107048                   5.012528  \n",
       "min                     1.000000                1996.000000  \n",
       "25%                   330.000000                2001.000000  \n",
       "50%                   480.000000                2006.500000  \n",
       "75%                   941.000000                2009.750000  \n",
       "max                  6417.000000                2013.000000  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gdc_entries.describe()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}