PHI augmentation-i2b2-2014 dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PHI augmentation : i2b2-2014 dataset processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The i2b2 2014 de-identification dataset can be accessed from https://portal.dbmi.hms.harvard.edu.\n",
    "\n",
    "The data processing mainly refers to the guidance from this link:\n",
    "https://github.com/juand-r/entity-recognition-datasets/tree/master/data/i2b2_2014"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.Converting training-PHI-Gold-Set1.tar, training-PHI-Gold-Set2.tar, testing-PHI-Gold-fixed.tar into BRAT format by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python xml_to_brat.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.Combine all the files, then randomly split the files into train, validation, test set with the proportion of 7:1:2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.Using Neuroner package (https://github.com/Franck-Dernoncourt/NeuroNER) to get train, validation, test file in BIO format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.As the processed file exists a few errors, we fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "123  B-STREET\n",
    "\n",
    "Alaska   B-STREET\n",
    "Street  I-STREET\n",
    "\n",
    "############### convert the above case into normal one #################\n",
    "\n",
    "123  B-STREET\n",
    "Alaska   I-STREET\n",
    "Street  I-STREET\n",
    "\"\"\"\n",
    "\n",
    "file_path = 'C:/Users/zhou/Desktop/valid_spacy.txt'\n",
    "file_preserve = []\n",
    "write_file = 'C:/Users/zhou/Desktop/valid_spacy_new.txt'\n",
    "\n",
    "with open(file_path, 'r', encoding='utf-8') as f_r:\n",
    "    content = f_r.readlines()\n",
    "    for j in range(0, len(content)):\n",
    "        line = content[j]\n",
    "        file_preserve.append(line)\n",
    "        if j >= 2 and len(content[j-1].split()) == 0:   # previous line is empty\n",
    "            previsou_label = content[j - 2].strip().split()[-1]\n",
    "            previsou_end_position = content[j - 2].strip().split()[-2]\n",
    "            label = content[j].strip().split()[-1]\n",
    "            start_position = content[j].strip().split()[-3]  # be cautious -3\n",
    "            if previsou_label == 'O' or label == 'O':\n",
    "                continue\n",
    "            else:\n",
    "                previsou_label_type = previsou_label.split('-')[1]\n",
    "                label_type = label.split('-')[1]\n",
    "\n",
    "            if previsou_label_type == label_type and label_type != 'DATE' and int(start_position) <= (int(previsou_end_position)+3):\n",
    "                file_preserve.pop()  # delete this line\n",
    "                file_preserve.pop()  # delete empty line\n",
    "                newly_adjusted_label = 'I-' + previsou_label_type\n",
    "                new_line = content[j].replace(label, newly_adjusted_label)\n",
    "                file_preserve.append(new_line)\n",
    "\n",
    "with open(write_file, 'w', encoding='utf-8') as f_ww:\n",
    "    for line in file_preserve:\n",
    "        f_ww.writelines(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If want to conduct PHI augmentation, run the step 5,6,7; If not, skip to step 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.Replace the label of low frequency PHI type, e.g., FAX, with 'O'. \n",
    "And merge four types of location labels into 'LOCATION'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "other_PHI = ['FAX', 'EMAIL', 'URL', 'DEVICE', 'HEALTHPLAN', 'BIOID', 'LOCATION_OTHER']\n",
    "location_phi = ['CITY', 'COUNTRY', 'STATE', 'STREET']\n",
    "\n",
    "reserve_content = []\n",
    "file_path = 'C:/Users/zhou/Desktop/train_spacy_new.txt'\n",
    "file_path_write = 'C:/Users/zhou/Desktop/processed_2014_train.txt'\n",
    "\n",
    "with open(file_path, 'r', encoding='utf-8') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for line in content:\n",
    "        if len(line.split()) == 0:  # preserve empty line\n",
    "            reserve_content.append(line)\n",
    "        else:\n",
    "            words = line.split()\n",
    "            label = words[-1]\n",
    "            if label == 'O':\n",
    "                reserve_content.append(line)\n",
    "            else:\n",
    "                label_type = label.split('-')[1]\n",
    "                if label_type in other_PHI:\n",
    "                    line = line.replace(label, 'O')  # replace the whole label\n",
    "                    reserve_content.append(line)\n",
    "                elif label_type in location_phi:\n",
    "                    line = line.replace(label_type, 'LOCATION')  # only replace the label_type\n",
    "                    reserve_content.append(line)\n",
    "                else:\n",
    "                    reserve_content.append(line)\n",
    "\n",
    "with open(file_path_write, 'w', encoding='utf-8') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.Conduct PHI augmentation, run the folowing code.\n",
    "Notice that we replace PHI entity in fine-grainity (in PHI type level)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "file_path_entity = 'C:/Users/zhou/Desktop/all_PHI_from_internet/'\n",
    "labels_in_2006 = ['PATIENT', 'DOCTOR', 'HOSPITAL', 'IDNUM', 'DATE', 'PHONE', 'AGE', 'ZIP',\n",
    "                  'MEDICALRECORD', 'ORGANIZATION', 'PROFESSION', 'USERNAME', 'LOCATION']\n",
    "\n",
    "label_content = [[] for i in range(len(labels_in_2006))]\n",
    "for k in range(len(labels_in_2006)):\n",
    "    file_path = file_path_entity + labels_in_2006[k] + '.txt'\n",
    "    with open(file_path, 'r', encoding='utf-8') as f_w:\n",
    "        content = f_w.readlines()\n",
    "        for j in content:\n",
    "            label_content[k].append(j.strip())\n",
    "\n",
    "\n",
    "def replace_PHI(label, line_number):\n",
    "    preserve_list = []\n",
    "    if label in labels_in_2006:\n",
    "        label_index = labels_in_2006.index(label)\n",
    "        words_list = label_content[label_index]\n",
    "\n",
    "        random.seed(global_seed)  # every epoch, it produces different dataset, also reimplementable.\n",
    "        random.shuffle(words_list)\n",
    "        words_num = len(words_list)\n",
    "\n",
    "        random.seed(line_number)  # line_number is defined for specific PHI instance\n",
    "        random_index = random.randint(0, words_num - 1)\n",
    "        random_w = words_list[random_index].strip()  # words_list is different at different epoch\n",
    "\n",
    "        words_num = random_w.split()\n",
    "        if len(words_num) == 1:\n",
    "            new_line = random_w+'   '+'B-'+label\n",
    "            preserve_list.append(new_line)\n",
    "        if len(words_num) > 1:\n",
    "            for i in range(len(words_num)):\n",
    "                if i == 0:\n",
    "                    new_line = words_num[i]+'   '+'B-'+label\n",
    "                    preserve_list.append(new_line)\n",
    "                if i > 0:\n",
    "                    new_line = words_num[i]+'   '+'I-'+label\n",
    "                    preserve_list.append(new_line)\n",
    "    return preserve_list\n",
    "\n",
    "\n",
    "file_path_2014 = 'C:/Users/zhou/Desktop/processed_2014_train.txt'\n",
    "global_seed = 2\n",
    "repalced_file_path = 'C:/Users/zhou/Desktop/new_replaced_2014_train_seed'+str(global_seed)+'.txt'\n",
    "\n",
    "\n",
    "reserve_content = []\n",
    "with open(file_path_2014, 'r', encoding='utf-8') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for row in range(0, len(content)-1):\n",
    "        if len(content[row]) == 1:  # cope with empty line\n",
    "            reserve_content.append('')\n",
    "        if len(content[row]) != 1 and len(content[row+1]) != 1:\n",
    "            all_token_Next = content[row + 1].split()\n",
    "            all_token = content[row].split()\n",
    "            this_line_label = all_token[-1]\n",
    "            next_line_label = all_token_Next[-1]\n",
    "            if this_line_label.startswith('O'):\n",
    "                new_line = all_token[0]+'   '+all_token[-1]\n",
    "                reserve_content.append(new_line)\n",
    "                continue\n",
    "            if this_line_label.startswith('B-') and next_line_label.startswith('I-'):\n",
    "                continue\n",
    "            if this_line_label.startswith('B-') and not next_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                new_words = replace_PHI(label, row)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "            if this_line_label.startswith('I-') and next_line_label.startswith('I-'):\n",
    "                continue\n",
    "            if this_line_label.startswith('I-') and not next_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                # print(label)\n",
    "                new_words = replace_PHI(label, row)\n",
    "                # print(new_words)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "\n",
    "        if len(content[row]) != 1 and len(content[row+1]) == 1:\n",
    "            all_token = content[row].split()\n",
    "            this_line_label = all_token[-1]\n",
    "            if this_line_label.startswith('B-') or this_line_label.startswith('I-'):\n",
    "                label = this_line_label.split('-')[-1]\n",
    "                new_words = replace_PHI(label, row)\n",
    "                for k in new_words:\n",
    "                    reserve_content.append(k)\n",
    "            if this_line_label.startswith('O'):\n",
    "                new_line = all_token[0]+'   '+all_token[-1]\n",
    "                reserve_content.append(new_line)\n",
    "                continue\n",
    "\n",
    "\n",
    "with open(repalced_file_path, 'w', encoding='utf-8') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.writelines(line+'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.Combine PHI into five PHI categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_contact = 'PHONE'   # delete FAX, EMAIL,URL\n",
    "label_date = 'DATE'\n",
    "label_id = ['IDNUM', 'MEDICALRECORD']  # delete DEVICE, HEALTHPLAN, BIOID\n",
    "label_location = ['HOSPITAL', 'CITY', 'STREET', 'ZIP', 'STATE', 'COUNTRY', 'ORGANIZATION', 'LOCATION']  \n",
    "label_name = ['PATIENT', 'DOCTOR', 'USERNAME']\n",
    "other_PHI = ['AGE','FAX', 'EMAIL', 'URL', 'DEVICE', 'HEALTHPLAN', 'BIOID', 'LOCATION_OTHER', 'PROFESSION']\n",
    "\n",
    "reserve_content = []\n",
    "file_path = 'C:/Users/zhou/Desktop/valid_spacy_new.txt'\n",
    "file_path_write = 'C:/Users/zhou/Desktop/5_category_2014_valid.txt'\n",
    "\n",
    "with open(file_path, 'r', encoding='utf-8') as f_wr:\n",
    "    content = f_wr.readlines()\n",
    "    for line in content:\n",
    "        if len(line.split()) == 0 :  # preserve empty line\n",
    "            reserve_content.append(line)\n",
    "        else:\n",
    "            words = line.split()\n",
    "            label = words[-1]\n",
    "            if label == 'O':\n",
    "                reserve_content.append(line)\n",
    "            else:\n",
    "                label_type = label.split('-')[1]\n",
    "                if label_type in label_location:\n",
    "                    new_label = label.replace(label_type, 'LOCATION')  # B-HOSPITAL --> B-LOCATION\n",
    "                    new_line = line.replace(label, new_label)  # XXX B-LOCATION\n",
    "                    reserve_content.append(new_line)\n",
    "                elif label_type == label_contact:\n",
    "                    new_label = label.replace(label_type,'CONTACT')\n",
    "                    new_line = line.replace(label, new_label)\n",
    "                    reserve_content.append(new_line)\n",
    "                elif label_type == label_date:\n",
    "                    reserve_content.append(line)\n",
    "                elif label_type in label_name:\n",
    "                    new_label = label.replace(label_type, 'NAME')\n",
    "                    new_line = line.replace(label, new_label)\n",
    "                    reserve_content.append(new_line)\n",
    "                elif label_type in label_id:\n",
    "                    line = line.replace(label_type, 'ID')\n",
    "                    reserve_content.append(line)\n",
    "                else:\n",
    "                    assert label_type in other_PHI\n",
    "                    line = line.replace(label, 'O')  # replace the whole label\n",
    "                    reserve_content.append(line)\n",
    "\n",
    "with open(file_path_write, 'w', encoding='utf-8') as f_w:\n",
    "    for line in reserve_content:\n",
    "        f_w.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.If want to conduct context augmentation, turn to relevant step."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}