02-11-2020 Data-Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning\n",
    "This Includes removing metadata like \\n, expanding the English word contractions and correcting some spelling checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "import contractions\n",
    "from spellchecker import SpellChecker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preliminary cleaning\n",
    "- Changing the column names\n",
    "- Removing '\\n' in the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_title</th>\n",
       "      <th>review_body</th>\n",
       "      <th>company_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I ordered an item and had a little…</td>\n",
       "      <td>I ordered an item and had a li...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I ordered few items for 800 euro</td>\n",
       "      <td>I ordered few items for 800 eu...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Always the first choice for apparel shopping.</td>\n",
       "      <td>Always the first choice for ap...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Support is bad, return items and don't get mon...</td>\n",
       "      <td>Ordered items for 1200€. Got t...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Utterley useless in every resepct</td>\n",
       "      <td>2 items ordered, the first cam...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        review_title  \\\n",
       "0                I ordered an item and had a little…   \n",
       "1                   I ordered few items for 800 euro   \n",
       "2      Always the first choice for apparel shopping.   \n",
       "3  Support is bad, return items and don't get mon...   \n",
       "4                  Utterley useless in every resepct   \n",
       "\n",
       "                                         review_body company_source  \n",
       "0                  I ordered an item and had a li...        zalando  \n",
       "1                  I ordered few items for 800 eu...        zalando  \n",
       "2                  Always the first choice for ap...        zalando  \n",
       "3                  Ordered items for 1200€. Got t...        zalando  \n",
       "4                  2 items ordered, the first cam...        zalando  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('Data/trustpilot_reviews.csv').iloc[:,1:]\n",
    "data.columns = ['review_title','review_body','company_source']\n",
    "data = data.replace('(\\\\n)','',regex=True)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Cleaning\n",
    "- Expanding the english word contractions\n",
    "- Correcting some spellings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "spell = SpellChecker(case_sensitive=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def correct_spelling(sentence):\n",
    "    \"\"\"Expand words like don't and won't and then correct words like resepct to respect\"\"\"\n",
    "    sentence = contractions.fix(sentence)\n",
    "    sentence = ' '.join([spell.correction(word) for word in spell.split_words(sentence)])\n",
    "    sentence = sentence.capitalize()\n",
    "    return(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['review_title'] = data['review_title'].apply(correct_spelling)\n",
    "data['review_body'] = data['review_body'].apply(correct_spelling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_title</th>\n",
       "      <th>review_body</th>\n",
       "      <th>company_source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I ordered an item and had a little</td>\n",
       "      <td>I ordered an item and had a little payment mis...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I ordered few items for 800 euro</td>\n",
       "      <td>I ordered few items for 800 euro came only che...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Always the first choice for apparel shopping</td>\n",
       "      <td>Always the first choice for apparel shopping i...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Support is bad return items and do not get mon...</td>\n",
       "      <td>Ordered items for 1200 got them all but in the...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Utterly useless in every respect</td>\n",
       "      <td>2 items ordered the first came the next day dh...</td>\n",
       "      <td>zalando</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        review_title  \\\n",
       "0                 I ordered an item and had a little   \n",
       "1                   I ordered few items for 800 euro   \n",
       "2       Always the first choice for apparel shopping   \n",
       "3  Support is bad return items and do not get mon...   \n",
       "4                   Utterly useless in every respect   \n",
       "\n",
       "                                         review_body company_source  \n",
       "0  I ordered an item and had a little payment mis...        zalando  \n",
       "1  I ordered few items for 800 euro came only che...        zalando  \n",
       "2  Always the first choice for apparel shopping i...        zalando  \n",
       "3  Ordered items for 1200 got them all but in the...        zalando  \n",
       "4  2 items ordered the first came the next day dh...        zalando  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.to_csv('clean_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is then stored for other preprocessing for topic modelling. This actual cleaning data was used also directly for sentiment analysis in AWS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}