Code/Day78-NLP_in_spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NLP Project using Spark\n",
    "\n",
    "For this code along we will build a spam filter! We'll use the various NLP tools we learned about as well as a new classifier, Naive Bayes.\n",
    "\n",
    "We'll use a classic dataset for this - UCI Repository SMS Spam Detection: https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('nlp').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = spark.read.csv(\"../datasets/SMSSpamCollection\",inferSchema=True,sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.withColumnRenamed('_c0','class').withColumnRenamed('_c1','text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "+-----+--------------------+\n|class|                text|\n+-----+--------------------+\n|  ham|Go until jurong p...|\n|  ham|Ok lar... Joking ...|\n| spam|Free entry in 2 a...|\n|  ham|U dun say so earl...|\n|  ham|Nah I don't think...|\n| spam|FreeMsg Hey there...|\n|  ham|Even my brother i...|\n|  ham|As per your reque...|\n| spam|WINNER!! As a val...|\n| spam|Had your mobile 1...|\n|  ham|I'm gonna be home...|\n| spam|SIX chances to wi...|\n| spam|URGENT! You have ...|\n|  ham|I've been searchi...|\n|  ham|I HAVE A DATE ON ...|\n| spam|XXXMobileMovieClu...|\n|  ham|Oh k...i'm watchi...|\n|  ham|Eh u remember how...|\n|  ham|Fine if thats th...|\n| spam|England v Macedon...|\n+-----+--------------------+\nonly showing top 20 rows\n\n"
    }
   ],
   "source": [
    "data.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Clean and Prepare the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "** Create a new length feature: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.withColumn('length',length(data['text']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "+-----+--------------------+------+\n|class|                text|length|\n+-----+--------------------+------+\n|  ham|Go until jurong p...|   111|\n|  ham|Ok lar... Joking ...|    29|\n| spam|Free entry in 2 a...|   155|\n|  ham|U dun say so earl...|    49|\n|  ham|Nah I don't think...|    61|\n| spam|FreeMsg Hey there...|   147|\n|  ham|Even my brother i...|    77|\n|  ham|As per your reque...|   160|\n| spam|WINNER!! As a val...|   157|\n| spam|Had your mobile 1...|   154|\n|  ham|I'm gonna be home...|   109|\n| spam|SIX chances to wi...|   136|\n| spam|URGENT! You have ...|   155|\n|  ham|I've been searchi...|   196|\n|  ham|I HAVE A DATE ON ...|    35|\n| spam|XXXMobileMovieClu...|   149|\n|  ham|Oh k...i'm watchi...|    26|\n|  ham|Eh u remember how...|    81|\n|  ham|Fine if thats th...|    56|\n| spam|England v Macedon...|   155|\n+-----+--------------------+------+\nonly showing top 20 rows\n\n"
    }
   ],
   "source": [
    "data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "+-----+-----------------+\n|class|      avg(length)|\n+-----+-----------------+\n|  ham|71.45431945307645|\n| spam|138.6706827309237|\n+-----+-----------------+\n\n"
    }
   ],
   "source": [
    "# Pretty Clear Difference\n",
    "data.groupby('class').mean().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Feature Transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Tokenizer,StopWordsRemover, CountVectorizer,IDF,StringIndexer\n",
    "\n",
    "tokenizer = Tokenizer(inputCol=\"text\", outputCol=\"token_text\")\n",
    "stopremove = StopWordsRemover(inputCol='token_text',outputCol='stop_tokens')\n",
    "count_vec = CountVectorizer(inputCol='stop_tokens',outputCol='c_vec')\n",
    "idf = IDF(inputCol=\"c_vec\", outputCol=\"tf_idf\")\n",
    "ham_spam_to_num = StringIndexer(inputCol='class',outputCol='label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.linalg import Vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_up = VectorAssembler(inputCols=['tf_idf','length'],outputCol='features')"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### The Model\n",
    "\n",
    "We'll use Naive Bayes, but feel free to play around with this choice!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import NaiveBayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use defaults\n",
    "nb = NaiveBayes()"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_prep_pipe = Pipeline(stages=[ham_spam_to_num,tokenizer,stopremove,count_vec,idf,clean_up])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaner = data_prep_pipe.fit(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data = cleaner.transform(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Training and Evaluation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data = clean_data.select(['label','features'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "+-----+--------------------+\n|label|            features|\n+-----+--------------------+\n|  0.0|(13424,[7,11,31,6...|\n|  0.0|(13424,[0,24,297,...|\n|  1.0|(13424,[2,13,19,3...|\n|  0.0|(13424,[0,70,80,1...|\n|  0.0|(13424,[36,134,31...|\n|  1.0|(13424,[10,60,139...|\n|  0.0|(13424,[10,53,103...|\n|  0.0|(13424,[125,184,4...|\n|  1.0|(13424,[1,47,118,...|\n|  1.0|(13424,[0,1,13,27...|\n|  0.0|(13424,[18,43,120...|\n|  1.0|(13424,[8,17,37,8...|\n|  1.0|(13424,[13,30,47,...|\n|  0.0|(13424,[39,96,217...|\n|  0.0|(13424,[552,1697,...|\n|  1.0|(13424,[30,109,11...|\n|  0.0|(13424,[82,214,47...|\n|  0.0|(13424,[0,2,49,13...|\n|  0.0|(13424,[0,74,105,...|\n|  1.0|(13424,[4,30,33,5...|\n+-----+--------------------+\nonly showing top 20 rows\n\n"
    }
   ],
   "source": [
    "clean_data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "(training,testing) = clean_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "spam_predictor = nb.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "root\n |-- class: string (nullable = true)\n |-- text: string (nullable = true)\n |-- length: integer (nullable = true)\n\n"
    }
   ],
   "source": [
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = spam_predictor.transform(testing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "+-----+--------------------+--------------------+--------------------+----------+\n|label|            features|       rawPrediction|         probability|prediction|\n+-----+--------------------+--------------------+--------------------+----------+\n|  0.0|(13424,[0,1,2,7,8...|[-807.85430102661...|[1.0,4.4092579857...|       0.0|\n|  0.0|(13424,[0,1,2,41,...|[-1077.5377991882...|[1.0,1.1632130699...|       0.0|\n|  0.0|(13424,[0,1,3,9,1...|[-570.94294546402...|[1.0,8.1052712228...|       0.0|\n|  0.0|(13424,[0,1,5,20,...|[-801.27699612116...|[1.0,9.4072746096...|       0.0|\n|  0.0|(13424,[0,1,11,32...|[-882.11701105115...|[1.0,5.3456886827...|       0.0|\n|  0.0|(13424,[0,1,14,31...|[-217.03030294557...|[1.0,5.0002531356...|       0.0|\n|  0.0|(13424,[0,1,17,19...|[-804.90457257807...|[1.0,3.4749002636...|       0.0|\n|  0.0|(13424,[0,1,20,27...|[-969.60401679996...|[1.0,7.4881874560...|       0.0|\n|  0.0|(13424,[0,1,23,63...|[-1317.5715795344...|[1.0,1.7454057728...|       0.0|\n|  0.0|(13424,[0,1,27,35...|[-1490.6108651732...|[0.99999864571112...|       0.0|\n|  0.0|(13424,[0,1,27,88...|[-1526.2357541458...|[3.03545555915430...|       1.0|\n|  0.0|(13424,[0,1,30,12...|[-595.26217108586...|[1.0,4.7561996455...|       0.0|\n|  0.0|(13424,[0,1,146,1...|[-254.64203858122...|[0.98857322140907...|       0.0|\n|  0.0|(13424,[0,2,4,5,1...|[-2491.8417525970...|[1.0,3.0675130635...|       0.0|\n|  0.0|(13424,[0,2,4,40,...|[-1582.8950432424...|[0.99999998451026...|       0.0|\n|  0.0|(13424,[0,2,4,128...|[-641.10959729575...|[1.0,8.2114513011...|       0.0|\n|  0.0|(13424,[0,2,5,8,4...|[-827.14005469804...|[1.0,3.6181096540...|       0.0|\n|  0.0|(13424,[0,2,7,11,...|[-1429.2435771662...|[1.0,3.6661634845...|       0.0|\n|  0.0|(13424,[0,2,7,60,...|[-1346.3199937597...|[1.0,7.2779144076...|       0.0|\n|  0.0|(13424,[0,2,7,114...|[-461.37475848093...|[1.0,9.7772311955...|       0.0|\n+-----+--------------------+--------------------+--------------------+----------+\nonly showing top 20 rows\n\n"
    }
   ],
   "source": [
    "test_results.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "Accuracy of model at predicting spam was: 0.915258928824168\n"
    }
   ],
   "source": [
    "acc_eval = MulticlassClassificationEvaluator()\n",
    "acc = acc_eval.evaluate(test_results)\n",
    "print(\"Accuracy of model at predicting spam was: {}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Not bad considering we're using straight math on text data! Try switching out the classification models! Or even try to come up with other engineered features!\n",
    "\n",
    "## Great Job!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7.5 64-bit",
   "language": "python",
   "name": "python37564bitedcdccb018dd4367a14589b436323787"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}