Spark_Machine_Learning_Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark MLib utilizes the Spark DataFrame syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the main quirks of using MLib is that need to format the data so that eventually it just has one or two columns:\n",
    "- Features, Labels (Supervised Learning)\n",
    "- Features (Unsupervised Learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Myself Information: \n",
    "- Spark documentations are really good formatted: [Spark](spark.apache.org)\n",
    "- Books recommendations: Introduction to Statistical Learning By Gareth James"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some common evaluation metrics for Regression methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Mean Absolute Error\n",
    "* Mean Squared Error\n",
    "* Root Mean Square Error\n",
    "* R Square Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MSE will make larger errors more notable than MAE, which makes MSE more popular;\n",
    "\n",
    "While RMSE is the most popular method for regression;\n",
    "\n",
    "R Squared Values is actually not quite an error metric, more of a statistical measure of your regression model. R2 is a measure of how much variance your model accounts for, its value is between 0-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Least Squares Method:\n",
    "A method to measure the Linear Regression.\n",
    "By minimizing the sum of squares of the residuals.\n",
    "The residuals fir an observation is the difference between the observation (the y-value) and the fitted line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. An example from Spark official documentation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName(\"Jerry_LR_sample\").getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "training = spark.read.format('libsvm').load('datasets/sample_linear_regression_data.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below shows the format we will be using in the future when doing machine learning in Spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+--------------------+\n",
      "|             label|            features|\n",
      "+------------------+--------------------+\n",
      "|-9.490009878824548|(10,[0,1,2,3,4,5,...|\n",
      "|0.2577820163584905|(10,[0,1,2,3,4,5,...|\n",
      "+------------------+--------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Sample data is already formatted\n",
    "training.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialisation the Linear Regression\n",
    "# Usually the default feature/label columns are names as features/label\n",
    "lr = LinearRegression(featuresCol='features',labelCol='label',predictionCol='prediction')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we need to apply our training dataset\n",
    "lr_model = lr.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DenseVector([0.0073, 0.8314, -0.8095, 2.4412, 0.5192, 1.1535, -0.2989, -0.5129, -0.6197, 0.6956])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14228558260358093"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_summary = lr_model.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.027839179518600154"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_summary.r2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.16309157133015"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_summary.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split data into training & testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data = spark.read.format('libsvm').load('datasets/sample_linear_regression_data.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The number indicates how you gonna split the dataset\n",
    "split_object = all_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[DataFrame[label: double, features: vector],\n",
       " DataFrame[label: double, features: vector]]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split_object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Usually we do\n",
    "training_data, test_data = all_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+--------------------+\n",
      "|              label|            features|\n",
      "+-------------------+--------------------+\n",
      "|-28.571478869743427|(10,[0,1,2,3,4,5,...|\n",
      "+-------------------+--------------------+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "training_data.show(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|              label|\n",
      "+-------+-------------------+\n",
      "|  count|                501|\n",
      "|   mean|0.25688882219498976|\n",
      "| stddev| 10.317884030544564|\n",
      "|    min|-28.571478869743427|\n",
      "|    max|  27.78383192005107|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "training.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|              label|\n",
      "+-------+-------------------+\n",
      "|  count|                149|\n",
      "|   mean|-0.8253147832832288|\n",
      "| stddev|  9.635891688501527|\n",
      "|    min|-28.046018037776633|\n",
      "|    max| 20.874246167942125|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_model = lr.fit(training_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using our test data to see how the model has been trained\n",
    "test_results = split_model.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "|-28.370129994491126|\n",
      "|-28.516278118222427|\n",
      "|-21.533356037904316|\n",
      "|-19.830850537094786|\n",
      "| -18.16071966175852|\n",
      "|-19.281523587878905|\n",
      "|-18.641883457094423|\n",
      "|  -19.2618007333354|\n",
      "|-17.125444851315443|\n",
      "|-16.961068020758777|\n",
      "|-14.780148269426743|\n",
      "|-16.601543885661577|\n",
      "|-14.872080695071702|\n",
      "|-14.996203091758307|\n",
      "|-17.945562205720012|\n",
      "|-15.287215336421413|\n",
      "|-12.920129064312961|\n",
      "|-16.714585704385076|\n",
      "| -13.54941866426533|\n",
      "| -9.944510707009712|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_results.residuals.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.80614462530596"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "unlabeled_data= test_data.select('features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|(10,[0,1,2,3,4,5,...|\n",
      "+--------------------+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabeled_data.show(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = split_model.transform(unlabeled_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|            features|          prediction|\n",
      "+--------------------+--------------------+\n",
      "|(10,[0,1,2,3,4,5,...|   0.324111956714493|\n",
      "|(10,[0,1,2,3,4,5,...|  1.7107946897393558|\n",
      "|(10,[0,1,2,3,4,5,...|  1.8660374225325946|\n",
      "|(10,[0,1,2,3,4,5,...|   1.555636971090154|\n",
      "|(10,[0,1,2,3,4,5,...|  0.3570934730940021|\n",
      "|(10,[0,1,2,3,4,5,...|  2.2161239620028907|\n",
      "|(10,[0,1,2,3,4,5,...|   1.949676435783316|\n",
      "|(10,[0,1,2,3,4,5,...|  3.0003704578826667|\n",
      "|(10,[0,1,2,3,4,5,...|  0.9740955000383327|\n",
      "|(10,[0,1,2,3,4,5,...|  0.8754089797372868|\n",
      "|(10,[0,1,2,3,4,5,...| -1.1713642963678303|\n",
      "|(10,[0,1,2,3,4,5,...|  0.7395345580910166|\n",
      "|(10,[0,1,2,3,4,5,...| -0.5653040983595159|\n",
      "|(10,[0,1,2,3,4,5,...| -0.3796546315539897|\n",
      "|(10,[0,1,2,3,4,5,...|   2.586017325887334|\n",
      "|(10,[0,1,2,3,4,5,...| 0.46506242667022407|\n",
      "|(10,[0,1,2,3,4,5,...|-0.11979899979165398|\n",
      "|(10,[0,1,2,3,4,5,...|  3.7923626010146565|\n",
      "|(10,[0,1,2,3,4,5,...|  1.0701384528138322|\n",
      "|(10,[0,1,2,3,4,5,...|  -2.523145674023148|\n",
      "+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Predicted values based on the features values\n",
    "predictions.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from pyspark.sql import SparkSession\n",
    "# spark = SparkSession.builder.appName(\"Jerry_LR_sample\").getOrCreate()\n",
    "# from pyspark.ml.regression import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = spark.read.csv(\"datasets/Ecommerce_Customers.csv\",inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------+------------------+-----------------+-----------------+--------------------+-------------------+\n",
      "|               Email|             Address|Avatar|Avg Session Length|      Time on App|  Time on Website|Length of Membership|Yearly Amount Spent|\n",
      "+--------------------+--------------------+------+------------------+-----------------+-----------------+--------------------+-------------------+\n",
      "|mstephenson@ferna...|835 Frank TunnelW...|Violet| 34.49726772511229|12.65565114916675|39.57766801952616|  4.0826206329529615|  587.9510539684005|\n",
      "+--------------------+--------------------+------+------------------+-----------------+-----------------+--------------------+-------------------+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.show(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.printSchema()\n",
    "# In this case we are going to predict the `yearly amount spent`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mstephenson@fernandez.com\n",
      "835 Frank TunnelWrightmouth, MI 82180-9605\n",
      "Violet\n",
      "34.49726772511229\n",
      "12.65565114916675\n",
      "39.57766801952616\n",
      "4.0826206329529615\n",
      "587.9510539684005\n"
     ]
    }
   ],
   "source": [
    "for i in data.head(1)[0]:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the dataframe to be used for MLib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Email',\n",
       " 'Address',\n",
       " 'Avatar',\n",
       " 'Avg Session Length',\n",
       " 'Time on App',\n",
       " 'Time on Website',\n",
       " 'Length of Membership',\n",
       " 'Yearly Amount Spent']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Init signature: VectorAssembler(inputCols=None, outputCol=None, handleInvalid='error')\n",
    "Docstring: A feature transformer that merges multiple columns into a vector column.\n",
    "'''\n",
    "# The idea here is to use this VectorAssembler function to take the columns you want\n",
    "# and merge them into a single Vector column 'features'\n",
    "assembler_col = VectorAssembler(inputCols=['Avg Session Length',\n",
    "                                           'Time on App',\n",
    "                                           'Time on Website',\n",
    "                                           'Length of Membership'],\n",
    "                                outputCol='features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now we want to transform our data\n",
    "output = assembler_col.transform(data)\n",
    "output.printSchema()\n",
    "# We can see there is a new column called 'features'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Email='mstephenson@fernandez.com', Address='835 Frank TunnelWrightmouth, MI 82180-9605', Avatar='Violet', Avg Session Length=34.49726772511229, Time on App=12.65565114916675, Time on Website=39.57766801952616, Length of Membership=4.0826206329529615, Yearly Amount Spent=587.9510539684005, features=DenseVector([34.4973, 12.6557, 39.5777, 4.0826]))]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_data = output.select('features','Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------------------+\n",
      "|            features|Yearly Amount Spent|\n",
      "+--------------------+-------------------+\n",
      "|[34.4972677251122...|  587.9510539684005|\n",
      "|[31.9262720263601...|  392.2049334443264|\n",
      "+--------------------+-------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "final_data.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train Test Split\n",
    "train_data, test_data = final_data.randomSplit([0.8,0.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                414|\n",
      "|   mean|    500.28523404033|\n",
      "| stddev|  79.19305190240036|\n",
      "|    min| 256.67058229005585|\n",
      "|    max|  765.5184619388373|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                 86|\n",
      "|   mean| 494.63874693719566|\n",
      "| stddev|  80.19904202231011|\n",
      "|    min|   266.086340948469|\n",
      "|    max|  725.5848140556806|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LinearRegression(featuresCol='features',labelCol='Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_model = lr.fit(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results = lr_model.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|          residuals|\n",
      "+-------------------+\n",
      "|-10.602109731667156|\n",
      "| -16.16771778351324|\n",
      "|  7.882524938824417|\n",
      "+-------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# residuals are the difference between the predicted values and the actual label from the test_data\n",
    "test_results.residuals.show(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9824941960796217"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.r2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.549222387277165"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.rootMeanSquaredError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to apply the model on the data only has 'features' column (no label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "unlabel_data = test_data.select('features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|[30.3931845423455...|\n",
      "|[30.8162006488763...|\n",
      "+--------------------+\n",
      "only showing top 2 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "unlabel_data.show(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------------------+\n",
      "|            features|        prediction|\n",
      "+--------------------+------------------+\n",
      "|[30.3931845423455...|330.53097953486076|\n",
      "|[30.8162006488763...|282.25405873198224|\n",
      "|[30.9716756438877...| 486.7560848180683|\n",
      "|[31.0613251567161...|492.61345638953935|\n",
      "|[31.1695067987115...| 416.3089240051486|\n",
      "|[31.2681042107507...| 426.4780116017662|\n",
      "|[31.3091926408918...|428.89299948792177|\n",
      "|[31.3123495994443...|443.77737837200857|\n",
      "|[31.3584771924370...| 490.2313154390522|\n",
      "|[31.3662121671876...|425.95264574533485|\n",
      "|[31.4252268808548...| 533.7239067652235|\n",
      "|[31.4459724827577...| 481.0956541511846|\n",
      "|[31.8124825597242...| 395.7856615582391|\n",
      "|[31.9262720263601...| 379.7647316622447|\n",
      "|[31.9453957483445...|  661.664166239279|\n",
      "|[31.9673209478824...| 449.4154678025375|\n",
      "|[32.0180740106320...|339.86516196499224|\n",
      "|[32.0542618511847...| 556.2535131672357|\n",
      "|[32.0609143984100...|  610.100716858474|\n",
      "|[32.1223647957977...| 531.6059588728715|\n",
      "+--------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions = lr_model.transform(unlabel_data)\n",
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}