Pytorch/Recommender System (Collaborative filtering).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaborative Filtering: Matrix Factorization Recommender System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Netflix Recommender Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basics of The Recommender System\n",
    "> The reason why consumers need reliable recommendations is straightforward: Given the availability of almost unlimited number of choices (e.g. different movie genres of various quality), and the natural tendency of human is to maximize his/her utility, the user needs guidance to the next best item that accommodates his/her needs or preferences. Specifically, Netflix has become so popular because of its capability to suggest the user a next movie (from thousands of other options) that matches to the best of user's taste."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of the Systems\n",
    "- There are many ways and complex algorithms used to build a recommender system. The following are fundamental approaches. While reading, the reader should think which one may be the most effective method when it comes to a movie recommendation.\n",
    "\n",
    "- >The Most Popular Item: It is the simplest strategy and requires no coding skills. It works based on the assumption that the most popular item attracts most consumers or most users. For example, any consumer shopping on Amazon would see the most frequently bought items. Conversely, Netflix would recommend every user the most popular movie in its list.\n",
    "\n",
    "- >Association & Market Based Model: The system makes recommendations based on the items in the consumer's basket. For instance, if the system detected that the buyer is purchasing ground coffee it would also suggest her to buy filters as well (observed association coffee - filters).\n",
    "\n",
    "- >Content Filtering: Uses metadata to determine the user's taste. For example, the system recommends the user movies based on their preferences of genres, actors, themes, etc. Such a system matches the user and the item based on similarity. For example, if the user watched and liked Terminator and Predator (both action movies with Arnold Schwarzenegger in the main role), it would probably recommend them to watch Commando.\n",
    "\n",
    "- >User-based (UBCF): It is a predecessor of Item-based CF. UBCF makes recommendations based on the user's characteristics that are similar to other users in the system. For example, if the end-user positively rates a movie, the algorithm finds other users who have previously rated the movie too, i.e. these users are similar to one another. In the next step, the system recommends the user an unseen movie but highly rated by other - referenced - users. See Figure 1.\n",
    "\n",
    "- >Item-based (IBCF): IBCF was originally developed by Amazon and is currently adopted by most online corporations (e.g. Netflix, YouTube, etc.).\n",
    "\n",
    "- >Hybrid Models: As the name suggests, the Hybrid Models combine two or more recommendation strategies. For instance, a Hybrid Content-Collaborative System can recommend the user a movie based on their gender but still focuses on the movie features the user exhibits to prefer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Netflix's recommendation engine is based on the assumption that similar users like and dislike similar items; i.e. Collaborative Filtering is the key to Netflix's success"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Issues with UBCF\n",
    "Even though CF is powerful, there are few challenges. Like other researchers, Sarwar et al. (2001) state that scalability and sparsity are the primary issues.\n",
    "\n",
    "Scalability: In a system where there are many new users, i.e. users with low records of ratings, it is computationally expensive to train the model. A typical example would be a web-based application with a recommender system incorporating millions of users and items. Such problems arise with Nearest Neighbour algorithms in UBCF where such algorithms require computations that grow simultaneously with the increasing numbers of users and items.\n",
    "\n",
    "Sparsity arises in a system when even the most active users have rated or purchased only a marginal number of available items. For example, these users may have experience with even less than 1% of available items.\n",
    "\n",
    "If a recommender system experiences either one or both of the described issues, the algorithm's performance decreases; i.e. the system does not recommend the user relevant items (Sarwar et al.,2001). Consequently, the user does not trust the recommendations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Load Packages "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import math\n",
    "import random\n",
    "\n",
    "import sklearn\n",
    "from nltk.corpus import stopwords\n",
    "from scipy.sparse import csr_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from scipy.sparse.linalg import svds\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "np.random.seed(42)\n",
    "seed = 42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns of movies_df: Index(['Unnamed: 0', 'machine_uuid', 'Email', 'Game', 'activity.play_duration',\n",
      "       'Platform', 'State', 'geoip.city_name', 'activity.display_firmware',\n",
      "       'geoip.timezone', 'date', 'Time', 'year', 'month', 'hour', 'Weekdays',\n",
      "       'timeoftheday'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "path = 'C:\\\\Users\\\\bokhy\\\\Desktop\\\\Python\\\\Python-Projects'\n",
    "df = pd.read_csv(os.path.join(path, 'data_final.csv'), engine = 'python')\n",
    "print('Columns of movies_df: {0}'.format(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Pre-Processing / Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- All selected users have played at least 5 titles in the data set (remove some of the noise from outliers)\n",
    "- However, the following code filters out the movies and users based on an arbitrary threshold and creates a new data frame ratings_flrd_df. Moreover, the chunk also prints the value of deleted movies with new and old dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### give ratings 1-10 based on playtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['activity.play_duration'] = df['activity.play_duration']*3600\n",
    "df['activity.play_duration'] = df['activity.play_duration'].apply(pd.to_numeric, errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABKIAAAJRCAYAAAB2sh8HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xuw7WdZH/DvwzmEm0ICBBqS1AR7KkRGA56GVDqOBU0CtAYtdqJFThEnjgVHWjs16EzB20zoiFQsxokSCV4IKYqkgsYYUTtTgZxIgFzEHCGSQ2ISDDfLFE18+sf6HVgc9t5nn8t6917rfD4za9Zaz3rX2u8z7177l/PN71LdHQAAAABYtIds9QQAAAAAOD4IogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhFhZEVdXDq+p9VfWBqrqlqn58qp9ZVe+tqtur6q1VdcJUf9j0fN/0+hlzn/XKqf7hqjp/rn7BVNtXVZcsqhcAAAAAjt4i94j6fJJnd/fXJzk7yQVVdW6S1yR5XXfvSvLJJC+dxr80ySe7+58ked00LlV1VpKLknxtkguS/EJV7aiqHUnekOS5Sc5K8l3TWAAAAAC2oYUFUT3zt9PTh063TvLsJG+b6lcmecH0+MLpeabXn1NVNdWv6u7Pd/dHk+xLcs5029fdH+nuv0ty1TQWAAAAgG1ooeeImvZcuinJvUmuS/KXST7V3Q9MQ/YnOXV6fGqSO5Nkev3TSR43Xz/oPevVAQAAANiGdi7yw7v7wSRnV9WJSd6e5KlrDZvua53X1quvFaL1GrVU1cVJLk6SRz3qUd/wlKc85RAzBwAAAGCzbrzxxk9098mHGrfQIOqA7v5UVf1RknOTnFhVO6e9nk5Lctc0bH+S05Psr6qdSR6T5P65+gHz71mvfvDPvzzJ5Umye/fu3rt377FoCwAAAIAkVfVXmxm3yKvmnTztCZWqekSSb0lyW5J3J3nhNGxPkndMj6+Znmd6/Q+7u6f6RdNV9c5MsivJ+5LckGTXdBW+EzI7ofk1i+oHAAAAgKOzyD2iTkly5XR1u4ckubq7f6eqbk1yVVX9VJL3J3njNP6NSX61qvZltifURUnS3bdU1dVJbk3yQJKXTYf8papenuTaJDuSXNHdtyywHwAAAACOQs12Ojp+ODQPAAAA4Niqqhu7e/ehxi30qnkAAAAAcIAgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEWFkRV1elV9e6quq2qbqmqH5rqr66qj1fVTdPteXPveWVV7auqD1fV+XP1C6bavqq6ZK5+ZlW9t6pur6q3VtUJi+oHAAAAgKOzyD2iHkjyw9391CTnJnlZVZ01vfa67j57ur0rSabXLkrytUkuSPILVbWjqnYkeUOS5yY5K8l3zX3Oa6bP2pXkk0leusB+AAAAADgKCwuiuvvu7v6z6fFnk9yW5NQN3nJhkqu6+/Pd/dEk+5KcM932dfdHuvvvklyV5MKqqiTPTvK26f1XJnnBYroBAAAA4GgNOUdUVZ2R5OlJ3juVXl5VH6yqK6rqpKl2apI75962f6qtV39ckk919wMH1QEAAADYhhYeRFXVVyT5zSSv6O7PJLksyVcnOTvJ3Ulee2DoGm/vI6ivNYeLq2pvVe297777DrMDAAAAAI6FhQZRVfXQzEKoX+/u30qS7r6nux/s7n9I8kuZHXqXzPZoOn3u7acluWuD+ieSnFhVOw+qf5nuvry7d3f37pNPPvnYNAcAAADAYVnkVfMqyRuT3NbdPztXP2Vu2LcnuXl6fE2Si6rqYVV1ZpJdSd6X5IYku6Yr5J2Q2QnNr+nuTvLuJC+c3r8nyTsW1Q8AAAAAR2fnoYccsWcl+Z4kH6qqm6baj2Z21buzMzuM7o4k358k3X1LVV2d5NbMrrj3su5+MEmq6uVJrk2yI8kV3X3L9Hk/kuSqqvqpJO/PLPgCAAAAYBuq2Y5Fx4/du3f33r17t3oaAAAAACujqm7s7t2HGjfkqnkAAAAAIIgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAyxsCCqqk6vqndX1W1VdUtV/dBUf2xVXVdVt0/3J031qqrXV9W+qvpgVT1j7rP2TONvr6o9c/VvqKoPTe95fVXVovoBAAAA4Ogsco+oB5L8cHc/Ncm5SV5WVWcluSTJ9d29K8n10/MkeW6SXdPt4iSXJbPgKsmrkjwzyTlJXnUgvJrGXDz3vgsW2A8AAAAAR2FhQVR3393dfzY9/myS25KcmuTCJFdOw65M8oLp8YVJ3twz70lyYlWdkuT8JNd19/3d/ckk1yW5YHrt0d39p93dSd4891kAAAAAbDNDzhFVVWckeXqS9yZ5YnffnczCqiRPmIadmuTOubftn2ob1fevUQcAAABgG1p4EFVVX5HkN5O8ors/s9HQNWp9BPW15nBxVe2tqr333XffoaYMAAAAwAIsNIiqqodmFkL9enf/1lS+ZzqsLtP9vVN9f5LT595+WpK7DlE/bY36l+nuy7t7d3fvPvnkk4+uKQAAAACOyCKvmldJ3pjktu7+2bmXrkly4Mp3e5K8Y67+4unqeecm+fR06N61Sc6rqpOmk5Sfl+Ta6bXPVtW508968dxnAQAAALDN7FzgZz8ryfck+VBV3TTVfjTJpUmurqqXJvlYku+cXntXkucl2Zfkc0lekiTdfX9V/WSSG6ZxP9Hd90+PfyDJm5I8IsnvTjcAAAAAtqGaXXDu+LF79+7eu3fvVk8DAAAAYGVU1Y3dvftQ44ZcNQ8AAAAABFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADLGpIKqqrt9MDQAAAADWs3OjF6vq4UkemeTxVXVSkppeenSSJy14bgAAAACskA2DqCTfn+QVmYVON+aLQdRnkrxhgfMCAAAAYMVsGER1988l+bmq+sHu/vlBcwIAAABgBR1qj6gkSXf/fFV9Y5Iz5t/T3W9e0LwAAAAAWDGbCqKq6leTfHWSm5I8OJU7iSAKAAAAgE3ZVBCVZHeSs7q7FzkZAAAAAFbXQzY57uYk/2iREwEAAABgtW02iHp8klur6tqquubAbaM3VNUVVXVvVd08V3t1VX28qm6abs+be+2VVbWvqj5cVefP1S+Yavuq6pK5+plV9d6qur2q3lpVJ2y+bQAAAABG2+yhea8+gs9+U5L/kS8/j9Truvtn5gtVdVaSi5J8bZInJfmDqvqn08tvSPKtSfYnuaGqrunuW5O8Zvqsq6rqF5O8NMllRzBPAAAAAAbY7FXz/vhwP7i7/6Sqztjk8AuTXNXdn0/y0aral+Sc6bV93f2RJKmqq5JcWFW3JXl2ku+exlyZWVgmiAIAAADYpjZ1aF5VfbaqPjPd/l9VPVhVnznCn/nyqvrgdOjeSVPt1CR3zo3ZP9XWqz8uyae6+4GD6gAAAABsU5sKorr7K7v70dPt4Un+TWaH3R2uy5J8dZKzk9yd5LVTvdb6sUdQX1NVXVxVe6tq73333Xd4MwYAAADgmNjsycq/RHf/dmaHxh3u++7p7ge7+x+S/FK+ePjd/iSnzw09LcldG9Q/keTEqtp5UH29n3t5d+/u7t0nn3zy4U4bAAAAgGNgU+eIqqrvmHv6kCS7s8EeSBt8zindfff09NuTHLii3jVJfqOqfjazk5XvSvK+zPZ82lVVZyb5eGYnNP/u7u6qeneSFya5KsmeJO843PkAAAAAMM5mr5r3r+ceP5DkjsxOML6uqnpLkm9O8viq2p/kVUm+uarOzizEuiPJ9ydJd99SVVcnuXX6/Jd194PT57w8ybVJdiS5ortvmX7EjyS5qqp+Ksn7k7xxk70AAAAAsAWq+7B3bFpqu3fv7r179271NAAAAABWRlXd2N27DzVus1fNO62q3l5V91bVPVX1m1V12tFPEwAAAIDjxWZPVv4rmZ3H6UlJTk3yv6YaAAAAAGzKZoOok7v7V7r7gen2piQuPwcAAADApm02iPpEVb2oqnZMtxcl+ZtFTgwAAACA1bLZIOp7k/zbJH+d5O4kL0zykkVNCgAAAIDVs3OT434yyZ7u/mSSVNVjk/xMZgEVAAAAABzSZveI+roDIVSSdPf9SZ6+mCkBAAAAsIo2G0Q9pKpOOvBk2iNqs3tTAQAAAMCmw6TXJvk/VfW2JJ3Z+aJ+emGzAgAAAGDlbCqI6u43V9XeJM9OUkm+o7tvXejMAAAAAFgpmz68bgqehE8AAAAAHJHNniMKAAAAAI6KIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMsXOrJzDahz7+6ZxxyTs3Pf6OS5+/wNkAAAAAHD/sEQUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwxMKCqKq6oqruraqb52qPrarrqur26f6kqV5V9fqq2ldVH6yqZ8y9Z880/vaq2jNX/4aq+tD0ntdXVS2qFwAAAACO3iL3iHpTkgsOql2S5Pru3pXk+ul5kjw3ya7pdnGSy5JZcJXkVUmemeScJK86EF5NYy6ee9/BPwsAAACAbWRhQVR3/0mS+w8qX5jkyunxlUleMFd/c8+8J8mJVXVKkvOTXNfd93f3J5Ncl+SC6bVHd/efdncnefPcZwEAAACwDY0+R9QTu/vuJJnunzDVT01y59y4/VNto/r+NeoAAAAAbFPb5WTla53fqY+gvvaHV11cVXurau+Dn/v0EU4RAAAAgKMxOoi6ZzqsLtP9vVN9f5LT58adluSuQ9RPW6O+pu6+vLt3d/fuHY98zFE3AQAAAMDhGx1EXZPkwJXv9iR5x1z9xdPV885N8unp0L1rk5xXVSdNJyk/L8m102ufrapzp6vlvXjuswAAAADYhnYu6oOr6i1JvjnJ46tqf2ZXv7s0ydVV9dIkH0vyndPwdyV5XpJ9ST6X5CVJ0t33V9VPJrlhGvcT3X3gBOg/kNmV+R6R5HenGwAAAADb1MKCqO7+rnVees4aYzvJy9b5nCuSXLFGfW+Spx3NHAEAAAAYZ7ucrBwAAACAFSeIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABD7NzqCWx3Z1zyzsMaf8elz1/QTAAAAACWmz2iAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhtiSIqqo7qupDVXVTVe2dao+tquuq6vbp/qSpXlX1+qraV1UfrKpnzH3Onmn87VW1Zyt6AQAAAGBztnKPqH/Z3Wd39+7p+SVJru/uXUmun54nyXOT7JpuFye5LJkFV0leleSZSc5J8qoD4RUAAAAA2892OjTvwiRXTo+vTPKCufqbe+Y9SU6sqlOSnJ/kuu6+v7s/meS6JBeMnjQAAAAAm7NVQVQn+f2qurGqLp5qT+zuu5Nkun/CVD81yZ1z790/1darAwAAALAN7dyin/us7r6rqp6Q5Lqq+vMNxtYatd6g/uUfMAu7Lk6SHY8++XDnCgAAAMAxsCV7RHX3XdP9vUnentk5nu6ZDrnLdH/vNHx/ktPn3n5akrs2qK/18y7v7t3dvXvHIx9zLFsBAAAAYJOGB1FV9aiq+soDj5Ocl+TmJNckOXDluz1J3jE9vibJi6er552b5NPToXvXJjmvqk6aTlJ+3lQDAAAAYBvaikPznpjk7VV14Of/Rnf/XlXdkOTqqnppko8l+c5p/LuSPC/JviSfS/KSJOnu+6vqJ5PcMI37ie6+f1wbAAAAAByO4UFUd38kydevUf+bJM9Zo95JXrbOZ12R5IpjPUcAAAAAjr2tumoeAAAAAMcZQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhdm71BFbNGZe887DG33Hp8xc0EwAAAIDtxR5RAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACGEEQBAAAAMIQgCgAAAIAhBFEAAAAADCGIAgAAAGAIQRQAAAAAQwiiAAAAABhCEAUAAADAEIIoAAAAAIYQRAEAAAAwhCAKAAAAgCEEUQAAAAAMIYgCAAAAYAhBFAAAAABDCKIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEPs3OoJHO/OuOSdhzX+jkufv6CZAAAAACyWPaIAAAAAGEIQBQAAAMAQgigAAAAAhhBEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBCCKAAAAACG2LnVE+DwnHHJOzc99o5Ln7/AmQAAAAAcHntEAQAAADCEIAoAAACAIQRRAAAAAAwhiAIAAABgCEEUAAAAAEMIogAAAAAYQhAFAAAAwBA7t3oCLM4Zl7zzsMbfcenzFzQTAAAAAHtEAQAAADCIIAoAAACAIQRRAAAAAAzhHFF8gXNKAQAAAItkjygAAAAAhrBHFEfMHlQAAADA4RBEMYzgCgAAAI5vDs0DAAAAYAh7RLFt2YMKAAAAVosgipUhuAIAAIDtbemDqKq6IMnPJdmR5Je7+9ItnhJL4nCDq8Ml6AIAAIAvtdRBVFXtSPKGJN+aZH+SG6rqmu6+dWtnBosPug7X4QZjhzN/oRsAAACbsdRBVJJzkuzr7o8kSVVdleTCJIIoOMgig7HtFrotmuANAADgyCx7EHVqkjvnnu9P8swtmgtwnDjegjeWx3YLSY+nQ6CPp14BAI5GdfdWz+GIVdV3Jjm/u79vev49Sc7p7h88aNzFSS6enj4tyc1DJzrW45N8YqsnsSCr3Fuiv2Wnv+W1yr0l+lt2+lteq9xbor9lp7/ltcq9Jfpbdl/V3ScfatCy7xG1P8npc89PS3LXwYO6+/IklydJVe3t7t1jpjfeKve3yr0l+lt2+lteq9xbor9lp7/ltcq9JfpbdvpbXqvcW6K/48VDtnoCR+mGJLuq6syqOiHJRUmu2eI5AQAAALCGpd4jqrsfqKqXJ7k2yY4kV3T3LVs8LQAAAADWsNRBVJJ097uSvOsw3nL5ouayTaxyf6vcW6K/Zae/5bXKvSX6W3b6W16r3Fuiv2Wnv+W1yr0l+jsuLPXJygEAAABYHst+jigAAAAAlsRxE0RV1QVV9eGq2ldVl2z1fI5UVd1RVR+qqpuqau9Ue2xVXVdVt0/3J031qqrXTz1/sKqesbWz/3JVdUVV3VtVN8/VDrufqtozjb+9qvZsRS9rWae/V1fVx6c1vKmqnjf32iun/j5cVefP1bfd729VnV5V766q26rqlqr6oam+Euu3QX+rsn4Pr6r3VdUHpv5+fKqfWVXvndbirdOFIFJVD5ue75teP2Pus9bse6ts0Nubquqjc2t39lRfqt/NA6pqR1W9v6p+Z3q+9Gs3b43+Vmb96hhty5esv1X523liVb2tqv68ZtuHf75ia7dWf6uydl8z18PGs65uAAAJ0UlEQVRNVfWZqnrFqqzfBv2txPolSVX9x5pt12+uqrfUbHt/Zq3Atm+d3lZpu/dDU2+3VNUrptpKfPeSdftbme/eQnT3yt8yO5H5XyZ5cpITknwgyVlbPa8j7OWOJI8/qPbfklwyPb4kyWumx89L8rtJKsm5Sd671fNfo59vSvKMJDcfaT9JHpvkI9P9SdPjk7a6tw36e3WS/7zG2LOm382HJTlz+p3dsV1/f5OckuQZ0+OvTPIXUw8rsX4b9Lcq61dJvmJ6/NAk753W5eokF031X0zyA9Pj/5DkF6fHFyV560Z9b9Pe3pTkhWuMX6rfzbl5/6ckv5Hkd6bnS792h+hvZdYvx2BbvoT9vTqr8bfzyiTfNz0+IcmJK7Z2a/W3Emt30Nx3JPnrJF+1Suu3Tn8rsX5JTk3y0SSPmJ5fneTfZwW2fRv09qaswHYvydOS3JzkkZmdo/oPkuxale/eBv2txHdvUbfjZY+oc5Ls6+6PdPffJbkqyYVbPKdj6cLM/sMh0/0L5upv7pn3JDmxqk7Zigmup7v/JMn9B5UPt5/zk1zX3fd39yeTXJfkgsXP/tDW6W89Fya5qrs/390fTbIvs9/dbfn72913d/efTY8/m+S2zDakK7F+G/S3nmVbv+7uv52ePnS6dZJnJ3nbVD94/Q6s69uSPKeqKuv3vWU26G09S/W7mSRVdVqS5yf55el5ZQXW7oCD+zuEpVu/dazE384jsDR/O6vq0Zn9D6Y3Jkl3/113fyorsnYb9LeepVm7NTwnyV92919lRdbvIPP9rWcZ129nkkdU1c7M/tF/d1Zn23dwb3dtMHbZfjefmuQ93f257n4gyR8n+faszndvvf7Ws4zfvWPueAmiTk1y59zz/dn4H5TbWSf5/aq6saounmpP7O67k9k/npM8Yaova9+H288y9vnyaVfTKw7shpol7m/a3fnpme15snLrd1B/yYqsX80Ofbopyb2Zbcz/Msmnpo1o8qVz/UIf0+ufTvK4bNP+Du6tuw+s3U9Pa/e6qnrYVFu6tUvy35P8lyT/MD1/XFZk7SYH93fAqqzfsdiWL1t/yfL/7XxykvuS/ErNDhv95ap6VFZn7dbrL1n+tTvYRUneMj1elfWbN99fsgLr190fT/IzST6WWQD16SQ3ZgW2fWv11t2/P728Ctu9m5N8U1U9rqoemdkeT6dndb576/WXrMB3b1GOlyCq1qgt6+UCn9Xdz0jy3CQvq6pv2mDsKvWdrN/PsvV5WZKvTnJ2Zhub1071peyvqr4iyW8meUV3f2ajoWvUlrG/lVm/7n6wu89Oclpm/xfmqWsNm+6Xqr+De6uqpyV5ZZKnJPlnme3W/SPT8KXqrar+VZJ7u/vG+fIaQ5dy7dbpL1mR9Zsci235svW3Cn87d2Z2uP1l3f30JP83s8NJ1rNMvSXr97cKa/cFNTuH0Lcl+Z+HGrpGbRn7W4n1m/4Rf2FmhzI9KcmjMvsbc7Cl2/at1VtVvSgrst3r7tuSvCaz/+n5e5kdcvbABm9Zlf5W4ru3KMdLELU/X0wlk9k/TDba3XHb6u67pvt7k7w9s3883nPgkLvp/t5p+LL2fbj9LFWf3X3P9I/kf0jyS/ni7sBL119VPTSzkObXu/u3pvLKrN9a/a3S+h0wHXrxR5kdh3/itFt48qVz/UIf0+uPyeyw023d31xvF/TscMvu7s8n+ZUs79o9K8m3VdUdme22/ezM9iBalbX7sv6q6tdWaP2O1bZ8qfpbkb+d+5Psn9vD8m2ZBTersnZr9rciazfvuUn+rLvvmZ6vyvod8CX9rdD6fUuSj3b3fd3990l+K8k3ZjW2fWv2tmLbvTd29zO6+5syW4fbs0LfvbX6W6Hv3kIcL0HUDUl21eyqCidktrvqNVs8p8NWVY+qqq888DjJeZntCnhNkj3TsD1J3jE9vibJi2vm3Mx287x78LSPxOH2c22S86rqpOn/KJw31bal+tLzdH17ZmuYzPq7qGZX+Tgzs5PcvS/b9Pd3Os7+jUlu6+6fnXtpJdZvvf5WaP1OrqoTp8ePyOw/gm5L8u4kL5yGHbx+B9b1hUn+sLs76/e9Zdbp7c/n/mOnMjsPwfzaLc3vZne/srtP6+4zMvt9+sPu/ndZgbVL1u3vRauyfsdwW75U/a3C387u/uskd1bV10yl5yS5NSuyduv1twprd5DvypcetrYS6zfnS/pbofX7WJJzq+qR03bgwPdvFbZ9a/V226ps95Kkqp4w3f/jJN+R2e/oynz31upvhb57i9Hb4IzpI26ZHav5F5mdA+XHtno+R9jDkzPb1e8DSW450Edmxztfn1myfH2Sx071SvKGqecPJdm91T2s0dNbMttV8e8zS4FfeiT9JPnezE70ti/JS7a6r0P096vT/D+Y2R+XU+bG/9jU34eTPHc7//4m+ReZ7S76wSQ3Tbfnrcr6bdDfqqzf1yV5/9THzUn+61R/cmYbw32Z7db/sKn+8On5vun1Jx+q723Y2x9Oa3dzkl/LF6+st1S/mwf1+s354lXlln7tDtHfSqxfjuG2fMn6W5W/nWcn2Tv18duZXblpJdZug/5WYu2meT0yyd8kecxcbZXWb63+Vmn9fjzJn2e2HfjVzK46thLbvnV6W4nt3jSv/51ZcPiBJM+Zaqv03Vurv5X57i3iVlPDAAAAALBQx8uheQAAAABsMUEUAAAAAEMIogAAAAAYQhAFAAAAwBCCqP/fzh3r2BRFYQD+V5iGEQ21iYQQhSk8gEYjnkDiDSiI3huoplAShUo0Cp1OJBoiaqXQKEaYwp2luKeYyAwzEvuO+L7m5Jz772Tv7uZP9gIAAABgCEUUAMBAVTWrqtdV9a6q3lTVrar65X+yqlqpqquj9ggA8LcoogAAxvrW3avdfS7JpSSXk9z5zZqVJIooAOCfV9296D0AAPw3qupLdy9veT+Z5FWSY0lOJHmY5PD08/XuflFVL5OcTfI+yYMkT7bLDToCAMAfU0QBAAz0cxE1ffuc5EyS9SSb3b1RVaeSPOruC1V1Mcnt7r4y5Q9tlxt7EgCAvTu46A0AAJCanktJ1qpqNcksyekd8rvNAQDsK4ooAIAFmq7mzZJ8ynxW1Mck5zOf5bmxw7Kbu8wBAOwrhpUDACxIVR1Pci/JWs/nJRxN8qG7N5NcS3Jgiq4nObJl6U45AIB9zYwoAICBqmqW5G3m1+u+Zz50/G53b07znh4n+ZrkeZIb3b1cVUtJnmU+0Px+kqfb5UafBQBgrxRRAAAAAAzhah4AAAAAQyiiAAAAABhCEQUAAADAEIooAAAAAIZQRAEAAAAwhCIKAAAAgCEUUQAAAAAMoYgCAAAAYIgf9M75h/MAC6QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(20,10))\n",
    "plt.hist(df['activity.play_duration'], density=False, bins=400)  # `density=False` would make counts\n",
    "plt.axis([0,10000, 0, 30000]) \n",
    "#axis([xmin,xmax,ymin,ymax])\n",
    "x_ticks = np.arange(0, 10000, 500)\n",
    "plt.xticks(x_ticks)\n",
    "plt.ylabel('count')\n",
    "plt.xlabel('Data');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_race(row):\n",
    "    if (row['activity.play_duration'] >= 0) & (row['activity.play_duration'] < 360):\n",
    "        return 0\n",
    "    if (row['activity.play_duration'] >= 361) & (row['activity.play_duration'] < 720) :\n",
    "        return 1\n",
    "    if (row['activity.play_duration'] >= 721) & (row['activity.play_duration'] < 1080) :\n",
    "        return 2\n",
    "    if (row['activity.play_duration'] >= 1081) & (row['activity.play_duration'] < 1440) :\n",
    "        return 3\n",
    "    if (row['activity.play_duration'] >= 1441) & (row['activity.play_duration'] < 1800) :\n",
    "        return 4\n",
    "    if (row['activity.play_duration'] >= 1801) & (row['activity.play_duration'] < 2160) :\n",
    "        return 5\n",
    "    if (row['activity.play_duration'] >= 2161) & (row['activity.play_duration'] < 2520) :\n",
    "        return 6\n",
    "    if (row['activity.play_duration'] >= 2521) & (row['activity.play_duration'] < 2880) :\n",
    "        return 7\n",
    "    if (row['activity.play_duration'] >= 2881) & (row['activity.play_duration'] < 3240) :\n",
    "        return 8\n",
    "    if (row['activity.play_duration'] >= 3241) & (row['activity.play_duration'] < 3600) :\n",
    "        return 9    \n",
    "    return 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['eventStrength'] = df.apply(lambda row: label_race(row), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For each log, we associate them with a different weight or strength, assuming that, for example, \n",
    "a comment in an article indicates a higher interest of the user on the item than a like, or than a simple view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# users: 1102\n",
      "# users with at least 5 interactions: 697\n"
     ]
    }
   ],
   "source": [
    "users_interactions_count_df = df.groupby(['machine_uuid', 'Game']).size().groupby('machine_uuid').size()\n",
    "print('# users: %d' % len(users_interactions_count_df))\n",
    "users_with_enough_interactions_df = users_interactions_count_df[users_interactions_count_df >= 5].reset_index()[['machine_uuid']]\n",
    "print('# users with at least 5 interactions: %d' % len(users_with_enough_interactions_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of interactions: 31902\n",
      "# of interactions from users with at least 5 interactions: 29734\n"
     ]
    }
   ],
   "source": [
    "print('# of interactions: %d' % len(df))\n",
    "interactions_from_selected_users_df = df.merge(users_with_enough_interactions_df, \n",
    "               how = 'right',\n",
    "               left_on = 'machine_uuid',\n",
    "               right_on = 'machine_uuid')\n",
    "print('# of interactions from users with at least 5 interactions: %d' % len(interactions_from_selected_users_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Users are allowed to play games many times. Thus, to model the user interest on a given article, we aggregate all the interactions the user has performed in an item by a weighted sum of interaction type strength and apply a log transformation to smooth the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of unique user/item interactions: 13291\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>machine_uuid</th>\n",
       "      <th>Game</th>\n",
       "      <th>eventStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>88001908A0000910</td>\n",
       "      <td>3-D Tic-Tac-Toe (2600)</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>88001908A0000910</td>\n",
       "      <td>64th Street - A Detective Story</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>88001908A0000910</td>\n",
       "      <td>Basketball (2600)</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>88001908A0000910</td>\n",
       "      <td>Big Run</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>88001908A0000910</td>\n",
       "      <td>Fix-It Felix Jr.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       machine_uuid                             Game  eventStrength\n",
       "0  88001908A0000910           3-D Tic-Tac-Toe (2600)            0.0\n",
       "1  88001908A0000910  64th Street - A Detective Story            0.0\n",
       "2  88001908A0000910                Basketball (2600)            0.0\n",
       "3  88001908A0000910                          Big Run            0.0\n",
       "4  88001908A0000910                 Fix-It Felix Jr.            1.0"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def smooth_user_preference(x):\n",
    "    return math.log(1+x, 2)\n",
    "    \n",
    "interactions_full_df = interactions_from_selected_users_df \\\n",
    "                    .groupby(['machine_uuid', 'Game'])['eventStrength'].sum() \\\n",
    "                    .apply(smooth_user_preference).reset_index()\n",
    "print('# of unique user/item interactions: %d' % len(interactions_full_df))\n",
    "interactions_full_df['eventStrength'] = interactions_full_df['eventStrength'].astype('float64') \n",
    "interactions_full_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_full_df.to_csv('interactions_full_df.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# interactions on Train set: 10632\n",
      "# interactions on Test set: 2659\n"
     ]
    }
   ],
   "source": [
    "interactions_train_df, interactions_test_df = train_test_split(interactions_full_df,\n",
    "                                                               stratify=interactions_full_df['machine_uuid'], \n",
    "                                                               test_size=0.2,\n",
    "                                                               random_state=623)\n",
    "\n",
    "print('# interactions on Train set: %d' % len(interactions_train_df))\n",
    "print('# interactions on Test set: %d' % len(interactions_test_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Evaluation is important for machine learning projects, because it allows to compare objectivelly different algorithms and hyperparameter choices for models.\n",
    "- One key aspect of evaluation is to ensure that the trained model generalizes for data it was not trained on, using Cross-validation techniques. We are using here a simple cross-validation approach named holdout, in which a random data sample (20% in this case) are kept aside in the training process, and exclusively used for evaluation. All evaluation metrics reported here are computed using the test set.\n",
    "\n",
    "- Ps. A more robust evaluation approach could be to split train and test sets by a reference date, where the train set is composed by all interactions before that date, and the test set are interactions after that date. For the sake of simplicity, we chose the first random approach for this notebook, but you may want to try the second approach to better simulate how the recsys would perform in production predicting \"future\" users interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In Recommender Systems, there are a set metrics commonly used for evaluation. We chose to work with Top-N accuracy metrics, which evaluates the accuracy of the top recommendations provided to a user, comparing to the items the user has actually interacted in test set.\n",
    "This evaluation method works as follows:\n",
    "\n",
    "- For each user\n",
    ">- For each item the user has interacted in test set\n",
    "  >- Sample 100 other items the user has never interacted.\n",
    "    >- Ps. Here we naively assume those non interacted items are not relevant to the user, which might not be true, as the user may simply not be aware of those not interacted items. But let's keep this assumption.\n",
    "    >- Ask the recommender model to produce a ranked list of recommended items, from a set composed one interacted item and the 100 non-interacted (\"non-relevant!) items\n",
    "    >- Compute the Top-N accuracy metrics for this user and interacted item from the recommendations ranked list\n",
    "    >- Aggregate the global Top-N accuracy metrics\n",
    "    \n",
    "> The Top-N accuracy metric choosen was Recall@N which evaluates whether the interacted item is among the top N items (hit) in the ranked list of 101 recommendations for a user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Indexing by machineID to speed up the searches during evaluation\n",
    "interactions_full_indexed_df = interactions_full_df.set_index('machine_uuid')\n",
    "interactions_train_indexed_df = interactions_train_df.set_index('machine_uuid')\n",
    "interactions_test_indexed_df = interactions_test_df.set_index('machine_uuid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_items_interacted(machine_uuid, interactions_df):\n",
    "    # Get the user's data and merge in the movie information.\n",
    "    interacted_items = interactions_df.loc[machine_uuid]['Game']\n",
    "    return set(interacted_items if type(interacted_items) == pd.Series else [interacted_items])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Basketball (2600)'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sample game title\n",
    "interactions_test_indexed_df.loc['88001908A0000910']['Game']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Top-N accuracy metrics consts\n",
    "EVAL_RANDOM_SAMPLE_NON_INTERACTED_ITEMS = 10\n",
    "\n",
    "class ModelEvaluator:\n",
    "\n",
    "\n",
    "    def get_not_interacted_items_sample(self, machine_uuid, sample_size, seed=42):\n",
    "        interacted_items = get_items_interacted(machine_uuid, interactions_full_indexed_df)\n",
    "        all_items = set(df['Game'])\n",
    "        non_interacted_items = all_items - interacted_items\n",
    "\n",
    "        random.seed(seed)\n",
    "        non_interacted_items_sample = random.sample(non_interacted_items, sample_size)\n",
    "        return set(non_interacted_items_sample)\n",
    "\n",
    "    def _verify_hit_top_n(self, item_id, recommended_items, topn):        \n",
    "            try:\n",
    "                index = next(i for i, c in enumerate(recommended_items) if c == item_id)\n",
    "            except:\n",
    "                index = -1\n",
    "            hit = int(index in range(0, topn))\n",
    "            return hit, index\n",
    "\n",
    "    def evaluate_model_for_user(self, model, machine_uuid):\n",
    "        #Getting the items in test set\n",
    "        interacted_values_testset = interactions_test_indexed_df.loc[machine_uuid]\n",
    "        if type(interacted_values_testset['Game']) == pd.Series:\n",
    "            person_interacted_items_testset = set(interacted_values_testset['Game'])\n",
    "        else:\n",
    "            person_interacted_items_testset = set(interacted_values_testset['Game'])  \n",
    "        interacted_items_count_testset = len(person_interacted_items_testset) \n",
    "\n",
    "        #Getting a ranked recommendation list from a model for a given user\n",
    "        person_recs_df = model.recommend_items(machine_uuid, \n",
    "                                               items_to_ignore=get_items_interacted(machine_uuid, \n",
    "                                                                                    interactions_train_indexed_df), \n",
    "                                               topn=1000000000)\n",
    "\n",
    "        hits_at_5_count = 0\n",
    "        hits_at_10_count = 0\n",
    "        #For each item the user has interacted in test set\n",
    "        for item_id in person_interacted_items_testset:\n",
    "            #Getting a random sample (5) items the user has not interacted \n",
    "            #(to represent items that are assumed to be no relevant to the user)\n",
    "            non_interacted_items_sample = self.get_not_interacted_items_sample(machine_uuid, \n",
    "                                                                               sample_size=EVAL_RANDOM_SAMPLE_NON_INTERACTED_ITEMS, seed=42)\n",
    "\n",
    "            #Combining the current interacted item with the 100 random items\n",
    "            items_to_filter_recs = non_interacted_items_sample.union(set([item_id]))\n",
    "\n",
    "            #Filtering only recommendations that are either the interacted item or from a random sample of 100 non-interacted items\n",
    "            valid_recs_df = person_recs_df[person_recs_df['Game'].isin(items_to_filter_recs)]                    \n",
    "            valid_recs = valid_recs_df['Game'].values\n",
    "            #Verifying if the current interacted item is among the Top-N recommended items\n",
    "            hit_at_5, index_at_5 = self._verify_hit_top_n(item_id, valid_recs, 5)\n",
    "            hits_at_5_count += hit_at_5\n",
    "            hit_at_10, index_at_10 = self._verify_hit_top_n(item_id, valid_recs, 10)\n",
    "            hits_at_10_count += hit_at_10\n",
    "\n",
    "        #Recall is the rate of the interacted items that are ranked among the Top-N recommended items, \n",
    "        #when mixed with a set of non-relevant items\n",
    "        recall_at_5 = hits_at_5_count / float(interacted_items_count_testset)\n",
    "        recall_at_10 = hits_at_10_count / float(interacted_items_count_testset)\n",
    "\n",
    "        person_metrics = {'hits@5_count':hits_at_5_count, \n",
    "                          'hits@10_count':hits_at_10_count, \n",
    "                          'interacted_count': interacted_items_count_testset,\n",
    "                          'recall@5': recall_at_5,\n",
    "                          'recall@10': recall_at_10}\n",
    "        return person_metrics\n",
    "\n",
    "    def evaluate_model(self, model):\n",
    "        #print('Running evaluation for users')\n",
    "        people_metrics = []\n",
    "        for idx, machine_uuid in enumerate(list(interactions_test_indexed_df.index.unique().values)):\n",
    "            #if idx % 100 == 0 and idx > 0:\n",
    "            #    print('%d users processed' % idx)\n",
    "            person_metrics = self.evaluate_model_for_user(model, machine_uuid)  \n",
    "            person_metrics['_machine_uuid'] = machine_uuid\n",
    "            people_metrics.append(person_metrics)\n",
    "        print('%d users processed' % idx)\n",
    "\n",
    "        detailed_results_df = pd.DataFrame(people_metrics).sort_values('interacted_count', ascending=False)\n",
    "        \n",
    "        global_recall_at_5 = detailed_results_df['hits@5_count'].sum() / float(detailed_results_df['interacted_count'].sum())\n",
    "        global_recall_at_10 = detailed_results_df['hits@10_count'].sum() / float(detailed_results_df['interacted_count'].sum())\n",
    "        \n",
    "        global_metrics = {'modelName': model.get_model_name(),\n",
    "                          'recall@5': global_recall_at_5,\n",
    "                          'recall@10': global_recall_at_10}    \n",
    "        return global_metrics, detailed_results_df\n",
    "    \n",
    "model_evaluator = ModelEvaluator() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Popularity model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- simply recommends to a user the most popular items that the user has not previously consumed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Game</th>\n",
       "      <th>eventStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Centipede® (Arcade)</td>\n",
       "      <td>843.609910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fix-It Felix Jr.</td>\n",
       "      <td>643.227683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BurgerTime??(Arcade)</td>\n",
       "      <td>617.466756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tetris®</td>\n",
       "      <td>432.876168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Tron</td>\n",
       "      <td>400.187357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Game  eventStrength\n",
       "0   Centipede® (Arcade)     843.609910\n",
       "1      Fix-It Felix Jr.     643.227683\n",
       "2  BurgerTime??(Arcade)     617.466756\n",
       "3               Tetris®     432.876168\n",
       "4                  Tron     400.187357"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Computes the most popular items\n",
    "item_popularity_df = interactions_full_df.groupby('Game')['eventStrength'].sum().sort_values(ascending=False).reset_index()\n",
    "item_popularity_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PopularityRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Popularity'\n",
    "    \n",
    "    def __init__(self, popularity_df, items_df=None):\n",
    "        self.popularity_df = popularity_df\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "        \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        # Recommend the more popular items that the user hasn't seen yet.\n",
    "        recommendations_df = self.popularity_df[~self.popularity_df['Game'].isin(items_to_ignore)].sort_values('eventStrength', ascending = False).head(topn)\n",
    "\n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'Game', \n",
    "                                                          right_on = 'Game')[['eventStrength', 'Game']]\n",
    "\n",
    "\n",
    "        return recommendations_df\n",
    "    \n",
    "popularity_model = PopularityRecommender(item_popularity_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Popularity recommendation model...\n",
      "696 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Popularity', 'recall@5': 0.49923497267759565, 'recall@10': 0.546448087431694}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>recall@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>_machine_uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>42</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>1.0</td>\n",
       "      <td>88001912B00093D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>35</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>0.813953</td>\n",
       "      <td>1.0</td>\n",
       "      <td>88001908A0069438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>24</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>0.960000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>88001908A04601E0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>683</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88001908A0121348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88001908A0389928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>657</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88001908A05528A8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88001908A03035E0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>639</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>88001910B0000888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>88001908A0095970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>19</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>0.950000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>88001908A03882F8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     hits@5_count  hits@10_count  interacted_count  recall@5  recall@10  \\\n",
       "7              42             43                43  0.976744        1.0   \n",
       "79             35             43                43  0.813953        1.0   \n",
       "14             24             25                25  0.960000        1.0   \n",
       "683             0              0                21  0.000000        0.0   \n",
       "142             0              0                21  0.000000        0.0   \n",
       "657             0              0                21  0.000000        0.0   \n",
       "563             0              0                21  0.000000        0.0   \n",
       "639             0              0                20  0.000000        0.0   \n",
       "271            20             20                20  1.000000        1.0   \n",
       "38             19             20                20  0.950000        1.0   \n",
       "\n",
       "        _machine_uuid  \n",
       "7    88001912B00093D0  \n",
       "79   88001908A0069438  \n",
       "14   88001908A04601E0  \n",
       "683  88001908A0121348  \n",
       "142  88001908A0389928  \n",
       "657  88001908A05528A8  \n",
       "563  88001908A03035E0  \n",
       "639  88001910B0000888  \n",
       "271  88001908A0095970  \n",
       "38   88001908A03882F8  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Popularity recommendation model...')\n",
    "pop_global_metrics, pop_detailed_results_df = model_evaluator.evaluate_model(popularity_model)\n",
    "print('\\nGlobal metrics:\\n%s' % pop_global_metrics)\n",
    "pop_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.Collaborative Filtering model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- leverage description or attributes from items the user has interacted to recommend similar items. It depends only on the user previous choices, making this method robust to avoid the cold-start problem. For textual items, like articles, news and books, it is simple to use the raw text to build item profiles and user profiles.\n",
    "\n",
    "- Here we are using a very popular technique in information retrieval (search engines) named TF-IDF. This technique converts unstructured text into a vector structure, where each word is represented by a position in the vector, and the value measures how relevant a given word is for an article"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Memory-based: This approach uses the memory of previous users interactions to compute users similarities based on items they've interacted (user-based approach) or compute items similarities based on the users that have interacted with them (item-based approach).\n",
    "> A typical example of this approach is User Neighbourhood-based CF, in which the top-N similar users (usually computed using Pearson correlation) for a user are selected and used to recommend items those similar users liked, but the current user have not interacted yet. This approach is very simple to implement, but usually do not scale well for many users. A nice Python implementation of this approach in available in Crab.\n",
    "\n",
    "- Model-based: This approach, models are developed using different machine learning algorithms to recommend items to users. There are many model-based CF algorithms, like neural networks, bayesian networks, clustering models, and latent factor models such as Singular Value Decomposition (SVD) and, probabilistic latent semantic analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix Factorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Latent factor models compress user-item matrix into a low-dimensional representation in terms of latent factors. One advantage of using this approach is that instead of having a high dimensional matrix containing abundant number of missing values we will be dealing with a much smaller matrix in lower-dimensional space.\n",
    "\n",
    "- A reduced presentation could be utilized for either user-based or item-based neighborhood algorithms that are presented in the previous section. There are several advantages with this paradigm. It handles the sparsity of the original matrix better than memory based ones. Also comparing similarity on the resulting matrix is much more scalable especially in dealing with large sparse datasets.\n",
    "\n",
    "- Here we a use popular latent factor model named Singular Value Decomposition (SVD)\n",
    "- https://en.wikipedia.org/wiki/Singular_value_decomposition\n",
    "\n",
    "- An important decision is the number of factors to factor the user-item matrix. The higher the number of factors, the more precise is the factorization in the original matrix reconstructions. Therefore, if the model is allowed to memorize too much details of the original matrix, it may not generalize well for data it was not trained on. Reducing the number of factors increases the model generalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Game</th>\n",
       "      <th>3-D Tic-Tac-Toe (2600)</th>\n",
       "      <th>64th Street - A Detective Story</th>\n",
       "      <th>8 Eyes</th>\n",
       "      <th>ART OF FIGHTING 2</th>\n",
       "      <th>ARTS-TOWER</th>\n",
       "      <th>Act-Fancer Cybernetick Hyper Weapon</th>\n",
       "      <th>Adventure (2600)</th>\n",
       "      <th>Adventure II (2600)</th>\n",
       "      <th>Air Cavalry</th>\n",
       "      <th>Air-Sea Battle (2600)</th>\n",
       "      <th>...</th>\n",
       "      <th>jzmain</th>\n",
       "      <th>krylz3-mars</th>\n",
       "      <th>lt2-edev</th>\n",
       "      <th>main</th>\n",
       "      <th>misty</th>\n",
       "      <th>orion</th>\n",
       "      <th>owner</th>\n",
       "      <th>poowhistles</th>\n",
       "      <th>rbowden</th>\n",
       "      <th>showtime is best</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>machine_uuid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>88001908A0000910</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A00010A8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A00012C0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A00017C8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A00027E0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A0007120</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A00081A8</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A0009810</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.70044</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A0011980</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.807355</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88001908A0015490</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 481 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Game              3-D Tic-Tac-Toe (2600)  64th Street - A Detective Story  \\\n",
       "machine_uuid                                                                \n",
       "88001908A0000910                     1.0                              1.0   \n",
       "88001908A00010A8                     0.0                              0.0   \n",
       "88001908A00012C0                     0.0                              0.0   \n",
       "88001908A00017C8                     0.0                              1.0   \n",
       "88001908A00027E0                     0.0                              0.0   \n",
       "88001908A0007120                     0.0                              0.0   \n",
       "88001908A00081A8                     0.0                              0.0   \n",
       "88001908A0009810                     0.0                              0.0   \n",
       "88001908A0011980                     0.0                              0.0   \n",
       "88001908A0015490                     0.0                              0.0   \n",
       "\n",
       "Game              8 Eyes  ART OF FIGHTING 2  ARTS-TOWER  \\\n",
       "machine_uuid                                              \n",
       "88001908A0000910     0.0            0.00000         0.0   \n",
       "88001908A00010A8     0.0            0.00000         0.0   \n",
       "88001908A00012C0     0.0            0.00000         0.0   \n",
       "88001908A00017C8     0.0            0.00000         0.0   \n",
       "88001908A00027E0     0.0            0.00000         0.0   \n",
       "88001908A0007120     0.0            0.00000         0.0   \n",
       "88001908A00081A8     0.0            0.00000         0.0   \n",
       "88001908A0009810     0.0            3.70044         0.0   \n",
       "88001908A0011980     0.0            0.00000         0.0   \n",
       "88001908A0015490     0.0            0.00000         0.0   \n",
       "\n",
       "Game              Act-Fancer Cybernetick Hyper Weapon  Adventure (2600)  \\\n",
       "machine_uuid                                                              \n",
       "88001908A0000910                                  0.0               0.0   \n",
       "88001908A00010A8                                  0.0               0.0   \n",
       "88001908A00012C0                                  0.0               0.0   \n",
       "88001908A00017C8                                  0.0               0.0   \n",
       "88001908A00027E0                                  0.0               0.0   \n",
       "88001908A0007120                                  0.0               0.0   \n",
       "88001908A00081A8                                  0.0               0.0   \n",
       "88001908A0009810                                  0.0               0.0   \n",
       "88001908A0011980                                  0.0               0.0   \n",
       "88001908A0015490                                  0.0               0.0   \n",
       "\n",
       "Game              Adventure II (2600)  Air Cavalry  Air-Sea Battle (2600)  \\\n",
       "machine_uuid                                                                \n",
       "88001908A0000910                  0.0          0.0               0.000000   \n",
       "88001908A00010A8                  0.0          0.0               0.000000   \n",
       "88001908A00012C0                  0.0          0.0               0.000000   \n",
       "88001908A00017C8                  0.0          1.0               0.000000   \n",
       "88001908A00027E0                  0.0          0.0               0.000000   \n",
       "88001908A0007120                  0.0          0.0               0.000000   \n",
       "88001908A00081A8                  0.0          1.0               0.000000   \n",
       "88001908A0009810                  0.0          0.0               0.000000   \n",
       "88001908A0011980                  0.0          0.0               2.807355   \n",
       "88001908A0015490                  0.0          0.0               0.000000   \n",
       "\n",
       "Game              ...  jzmain  krylz3-mars  lt2-edev  main  misty  orion  \\\n",
       "machine_uuid      ...                                                      \n",
       "88001908A0000910  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A00010A8  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A00012C0  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A00017C8  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A00027E0  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A0007120  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A00081A8  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A0009810  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A0011980  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "88001908A0015490  ...     0.0          0.0       0.0   0.0    0.0    0.0   \n",
       "\n",
       "Game              owner  poowhistles  rbowden  showtime is best  \n",
       "machine_uuid                                                     \n",
       "88001908A0000910    0.0          0.0      0.0               0.0  \n",
       "88001908A00010A8    0.0          0.0      0.0               0.0  \n",
       "88001908A00012C0    0.0          0.0      0.0               0.0  \n",
       "88001908A00017C8    0.0          0.0      0.0               0.0  \n",
       "88001908A00027E0    0.0          0.0      0.0               0.0  \n",
       "88001908A0007120    0.0          0.0      0.0               0.0  \n",
       "88001908A00081A8    0.0          0.0      0.0               0.0  \n",
       "88001908A0009810    0.0          0.0      0.0               0.0  \n",
       "88001908A0011980    0.0          0.0      0.0               0.0  \n",
       "88001908A0015490    0.0          0.0      0.0               0.0  \n",
       "\n",
       "[10 rows x 481 columns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Creating a sparse pivot table with users in rows and items in columns\n",
    "users_items_pivot_matrix_df = interactions_train_df.pivot(index='machine_uuid', \n",
    "                                                          columns='Game', \n",
    "                                                          values='eventStrength').fillna(0)\n",
    "\n",
    "users_items_pivot_matrix_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\bokhy\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_items_pivot_matrix = users_items_pivot_matrix_df.as_matrix()\n",
    "users_items_pivot_matrix[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['88001908A0000910',\n",
       " '88001908A00010A8',\n",
       " '88001908A00012C0',\n",
       " '88001908A00017C8',\n",
       " '88001908A00027E0',\n",
       " '88001908A0007120',\n",
       " '88001908A00081A8',\n",
       " '88001908A0009810',\n",
       " '88001908A0011980',\n",
       " '88001908A0015490']"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_ids = list(users_items_pivot_matrix_df.index)\n",
    "users_ids[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<697x481 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 10632 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_items_pivot_sparse_matrix = csr_matrix(users_items_pivot_matrix)\n",
    "users_items_pivot_sparse_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The number of factors to factor the user-item matrix.\n",
    "NUMBER_OF_FACTORS_MF = 25\n",
    "#Performs matrix factorization of the original user item matrix\n",
    "#U, sigma, Vt = svds(users_items_pivot_matrix, k = NUMBER_OF_FACTORS_MF)\n",
    "U, sigma, Vt = svds(users_items_pivot_sparse_matrix, k = NUMBER_OF_FACTORS_MF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(697, 25)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25, 25)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma = np.diag(sigma)\n",
    "sigma.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25, 481)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Vt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- After the factorization, we try to to reconstruct the original matrix by multiplying its factors. The resulting matrix is not sparse any more. It was generated predictions for items the user have not yet interaction, which we will exploit for recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7.96947536e-02,  9.90149301e-01,  8.80994011e-02, ...,\n",
       "        -4.74300731e-03, -1.68860884e-02, -3.29838755e-04],\n",
       "       [-4.51442033e-04,  4.33180887e-02, -9.61015396e-03, ...,\n",
       "         2.20047858e-03,  1.96956728e-02,  1.84427473e-03],\n",
       "       [ 4.16232449e-02, -4.69834786e-03,  5.34394134e-02, ...,\n",
       "         1.72254836e-02,  1.31021138e-02,  3.23075492e-04],\n",
       "       ...,\n",
       "       [-3.45797573e-02,  1.23847867e-01,  2.85533561e-02, ...,\n",
       "        -1.05366199e-02,  2.19477041e-02, -2.86542333e-03],\n",
       "       [ 1.30442246e-02,  1.21787204e-01, -4.30568483e-02, ...,\n",
       "        -3.89209511e-03,  1.84580222e-02,  8.96373610e-04],\n",
       "       [ 9.93599917e-02, -9.90343437e-02,  8.33598128e-02, ...,\n",
       "         6.67514757e-03, -2.55405373e-02, -2.42872730e-03]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) \n",
    "all_user_predicted_ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_user_predicted_ratings_norm = (all_user_predicted_ratings - all_user_predicted_ratings.min()) / (all_user_predicted_ratings.max() - all_user_predicted_ratings.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>88001908A0000910</th>\n",
       "      <th>88001908A00010A8</th>\n",
       "      <th>88001908A00012C0</th>\n",
       "      <th>88001908A00017C8</th>\n",
       "      <th>88001908A00027E0</th>\n",
       "      <th>88001908A0007120</th>\n",
       "      <th>88001908A00081A8</th>\n",
       "      <th>88001908A0009810</th>\n",
       "      <th>88001908A0011980</th>\n",
       "      <th>88001908A0015490</th>\n",
       "      <th>...</th>\n",
       "      <th>88001912B0171030</th>\n",
       "      <th>88001912B0176210</th>\n",
       "      <th>88001912B01769D8</th>\n",
       "      <th>88001912B01774B0</th>\n",
       "      <th>880019PPA0004158</th>\n",
       "      <th>88002002B00001C0</th>\n",
       "      <th>88002002B00069E8</th>\n",
       "      <th>88002002B00080D8</th>\n",
       "      <th>88002002B00087B8</th>\n",
       "      <th>88002002B00098B8</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Game</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3-D Tic-Tac-Toe (2600)</th>\n",
       "      <td>0.224455</td>\n",
       "      <td>0.217191</td>\n",
       "      <td>0.221004</td>\n",
       "      <td>0.234721</td>\n",
       "      <td>0.230746</td>\n",
       "      <td>0.218288</td>\n",
       "      <td>0.225084</td>\n",
       "      <td>0.229515</td>\n",
       "      <td>0.233847</td>\n",
       "      <td>0.216625</td>\n",
       "      <td>...</td>\n",
       "      <td>0.222432</td>\n",
       "      <td>0.229216</td>\n",
       "      <td>0.213741</td>\n",
       "      <td>0.254047</td>\n",
       "      <td>0.238223</td>\n",
       "      <td>0.226667</td>\n",
       "      <td>0.230760</td>\n",
       "      <td>0.214097</td>\n",
       "      <td>0.218414</td>\n",
       "      <td>0.226237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64th Street - A Detective Story</th>\n",
       "      <td>0.306975</td>\n",
       "      <td>0.221158</td>\n",
       "      <td>0.216806</td>\n",
       "      <td>0.303998</td>\n",
       "      <td>0.215031</td>\n",
       "      <td>0.219848</td>\n",
       "      <td>0.239975</td>\n",
       "      <td>0.262446</td>\n",
       "      <td>0.246601</td>\n",
       "      <td>0.226521</td>\n",
       "      <td>...</td>\n",
       "      <td>0.216970</td>\n",
       "      <td>0.220747</td>\n",
       "      <td>0.220319</td>\n",
       "      <td>0.226617</td>\n",
       "      <td>0.240760</td>\n",
       "      <td>0.289277</td>\n",
       "      <td>0.422385</td>\n",
       "      <td>0.228457</td>\n",
       "      <td>0.228270</td>\n",
       "      <td>0.208256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8 Eyes</th>\n",
       "      <td>0.225217</td>\n",
       "      <td>0.216361</td>\n",
       "      <td>0.222075</td>\n",
       "      <td>0.222273</td>\n",
       "      <td>0.216639</td>\n",
       "      <td>0.215985</td>\n",
       "      <td>0.236505</td>\n",
       "      <td>0.225443</td>\n",
       "      <td>0.222624</td>\n",
       "      <td>0.216673</td>\n",
       "      <td>...</td>\n",
       "      <td>0.217828</td>\n",
       "      <td>0.222575</td>\n",
       "      <td>0.216088</td>\n",
       "      <td>0.227764</td>\n",
       "      <td>0.235469</td>\n",
       "      <td>0.225566</td>\n",
       "      <td>0.246759</td>\n",
       "      <td>0.219820</td>\n",
       "      <td>0.213329</td>\n",
       "      <td>0.224787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ART OF FIGHTING 2</th>\n",
       "      <td>0.217635</td>\n",
       "      <td>0.218602</td>\n",
       "      <td>0.225729</td>\n",
       "      <td>0.221737</td>\n",
       "      <td>0.211763</td>\n",
       "      <td>0.218159</td>\n",
       "      <td>0.206993</td>\n",
       "      <td>0.265739</td>\n",
       "      <td>0.211113</td>\n",
       "      <td>0.216257</td>\n",
       "      <td>...</td>\n",
       "      <td>0.219025</td>\n",
       "      <td>0.213900</td>\n",
       "      <td>0.220704</td>\n",
       "      <td>0.219657</td>\n",
       "      <td>0.223215</td>\n",
       "      <td>0.218289</td>\n",
       "      <td>0.248402</td>\n",
       "      <td>0.234852</td>\n",
       "      <td>0.219492</td>\n",
       "      <td>0.218010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ARTS-TOWER</th>\n",
       "      <td>0.216521</td>\n",
       "      <td>0.216349</td>\n",
       "      <td>0.217243</td>\n",
       "      <td>0.220297</td>\n",
       "      <td>0.219193</td>\n",
       "      <td>0.217582</td>\n",
       "      <td>0.227639</td>\n",
       "      <td>0.215822</td>\n",
       "      <td>0.218320</td>\n",
       "      <td>0.217609</td>\n",
       "      <td>...</td>\n",
       "      <td>0.218059</td>\n",
       "      <td>0.222169</td>\n",
       "      <td>0.216197</td>\n",
       "      <td>0.220762</td>\n",
       "      <td>0.216758</td>\n",
       "      <td>0.216227</td>\n",
       "      <td>0.217035</td>\n",
       "      <td>0.215783</td>\n",
       "      <td>0.217413</td>\n",
       "      <td>0.220815</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 697 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 88001908A0000910  88001908A00010A8  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.224455          0.217191   \n",
       "64th Street - A Detective Story          0.306975          0.221158   \n",
       "8 Eyes                                   0.225217          0.216361   \n",
       "ART OF FIGHTING 2                        0.217635          0.218602   \n",
       "ARTS-TOWER                               0.216521          0.216349   \n",
       "\n",
       "                                 88001908A00012C0  88001908A00017C8  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.221004          0.234721   \n",
       "64th Street - A Detective Story          0.216806          0.303998   \n",
       "8 Eyes                                   0.222075          0.222273   \n",
       "ART OF FIGHTING 2                        0.225729          0.221737   \n",
       "ARTS-TOWER                               0.217243          0.220297   \n",
       "\n",
       "                                 88001908A00027E0  88001908A0007120  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.230746          0.218288   \n",
       "64th Street - A Detective Story          0.215031          0.219848   \n",
       "8 Eyes                                   0.216639          0.215985   \n",
       "ART OF FIGHTING 2                        0.211763          0.218159   \n",
       "ARTS-TOWER                               0.219193          0.217582   \n",
       "\n",
       "                                 88001908A00081A8  88001908A0009810  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.225084          0.229515   \n",
       "64th Street - A Detective Story          0.239975          0.262446   \n",
       "8 Eyes                                   0.236505          0.225443   \n",
       "ART OF FIGHTING 2                        0.206993          0.265739   \n",
       "ARTS-TOWER                               0.227639          0.215822   \n",
       "\n",
       "                                 88001908A0011980  88001908A0015490  ...  \\\n",
       "Game                                                                 ...   \n",
       "3-D Tic-Tac-Toe (2600)                   0.233847          0.216625  ...   \n",
       "64th Street - A Detective Story          0.246601          0.226521  ...   \n",
       "8 Eyes                                   0.222624          0.216673  ...   \n",
       "ART OF FIGHTING 2                        0.211113          0.216257  ...   \n",
       "ARTS-TOWER                               0.218320          0.217609  ...   \n",
       "\n",
       "                                 88001912B0171030  88001912B0176210  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.222432          0.229216   \n",
       "64th Street - A Detective Story          0.216970          0.220747   \n",
       "8 Eyes                                   0.217828          0.222575   \n",
       "ART OF FIGHTING 2                        0.219025          0.213900   \n",
       "ARTS-TOWER                               0.218059          0.222169   \n",
       "\n",
       "                                 88001912B01769D8  88001912B01774B0  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.213741          0.254047   \n",
       "64th Street - A Detective Story          0.220319          0.226617   \n",
       "8 Eyes                                   0.216088          0.227764   \n",
       "ART OF FIGHTING 2                        0.220704          0.219657   \n",
       "ARTS-TOWER                               0.216197          0.220762   \n",
       "\n",
       "                                 880019PPA0004158  88002002B00001C0  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.238223          0.226667   \n",
       "64th Street - A Detective Story          0.240760          0.289277   \n",
       "8 Eyes                                   0.235469          0.225566   \n",
       "ART OF FIGHTING 2                        0.223215          0.218289   \n",
       "ARTS-TOWER                               0.216758          0.216227   \n",
       "\n",
       "                                 88002002B00069E8  88002002B00080D8  \\\n",
       "Game                                                                  \n",
       "3-D Tic-Tac-Toe (2600)                   0.230760          0.214097   \n",
       "64th Street - A Detective Story          0.422385          0.228457   \n",
       "8 Eyes                                   0.246759          0.219820   \n",
       "ART OF FIGHTING 2                        0.248402          0.234852   \n",
       "ARTS-TOWER                               0.217035          0.215783   \n",
       "\n",
       "                                 88002002B00087B8  88002002B00098B8  \n",
       "Game                                                                 \n",
       "3-D Tic-Tac-Toe (2600)                   0.218414          0.226237  \n",
       "64th Street - A Detective Story          0.228270          0.208256  \n",
       "8 Eyes                                   0.213329          0.224787  \n",
       "ART OF FIGHTING 2                        0.219492          0.218010  \n",
       "ARTS-TOWER                               0.217413          0.220815  \n",
       "\n",
       "[5 rows x 697 columns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Convert reconstructed matrix to dataframe\n",
    "cf_preds_df = pd.DataFrame(all_user_predicted_ratings_norm, columns = users_items_pivot_matrix_df.columns, index=users_ids).transpose()\n",
    "cf_preds_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CFRecommender:\n",
    "    \n",
    "    MODEL_NAME = 'Collaborative Filtering'\n",
    "    \n",
    "    def __init__(self, cf_predictions_df, items_df=None):\n",
    "        self.cf_predictions_df = cf_predictions_df\n",
    "        self.items_df = items_df\n",
    "        \n",
    "    def get_model_name(self):\n",
    "        return self.MODEL_NAME\n",
    "        \n",
    "    def recommend_items(self, user_id, items_to_ignore=[], topn=10, verbose=False):\n",
    "        # Get and sort the user's predictions\n",
    "        sorted_user_predictions = self.cf_predictions_df[user_id].sort_values(ascending=False) \\\n",
    "                                    .reset_index().rename(columns={user_id: 'recStrength'})\n",
    "\n",
    "        # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
    "        recommendations_df = sorted_user_predictions[~sorted_user_predictions['Game'].isin(items_to_ignore)] \\\n",
    "                               .sort_values('recStrength', ascending = False) \\\n",
    "                               .head(topn)\n",
    "\n",
    "        if verbose:\n",
    "            if self.items_df is None:\n",
    "                raise Exception('\"items_df\" is required in verbose mode')\n",
    "\n",
    "            recommendations_df = recommendations_df.merge(self.items_df, how = 'left', \n",
    "                                                          left_on = 'Game', \n",
    "                                                          right_on = 'Game')[['recStrength', 'Game']]\n",
    "        return recommendations_df\n",
    "    \n",
    "cf_recommender_model = CFRecommender(cf_preds_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating Collaborative Filtering (SVD Matrix Factorization) model...\n",
      "696 users processed\n",
      "\n",
      "Global metrics:\n",
      "{'modelName': 'Collaborative Filtering', 'recall@5': 0.40502732240437156, 'recall@10': 0.5320218579234973}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hits@5_count</th>\n",
       "      <th>hits@10_count</th>\n",
       "      <th>interacted_count</th>\n",
       "      <th>recall@5</th>\n",
       "      <th>recall@10</th>\n",
       "      <th>_machine_uuid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>42</td>\n",
       "      <td>42</td>\n",
       "      <td>43</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>88001912B00093D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>38</td>\n",
       "      <td>42</td>\n",
       "      <td>43</td>\n",
       "      <td>0.883721</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>88001908A0069438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>22</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>0.880000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>88001908A04601E0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>683</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>88001908A0121348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>88001908A0389928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>657</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>88001908A05528A8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>88001908A03035E0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>639</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>88001910B0000888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>0.850000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>88001908A0095970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>0.850000</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>88001908A03882F8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     hits@5_count  hits@10_count  interacted_count  recall@5  recall@10  \\\n",
       "7              42             42                43  0.976744   0.976744   \n",
       "79             38             42                43  0.883721   0.976744   \n",
       "14             22             25                25  0.880000   1.000000   \n",
       "683             0              0                21  0.000000   0.000000   \n",
       "142             0              0                21  0.000000   0.000000   \n",
       "657             0              0                21  0.000000   0.000000   \n",
       "563             0              0                21  0.000000   0.000000   \n",
       "639             0              0                20  0.000000   0.000000   \n",
       "271            17             20                20  0.850000   1.000000   \n",
       "38             17             18                20  0.850000   0.900000   \n",
       "\n",
       "        _machine_uuid  \n",
       "7    88001912B00093D0  \n",
       "79   88001908A0069438  \n",
       "14   88001908A04601E0  \n",
       "683  88001908A0121348  \n",
       "142  88001908A0389928  \n",
       "657  88001908A05528A8  \n",
       "563  88001908A03035E0  \n",
       "639  88001910B0000888  \n",
       "271  88001908A0095970  \n",
       "38   88001908A03882F8  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Evaluating Collaborative Filtering (SVD Matrix Factorization) model...')\n",
    "cf_global_metrics, cf_detailed_results_df = model_evaluator.evaluate_model(cf_recommender_model)\n",
    "print('\\nGlobal metrics:\\n%s' % cf_global_metrics)\n",
    "cf_detailed_results_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Compare Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recall@5</th>\n",
       "      <th>recall@10</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>modelName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Popularity</th>\n",
       "      <td>0.499235</td>\n",
       "      <td>0.546448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Collaborative Filtering</th>\n",
       "      <td>0.405027</td>\n",
       "      <td>0.532022</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         recall@5  recall@10\n",
       "modelName                                   \n",
       "Popularity               0.499235   0.546448\n",
       "Collaborative Filtering  0.405027   0.532022"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "global_metrics_df = pd.DataFrame([pop_global_metrics, cf_global_metrics]) \\\n",
    "                        .set_index('modelName')\n",
    "global_metrics_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAH+CAYAAAAPn+YOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xucl3Wd///nWxBELfPYJmCjaR44CDqoecbzccpNC8uvmBnVamVZZrulVFu55qo3sta176bVVuhiBn49lLWSmScgjTxkujnKaOs5PIfg+/cHOD9AlFEH5hrmfr/d9rZzHeYzr8/ELXpwXZ/3VWqtAQAAoJlW6+kBAAAAeGWiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDB+vfUD95ggw1qS0tLT/14AACAHjVr1qxHa60bLu+8Hou2lpaWzJw5s6d+PAAAQI8qpdzXlfPcHgkAANBgog0AAKDBRBsAAECD9dhn2pblhRdeSEdHR55//vmeHoVeao011siQIUOy+uqr9/QoAKzirrrqqnzqU5/KggULctxxx+WUU05Z4viFF16Yz33ucxk8eHCS5IQTTshxxx3XefzJJ5/M1ltvncMOOyznnntukmTevHk54YQTMn369Ky22mr52te+lve+970r700BjdSoaOvo6Mib3vSmtLS0pJTS0+PQy9Ra89hjj6WjoyObbrppT48DwCpswYIFOf7443P11VdnyJAhGTNmTNra2rLNNtsscd773//+ziBb2pe+9KXsscceS+z72te+lo022ih/+tOf8uKLL+bxxx9fYe8B6D0adXvk888/n/XXX1+w8bqUUrL++uu7UgvACnfzzTdn8803z2abbZYBAwZk3LhxmTp1ape/f9asWXnooYey3377LbH/e9/7Xr7whS8kSVZbbbVssMEG3To30Ds1KtqSCDbeEH9+AFgZHnjggQwdOrRze8iQIXnggQdedt4ll1ySkSNH5vDDD8+cOXOSJC+++GJOOumkfPOb31zi3L/+9a9JFl6B22677XLEEUfkoYceWoHvAugtGhdtfVVLS0seffTRLp9TSslJJ53UeezMM8/MxIkTV+SIAMAitdaX7Vv6Hw4PPfTQtLe3Z/bs2dlnn30yfvz4JMl3vvOdHHTQQUtEX5LMnz8/HR0d2WWXXfK73/0u73rXu/LZz352xb0JoNcQbb3UwIED89Of/nS5oQcAdL8hQ4Z0XjlLFn4uf+ONN17inPXXXz8DBw5MknzkIx/JrFmzkiQ33HBDzj333LS0tOSzn/1sfvCDH+SUU07J+uuvnzXXXDOHHXZYkuSII47I7373u5X0joAmE21vQHt7e7baaqscd9xxGT58eD74wQ/ml7/8ZXbZZZdsscUWufnmm/P444/nPe95T0aOHJmddtops2fPTpI89thj2W+//TJ69Oh89KMfXeJf7P7zP/8zO+ywQ0aNGpWPfvSjWbBgwct+dv/+/TNhwoScffbZLzt22WWXZccdd8zo0aOzzz77dN5aMXHixIwfPz777bdfWlpa8tOf/jQnn3xyRowYkQMOOCAvvPBCkoX32e+xxx7Zfvvts//+++cvf/nLivj1AUCvNWbMmNx999259957M2/evEyePDltbW1LnLP435/Tpk3L1ltvnST50Y9+lPvvvz/t7e0588wzc/TRR+f0009PKSWHHnpopk+fniT51a9+9bKFTYC+SbS9Qffcc08+9alPZfbs2fnjH/+YH//4x7nuuuty5pln5utf/3pOO+20jB49OrNnz87Xv/71HH300UmSL3/5y9l1111zyy23pK2tLffff3+S5M4778xFF12U3/72t7n11lvTr1+//OhHP1rmzz7++OPzox/9KHPnzl1i/6677pobb7wxt9xyS8aNG5czzjij89j//M//5PLLL8/UqVNz1FFHZezYsfnDH/6QQYMG5fLLL88LL7yQT3ziE5kyZUpmzZqVY489Nv/0T/+0gn57ANA79e/fP+eee27233//bL311nnf+96XYcOG5dRTT820adOSJJMmTcqwYcOy7bbbZtKkSbnwwguX+7r/8i//kokTJ2bkyJH54Q9/mH/9139dwe8E6A0ateR/b7TppptmxIgRSZJhw4Zl7733TiklI0aMSHt7e+67775ccsklSZK99torjz32WObOnZtrr702P/3pT5MkBx98cNZdd90kC/9VbdasWRkzZkyS5LnnnstGG220zJ/95je/OUcffXQmTZqUQYMGde7v6OjI+9///vzlL3/JvHnzllj+/sADD8zqq6+eESNGZMGCBTnggAOSpHPeu+66K7fddlv23XffJAuXNH7b297Wnb8yAFglHHTQQTnooIOW2PeVr3yl8+tvfOMb+cY3vvGqr3HMMcfkmGOO6dx++9vfnmuvvbZb5wR6P9H2Br10r3qycGnel7ZXW221zJ8/P/37v/xX/NIHlZe10mGtNePHj1/uf8m/5MQTT8x2222XD33oQ537PvGJT+Qzn/lM2traMn369CUWKFl8vtVXX71zhpfmrbVm2LBhueGGG7r08wEAgBXL7ZEr2O677955e+P06dOzwQYb5M1vfvMS+6+88so88cQTSZK99947U6ZMycMPP5wkefzxx3Pfffe94uuvt956ed/73pf/+I//6Nw3d+7cDB48OEny/e9//zXNu+WWW+aRRx7pjLYXXnght99++2t6DQAAoPuIthVs4sSJmTlzZkaOHJlTTjmlM6JOO+20XHvttdluu+3yi1/8IptsskmSZJtttsk///M/Z7/99svIkSOz7777LnchkJNOOmmJVSQnTpyYI444IrvttttrfijngAEDMmXKlHz+85/Ptttum1GjRuX6669/je8aAKDnXXXVVdlyyy2z+eab5/TTT3/Z8QsvvDAbbrhhRo0alVGjRuX//t//myS57777sv3222fUqFEZNmxYzjvvvCTJs88+m4MPPjhbbbVVhg0bllNOOWWlvh/6rrKs54ysDK2trXXmzJlL7Lvzzjs7V1aC18ufIwBgwYIFeec735mrr746Q4YMyZgxY/KTn/xkiRU5L7zwwsycOTPnnnvuEt87b9681FozcODAPP300xk+fHiuv/76vOUtb8lNN92UsWPHZt68edl7773zj//4jznwwANX9ttjFVFKmVVrbV3eea60AQCwyrn55puz+eabZ7PNNsuAAQMybty4TJ06tUvfO2DAgM51AP72t7/lxRdfTJKsueaaGTt2bOc52223XTo6OlbMG4DFiDYAAFY5DzzwQIYOHdq5PWTIkDzwwAMvO++SSy7JyJEjc/jhhy/xwPQ5c+Zk5MiRGTp0aD7/+c+/7OHpf/3rX3PZZZdl7733XnFvAhaxeiQA0Egtp1ze0yOwHO2nH9zTI7yiZX0EaOmVuw899NAceeSRGThwYM4777yMHz8+//3f/50kGTp0aGbPnp0HH3ww73nPe3L44YfnrW99a5Jk/vz5OfLII/PJT34ym2222Yp/M/R5rrQBALDKGTJkyBJXzjo6Ol52tWz99dfvvA3yIx/5SGbNmvWy19l4440zbNiw/OY3v+ncN2HChGyxxRY58cQTV9D0sCTRBgDAKmfMmDG5++67c++992bevHmZPHly2traljhn8RW6p02b1rmQWUdHR5577rkkyRNPPJHf/va32XLLLZMkX/ziFzN37tycc845K+mdgNsjAQBYBfXv3z/nnntu9t9//yxYsCDHHntshg0bllNPPTWtra1pa2vLpEmTMm3atPTv3z/rrbdeLrzwwiQLV6I+6aSTUkpJrTWf/exnM2LEiHR0dORrX/tattpqq2y33XZJkhNOOCHHHXdcD75T+gJL/i+lX79+GTFiRObPn5+tt9463//+97Pmmmt22+u/0tKyr2bmzJn5wQ9+kEmTJmX69OkZMGBAdt55526baVXThD9HALxxPtPWfE3+TBv0Bl1d8r/RV9q6+7+su/JfLIMGDcqtt96aJPngBz+Y8847L5/5zGe6dY7XYv78+WltbU1r68L/LKdPn561115btAEAQB/hM22vYrfddss999yTJDnrrLMyfPjwDB8+vPMe5vb29my11VYZP35851Kxzz77bJKkpaUljz76aJKFV8r23HPPl73+ZZddlh133DGjR4/OPvvsk4ceeihJMnHixEyYMCH77bdfjj766EyfPj2HHHJI2tvbc9555+Xss8/OqFGj8pvf/CabbrppXnjhhSTJk08+mZaWls5tAACg9xNtr2D+/Pm58sorM2LEiMyaNSsXXHBBbrrpptx444357ne/m1tuuSVJctddd2XChAmZPXt23vzmN+c73/lOl3/GrrvumhtvvDG33HJLxo0blzPOOKPz2KxZszJ16tT8+Mc/7tzX0tKSj33sY/n0pz+dW2+9Nbvttlv23HPPXH75wiuSkydPznvf+96svvrq3fRbAAAAeppoW8pzzz2XUaNGpbW1NZtsskk+/OEP57rrrsthhx2WtdZaK2uvvXb+/u//vnPZ16FDh2aXXXZJkhx11FG57rrruvyzOjo6sv/++2fEiBH55je/mdtvv73zWFtbWwYNGrTc1zjuuONywQUXJEkuuOCCfOhDH3otbxcAAGi4Rn+mrScs/pm2l7zaYi1LP6Txpe3+/fvnxRdfTJI8//zzy/zeT3ziE/nMZz6Ttra2TJ8+PRMnTuw8ttZaa3Vp3l122SXt7e359a9/nQULFmT48OFd+j4AAKB3cKWtC3bffff87Gc/y7PPPptnnnkml156aXbbbbckyf33358bbrghSfKTn/wku+66a5KFtzK+9IDGSy65ZJmvO3fu3AwePDhJ8v3vf79Ls7zpTW/KU089tcS+o48+OkceeaSrbAAAsApypa0LtttuuxxzzDHZYYcdkiy8JXH06NFpb2/vfCzARz/60WyxxRb5+Mc/niQ57bTT8uEPfzhf//rXs+OOOy7zdSdOnJgjjjgigwcPzk477ZR77713ubMceuihOfzwwzN16tR861vfym677ZYPfvCD+eIXv5gjjzyy+940AMDyTFynpyfg1Uyc29MT0E08p+0NaG9vzyGHHJLbbrutR+eYMmVKpk6dmh/+8Ic9OkdT9LY/RwAsm+e0NV/7Gh/o6RF4NaKt8VaJ57SxfJ/4xCdy5ZVX5oorrujpUQAAgBVAtL0BLS0tPX6V7Vvf+laP/nwAAGDFshAJAABAg4k2AACABhNtAAAADSbaAAAAGky0LeV///d/M27cuLzjHe/INttsk4MOOih/+tOfXvH89vb2DB8+PEkyffr0HHLIIa/6+hdeeGFOOOGEbp35Jeecc06effbZzu2DDjoof/3rX9/w67a0tGTEiBEZNWpURo0aleuvvz4PPvhgDj/88CRLvu/p06fn+uuvf80/Y+bMmfnkJz/5hmcFAIBVTbNXj+zuBzYu51kVtdYcdthhGT9+fCZPnpwkufXWW/PQQw/lne98Z/fO8jrUWlNrzWqrLbu1zznnnBx11FFZc801k6RbHwNwzTXXZIMNNlhi35QpU1523vTp07P22mtn55137vJrz58/P62trWltXe4jKgAAoM9xpW0x11xzTVZfffV87GMf69w3atSo7Lbbbqm15nOf+1yGDx+eESNG5KKLLnrV17r55puz8847Z/To0dl5551z1113dR6bM2dODjjggGy55Zb58pe/3Ln/rLPOyvDhwzN8+PCcc845SRZeydt6663zD//wD9luu+0yZ86cfPzjH09ra2uGDRuW0047LUkyadKkPPjggxk7dmzGjh2bZOEVskcffTSf//zn853vfKfz50ycODH/+q//miT55je/mTFjxmTkyJGdr9UVi19hXHzfeeedl7PPPjujRo3Kb37zmzzyyCN573vfmzFjxmTMmDH57W9/2znDhAkTst9+++Xoo49e4mrdxIkTc+yxx2bPPffMZpttlkmTJnX+jK9+9avZaqutMnbs2Bx00EE56aST8pe//OUV55wyZUpKKXnpQe7z5s3Lhz70oYwYMSLbbrttpk+f3nnuRRddlJEjR2bYsGE5+eSTu/y7AACAFanZV9pWsttuuy3bb7/9Mo/99Kc/za233prf//73efTRRzNmzJjsvvvur/haW221Va699tr0798/v/zlL/OP//iPueSSS5IsDLrbbrsta665ZsaMGZODDz44pZRccMEFuemmm1JrzY477pg99tgj6667bu66665ccMEFneH1ta99Leutt14WLFiQvffeO7Nnz84nP/nJnHXWWcu8IjZu3LiceOKJ+Yd/+IckycUXX5yrrroqv/jFL3L33Xfn5ptvTq01bW1tufbaa5f5vsaOHZt+/fpl4MCBuemmm5b5nltaWvKxj30sa6+9dj772c8mST7wgQ/k05/+dHbdddfcf//92X///XPnnXcmSWbNmpXrrrsugwYNWiKekuSPf/xjrrnmmjz11FPZcsst8/GPfzy///3vc8kll+R3v/tdZs+enaOOOipjx47N448/nre85S0ZNGjQEq/x1FNPZdKkSdlxxx079333u99NkvzhD3/Iww8/nAMPPDAzZszIE088kc997nOZNWtWNtxww4wfPz6/+tWvsvfee7/if8YAALAyiLYuuu6663LkkUemX79+eetb35o99tgjM2bMyMiRI5d5/ty5czN+/PjcfffdKaXkhRde6Dy27777Zv3110+S/P3f/32uu+66lFJy2GGHZa211urc/5vf/CZtbW15+9vfnp122qnz+y+++OKcf/75mT9/fv7yl7/kjjvueMU5kmT06NF5+OGH8+CDD+aRRx7Juuuum0022SSTJk3KL37xi4wePTpJ8vTTT+fuu+9eZrQtKwa74pe//GXuuOOOzu0nn3wyTz31VJKkra3tZaH1koMPPjgDBw7MwIEDs9FGG+Whhx7Kddddl3e/+9158cUXs95666WtrS2llKy33nr561//+rLX+tKXvpSTTz45Z555Zue+O+64ozPENtpoo7zlLW/JzJkzU0rJO9/5zmy44YZJkn322SeXXHKJaAMAoMeJtsUMGzZsmZ/TShZ+nuy1+NKXvpSxY8fm0ksvTXt7e/bcc8/OY6WUJc4tpbzq678Uckly77335swzz8yMGTOy7rrr5phjjsnzzz+/3HkOP/zwTJkypXOhlZfe0xe+8IV89KMffU3v7bV48cUXc8MNNywzzhZ/X0sbOHBg59f9+vXL/PnzO39H8+bNy4ABAzqPDxgwIM8888wS33/LLbdkzpw5OeSQQ5aItm233TZTp07NuHHjMmfOnMyaNStz5szJXnvtlT/+8Y9pb2/PkCFD8rOf/Szz5s173e8bAAC6i8+0LWavvfbK3/72t85b6JJkxowZ+fWvf53dd989F110URYsWJBHHnkk1157bXbYYYdXfK25c+dm8ODBSRauGLm4q6++Oo8//niee+65/OxnP8suu+yS3XffPT/72c/y7LPP5plnnsmll16a3Xbb7WWv++STT2attdbKOuusk4ceeihXXnll57E3velNnVexljZu3LhMnjw5U6ZM6Vz1cf/998/3vve9PP3000mSBx54IA8//HDXflmvYOkZ9ttvv5x77rmd27feeuvrfu1dd901l112WZ5//vk888wzufzyy5d5Xq01n/70pzs/t7e4Y489NkOGDElra2tOPPHE7Lzzzunfv3/WXXfd/Nu//Vve//73Z7fddktLS0v69/dvGgAA9Dz/q3QxpZRceumlOfHEE3P66adnjTXWSEtLS84555zsvvvuueGGG7LtttumlJIzzjgjf/d3f5f29vZlvtbJJ5+c8ePH56yzzspee+21xLFdd901/+f//J/cc889+cAHPtC5auIxxxzTGYLHHXdcRo8e/bLX33bbbTN69OgMGzYsm222WXbZZZfOYxMmTMiBBx6Yt73tbbnmmmuW+L5hw4blqaeeyuDBg/O2t70tycKguvPOO/Oud70rSbL22mvnP//zP7PRRhu97t/hoYcemsMPPzxTp07Nt771rUyaNCnHH398Ro4cmfnz52f33XfPeeed97pee8yYMWlra8see+yRt771rWltbc0666yTefPmZfXVV+88r9aa2267rfPq5v/+7/+mra0t06ZNS2tra84+++zOc3feeedsscUWnbMfeuihSZLzzz8//fr1e52/BQAA6D7ltd72111aW1vrSyv6veTOO+/M1ltv3SPz0Ds8/fTTWWuttTJjxox8/OMfz7//+79n0KBB2WyzzTpvwVz6z9Gee+6ZM888M62trXn22WdTa81aa62Vq6++Ol/96ldz7bXXJkkefvjhbLTRRnniiScyduzYXHzxxY141ANAX9VyyrLvqKA52tf4QE+PwKtZzuOu6HmllFm11uU+98qVNnqVCRMm5I477sizzz6bAw88MAMHDsx6662XQYMG5YEHHnjVz8klC8Ns//33z2qrrZbBgwfnhz/8YeexT33qU/n973+fJDn11FMFGwAAjSDa6FV+/OMfv+Kxlz5DuPRz2xZ/nEBLS8sSz8xb3E9+8pM3PiAAAHQzC5EAAAA0WOOirac+Y8eqwZ8fAABWNV2KtlLKAaWUu0op95RSTlnG8WNKKY+UUm5d9H/HvZ5h1lhjjTz22GP+hzevS601jz32WNZYY42eHgUAALrNcj/TVkrpl+TbSfZN0pFkRillWq31jqVOvajWesIbGWbIkCHp6OjII4888kZehj5sjTXWyJAhQ3p6DAAA6DZdWYhkhyT31Fr/nCSllMlJ3p1k6Wh7w1ZfffVsuumm3f2yAAAAvVZXbo8cnGTOYtsdi/Yt7b2llNmllCmllKHdMh0AAEAf15UrbWUZ+5b+0NllSX5Sa/1bKeVjSb6fZK+XvVApE5JMSJJNNtnkNY7Kyuahps3WfvrBPT0CAAArQVeutHUkWfzK2ZAkDy5+Qq31sVrr3xZtfjfJ9st6oVrr+bXW1lpr64Ybbvh65gUAAOhTuhJtM5JsUUrZtJQyIMm4JNMWP6GU8rbFNtuS3Nl9IwIAAPRdy709stY6v5RyQpKfJ+mX5Hu11ttLKV9JMrPWOi3JJ0spbUnmJ3k8yTErcGYAAIA+oyufaUut9YokVyy179TFvv5Cki9072gAAAB06eHaAAAA9AzRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNmCFuOqqq7Lllltm8803z+mnn/6K502ZMiWllMycObNz3ze+8Y1svvnm2XLLLfPzn/+8c39LS0tGjBiRUaNGpbW1dYXODwDQFP17egBg1bNgwYIcf/zxufrqqzNkyJCMGTMmbW1t2WabbZY476mnnsqkSZOy4447du674447Mnny5Nx+++158MEHs88+++RPf/pT+vXrlyS55pprssEGG6zU9wMA0JNcaQO63c0335zNN988m222WQYMGJBx48Zl6tSpLzvvS1/6Uk4++eSsscYanfumTp2acePGZeDAgdl0002z+eab5+abb16Z4wMANIpoA7rdAw88kKFDh3ZuDxkyJA888MAS59xyyy2ZM2dODjnkkC5/bykl++23X7bffvucf/75K/AdAAA0h9sjgW5Xa33ZvlJK59cvvvhiPv3pT+fCCy98Td/729/+NhtvvHEefvjh7Lvvvtlqq62y++67d9/gAAAN5Eob0O2GDBmSOXPmdG53dHRk44037tx+6qmnctttt2XPPfdMS0tLbrzxxrS1tWXmzJmv+r0v/f+NNtoohx12mNsmAYA+QbQB3W7MmDG5++67c++992bevHmZPHly2traOo+vs846efTRR9Pe3p729vbstNNOmTZtWlpbW9PW1pbJkyfnb3/7W+69997cfffd2WGHHfLMM8/kqaeeSpI888wz+cUvfpHhw4f31FsEAFhp3B4JdLv+/fvn3HPPzf77758FCxbk2GOPzbBhw3Lqqad2htkrGTZsWN73vvdlm222Sf/+/fPtb387/fr1y0MPPZTDDjssSTJ//vx84AMfyAEHHLCy3hIAQI8py/r8yMrQ2tpaF38uE83TcsrlPT0Cr6L99IN7egSAFcrfQ83XvsYHenoEXs3EuT09ActRSplVa13uw2fdHgkAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAAN1qVoK6UcUEq5q5RyTynllFc57/BSSi2lLHcFFAAAAJZvuc9pK6X0S/LtJPsm6Ugyo5QyrdZ6x1LnvSnJJ5PctCIGBZYycZ2enoDlsdQyANANunKlbYck99Ra/1xrnZdkcpJ3L+O8ryY5I8nz3TgfAABAn9aVaBucZM5i2x2L9nUqpYxOMrTW+v+6cTYAAIA+ryvRVpaxr3YeLGW1JGcnOWm5L1TKhFLKzFLKzEceeaTrUwIAAPRRXYm2jiRDF9sekuTBxbbflGR4kumllPYkOyWZtqzFSGqt59daW2utrRtuuOHrnxoAAKCP6Eq0zUiyRSll01LKgCTjkkx76WCtdW6tdYNaa0uttSXJjUnaaq0zV8jEAAAAfchyo63WOj/JCUl+nuTOJBfXWm8vpXyllNK2ogcEAADoy5a75H+S1FqvSHLFUvtOfYVz93zjYwEAAJB08eHaAAAA9AzRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QZY7qgUAAASH0lEQVQAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwboUbaWUA0opd5VS7imlnLKM4x8rpfyhlHJrKeW6Uso23T8qAABA37PcaCul9Evy7SQHJtkmyZHLiLIf11pH1FpHJTkjyVndPikAAEAf1JUrbTskuafW+uda67wkk5O8e/ETaq1PLra5VpLafSMCAAD0Xf27cM7gJHMW2+5IsuPSJ5VSjk/ymSQDkuzVLdMBAAD0cV250laWse9lV9Jqrd+utb4jyeeTfHGZL1TKhFLKzFLKzEceeeS1TQoAANAHdSXaOpIMXWx7SJIHX+X8yUnes6wDtdbza62ttdbWDTfcsOtTAgAA9FFdibYZSbYopWxaShmQZFySaYufUErZYrHNg5Pc3X0jAgAA9F3L/UxbrXV+KeWEJD9P0i/J92qtt5dSvpJkZq11WpITSin7JHkhyRNJxq/IoQEAAPqKrixEklrrFUmuWGrfqYt9/alungsAAIB08eHaAAAA9AzRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwboUbaWUA0opd5VS7imlnLKM458ppdxRSpldSvlVKeXt3T8qAABA37PcaCul9Evy7SQHJtkmyZGllG2WOu2WJK211pFJpiQ5o7sHBQAA6Iu6cqVthyT31Fr/XGudl2RykncvfkKt9Zpa67OLNm9MMqR7xwQAAOibuhJtg5PMWWy7Y9G+V/LhJFe+kaEAAABYqH8XzinL2FeXeWIpRyVpTbLHKxyfkGRCkmyyySZdHBEAAKDv6sqVto4kQxfbHpLkwaVPKqXsk+SfkrTVWv+2rBeqtZ5fa22ttbZuuOGGr2deAACAPqUr0TYjyRallE1LKQOSjEsybfETSimjk/x7Fgbbw90/JgAAQN+03Girtc5PckKSnye5M8nFtdbbSylfKaW0LTrtm0nWTvJfpZRbSynTXuHlAAAAeA268pm21FqvSHLFUvtOXezrfbp5LgAAANLFh2sDAADQM0QbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAG61K0lVIOKKXcVUq5p5RyyjKO715K+V0pZX4p5fDuHxMAAKBvWm60lVL6Jfl2kgOTbJPkyFLKNkuddn+SY5L8uLsHBAAA6Mv6d+GcHZLcU2v9c5KUUiYneXeSO146odbavujYiytgRgAAgD6rK7dHDk4yZ7HtjkX7AAAAWMG6Em1lGfvq6/lhpZQJpZSZpZSZjzzyyOt5CQAAgD6lK9HWkWToYttDkjz4en5YrfX8WmtrrbV1ww03fD0vAQAA0Kd0JdpmJNmilLJpKWVAknFJpq3YsQAAAEi6EG211vlJTkjy8yR3Jrm41np7KeUrpZS2JCmljCmldCQ5Ism/l1JuX5FDAwAA9BVdWT0ytdYrklyx1L5TF/t6RhbeNgkAAEA36tLDtQEAAOgZog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIOJNgAAgAYTbQAAAA0m2gAAABpMtAEAADSYaAMAAGgw0QYAANBgog0AAKDBRBsAAECDiTYAAIAGE20AAAANJtoAAAAaTLQBAAA0mGgDAABoMNEGAADQYKINAACgwUQbAABAg4k2AACABhNtAAAADSbaAAAAGky0AQAANJhoAwAAaDDRBgAA0GCiDQAAoMFEGwAAQIN1KdpKKQeUUu4qpdxTSjllGccHllIuWnT8plJKS3cPCgAA0BctN9pKKf2SfDvJgUm2SXJkKWWbpU77cJInaq2bJzk7yb9096AAAAB9UVeutO2Q5J5a659rrfOSTE7y7qXOeXeS7y/6ekqSvUsppfvGBAAA6Ju6Em2Dk8xZbLtj0b5lnlNrnZ9kbpL1u2NAAACAvqx/F85Z1hWz+jrOSSllQpIJizafLqXc1YWfDyxDSTZI8mhPz8Gr+LIbDoBVm7+LGs7fQ73B27tyUleirSPJ0MW2hyR58BXO6Sil9E+yTpLHl36hWuv5Sc7vymDAqyulzKy1tvb0HAD0Xf4ugpWjK7dHzkiyRSll01LKgCTjkkxb6pxpScYv+vrwJP9da33ZlTYAAABem+Veaau1zi+lnJDk50n6JflerfX2UspXksystU5L8h9JflhKuScLr7CNW5FDAwAA9BXFBTHonUopExbdcgwAPcLfRbByiDYAAIAG68pn2gAAAOghog0AAKDBRBsAAECDdeU5bUBDlVLWq7W+7JmIALCilFK2SvLuJIOT1Cx8fu+0WuudPToYrMJcaYNeopTyxcW+3qaU8qcks0op7aWUHXtwNAD6iFLK55NMTlKS3JyFz/MtSX5SSjmlJ2eDVZnVI6GXKKX8rta63aKvL09ybq31ylLKDknOqbXu3LMTArCqW/QPhsNqrS8stX9AkttrrVv0zGSwanOlDXqnjWutVyZJrfXmJIN6eB4A+oYXk2y8jP1vW3QMWAF8pg16j81KKdOy8DaUIaWUNWutzy46tnoPzgVA33Fikl+VUu5OMmfRvk2SbJ7khB6bClZxog16j3cvtb1akpRS3prk31b+OAD0NbXWq0op70yyQxYuRFKSdCSZUWtd0KPDwSrMZ9oAAHjDSilr11qf7uk5YFUk2qCXKaUcnuSoJG9K8nySKbXWC3p2KgD6ulLK/bXWTXp6DlgVuT0SeolSympZuMzynUnG11rnLlqt6/OllBOT/Fet9YEeHRKAVVop5TOvdCjJ2itzFuhLRBv0HickuaXW+o1SyjmllDcv2r9akm2SPLTo1pTv9tyIAKzivp7km0nmL+OYVclhBRFt0Hu8P8k+i75+Isl9Sa5Msn+SPye5NMkVSUQbACvK75L8rNY6a+kDpZTjemAe6BN8pg16iVLK7FrryEVfz6i1jlns2Ixa65jFH8ANAN2tlLJlksdqrY8u49hba60P9cBYsMpzpQ16j/ZSyta11juT3FRKOSvJVVl4pW1GKWVIkod7dEIAVmm11rte5ZhggxXElTboJUopY5N8LsnBi3YdmmSrJH/Mwtsif5LkP2qtV/XMhAD0FVYyhpXLB0ahl6i1XpPksiS/TLJXkl8lmZTk6SS/SXKDYANgRSqlrFZKuTjJiCxcyXjvJIclGVJKObGUMrhnJ4RVkytt0MuUUjZP8qEk2y7a9YckP1h02yQArDCllE8mWeullYyTLL2S8dlJrGQM3Uy0AQDQJaWU3ybZp9b6XCnltCRP5v9fyXidJGckuaLWulcPjgmrHAuRQC9RSrksySv+K0uttW0ljgNA3/SmWutzi74+ZLGVjP+4aCXjr5RS3tJTw8GqSrRB73FmTw8AQJ9nJWPoAW6PBACgS6xkDD3DlTboJUopf8ir3x45ciWOA0AfVGu9ppSyVRauZPz1LFzJ+BdJds7ClYz/S7BB93OlDXqJUsrbX+14rfW+lTULAH2blYxh5RJtAAAADeb2SOhlSik7JflWkq2TDEjSL8kztdY3v+o3AsAbZCVj6BmiDXqfc5OMS/JfSVqTHJ1k8x6dCIC+wkrG0ANEG/RCtdZ7Sin9aq0LklxQSrm+p2cCYNVXa/11T88AfZFog97n2VLKgCS3llLOSPKXJGv18EwA9AFWMoaeYSES6GUWrSL5UBZ+nu3TSdZJ8p1a6z09OhgAqzwrGUPPEG3Qy5RS1kryXK31xUXb/ZIMrLU+27OTAQCwIqzW0wMAr9mvkqy52PagLHzIKQCsFKWUnUopM0opT5dS5pVSFpRSnuzpuWBVJdqg91mj1vr0SxuLvl7zVc4HgO52bpIjk9ydhf94eFwWPo4GWAFEG/Q+z5RStntpo5SyfZLnenAeAPqgRZ+l7ldrXVBrvSDJ2J6eCVZVVo+E3ufEJP9VSnlw0fbbkry/B+cBoO+xkjGsRBYigV6olLJ6ki2TlCR/rLW+0MMjAdCHWMkYVi7RBr1MKWXNJJ9J8vZa60dKKVsk2bLW+v96eDQA+ggrGcPK5TNt0PtckGReknct2u5I8s89Nw4AfZCVjGElEm3Q+7yj1npGkheSpNb6XBbeJgkAK4uVjGElEm3Q+8wrpQxKUpOklPKOJH/r2ZEA6GOsZAwrkdUjoRcppZQk5yW5KsnQUsqPkuyS5JienAuAPsdKxrASWYgEeplSyqwk+yXZKQtvi7yx1vpoz04FQF9jJWNYeVxpg97nxiSb1Vov7+lBAOiblrWScSnFSsawgrjSBr1MKeWOJO9Mcl+SZ7LwXzhrrXVkjw4GQJ9RSrkoyawkR9dahy/6rPUNtdZRPTwarJJcaYPe58CeHgCAPu8dtdb3l1KOTBauZLzoc9fACiDaoJeptd7X0zMA0OdZyRhWItEGAECXWckYVj6faQMA4DWxkjGsXK60AQDwWlnJGFYiV9oAAHhNrGQMK5doAwDgNSmlvH1Z+y2WBSuGaAMAAGiw1Xp6AAAAAF6ZaAMAAGgw0QYAANBgog0AAKDBRBsAAECD/X8STf4MvgvZswAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "ax = global_metrics_df.transpose().plot(kind='bar', figsize=(15,8))\n",
    "for p in ax.patches:\n",
    "    ax.annotate(\"%.3f\" % p.get_height(), (p.get_x() + p.get_width() / 2., p.get_height()), ha='center', va='center', xytext=(0, 10), textcoords='offset points')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Game</th>\n",
       "      <th>recStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Asteroids® (Arcade)</td>\n",
       "      <td>0.577550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BurgerTime??(Arcade)</td>\n",
       "      <td>0.435864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tempest® (Arcade)</td>\n",
       "      <td>0.432169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Asteroids® Deluxe (Arcade)</td>\n",
       "      <td>0.380093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Missile Command® (Arcade)</td>\n",
       "      <td>0.372521</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         Game  recStrength\n",
       "0         Asteroids® (Arcade)     0.577550\n",
       "1        BurgerTime??(Arcade)     0.435864\n",
       "2           Tempest® (Arcade)     0.432169\n",
       "3  Asteroids® Deluxe (Arcade)     0.380093\n",
       "4   Missile Command® (Arcade)     0.372521"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cf_recommender_model.recommend_items('88001908A00017C8', topn=5, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Game</th>\n",
       "      <th>eventStrength</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Centipede® (Arcade)</td>\n",
       "      <td>843.609910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fix-It Felix Jr.</td>\n",
       "      <td>643.227683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BurgerTime??(Arcade)</td>\n",
       "      <td>617.466756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tetris®</td>\n",
       "      <td>432.876168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Tron</td>\n",
       "      <td>400.187357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Tempest® (Arcade)</td>\n",
       "      <td>393.066848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Asteroids® (Arcade)</td>\n",
       "      <td>391.680449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>64th Street - A Detective Story</td>\n",
       "      <td>350.610033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Aladdin</td>\n",
       "      <td>319.033232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Crystal Castles® (Arcade)</td>\n",
       "      <td>280.821674</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Game  eventStrength\n",
       "0              Centipede® (Arcade)     843.609910\n",
       "1                 Fix-It Felix Jr.     643.227683\n",
       "2             BurgerTime??(Arcade)     617.466756\n",
       "3                          Tetris®     432.876168\n",
       "4                             Tron     400.187357\n",
       "5                Tempest® (Arcade)     393.066848\n",
       "6              Asteroids® (Arcade)     391.680449\n",
       "7  64th Street - A Detective Story     350.610033\n",
       "8                          Aladdin     319.033232\n",
       "9        Crystal Castles® (Arcade)     280.821674"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "popularity_model.recommend_items('88001908A00017C8', topn=10, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- From https://www.kaggle.com/gspmoreira/recommender-systems-in-python-101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Addition https://github.com/jiristo/recsys_matrixfactorization/blob/master/recsys.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}