03a_Creating_and_Evaluating_Solutions_(Console).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Evaluating Solutions (AWS Console)\n",
    "\n",
    "In this notebook, we'll train some models in Amazon Personalize and review their metrics - using using the [Amazon Personalize console UI](https://console.aws.amazon.com/personalize/home).\n",
    "\n",
    "> For an **alternative** approach to the same steps same steps *programmatically* using [Boto3, the AWS SDK for Python](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) - see Notebook [03b_Creating_and_Evaluating_Solutions_(Python_SDK).ipynb](03b_Creating_and_Evaluating_Solutions_(Python_SDK).ipynb) instead.\n",
    "\n",
    "⚠️ You'll need to already have run the previous notebooks in this series to set up your environment; and prepare and import your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "As discussed in our [data preparation notebook](01_Preparing_Input_Data.ipynb), different **recipe types** in Amazon Personalize look to solve **different tasks**.\n",
    "\n",
    "In this notebook, you will train three **\"solutions\"** (models) for different use-cases:\n",
    "\n",
    "1. A `User-Personalization` solution for recommending items relevant to a particular user\n",
    "1. A `SIMS` solution to recommend *similar items* for a given item ID\n",
    "1. A `Personalized-Ranking` solution which, given a user and a collection of possible items, ranks the items in order of decreasing relevance\n",
    "\n",
    "The `Popularity-Count` recipe (which just ranks items by popularity) may also be useful as a **baseline** for understanding how the metrics of trained solutions compare against a trivial solution - but we won't specifically cover it here.\n",
    "\n",
    "The full list of recipes (algorithms) offered by Amazon Personalize is provided in the [Choosing a Recipe](https://docs.aws.amazon.com/personalize/latest/dg/working-with-predefined-recipes.html) section of the developer guide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Solutions\n",
    "\n",
    "In Amazon Personalize a model is conceptually called a **solution**, and an actual trained model is a **solution version** - reflecting that the model can be re-trained with new updated data.\n",
    "\n",
    "The UI workflow will guide us to create a solution version immediately, but bear in mind (in case you need to abort for some reason) that only creating the *solution version* actually kicks off training an actual model!\n",
    "\n",
    "### A Note on `EVENT_VALUE`\n",
    "\n",
    "In our example, we used `EVENT_VALUE` to record the rating awarded for each review event - so we'll apply a **threshold** when creating each of our solutions, to train models that recommend movies our users are likely to *enjoy*, not just ones they might *watch*.\n",
    "\n",
    "Configuring this threshold (or alternatively, what *type* of events to train on e.g. click vs view vs purchase) is a **trade-off**:\n",
    "\n",
    "- Set the bar low, and the model will have more data to learn about each user's interests and preferences... But there may be more 'noise' from items a user was casually interested in but decided they didn't want/like.\n",
    "- Set the bar high, and each data point is a strong signal that a user purchased/liked each item... But there are fewer events available for the model to learn each user's interests from. This *sparsity* makes it harder to train good models.\n",
    "\n",
    "In practice, you may want to experiment with what you optimize the model for, to find the best balance for your use-case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User Personalization\n",
    "\n",
    "The User-Personalization (`aws-user-personalization`) recipe is optimized for all user->items recommendation scenarios. When recommending items, it uses automatic item exploration.\n",
    "\n",
    "With automatic exploration, Amazon Personalize automatically tests different item recommendations, learns from how users interact with these recommended items, and boosts recommendations for items that drive better engagement and conversion. This improves item discovery and engagement when you have a fast-changing catalog, or when new items, such as news articles or promotions, are more relevant to users when fresh.\n",
    "\n",
    "You can balance how much to explore (where items with less interactions data or relevance are recommended more frequently) against how much to exploit (where recommendations are based on what we know or relevance). Amazon Personalize automatically adjusts future recommendations based on implicit user feedback.\n",
    "\n",
    "Let's get started by creating your first solution using User-Personalization:\n",
    "\n",
    "▶️ **Click Start** in the 'Solution creation' section of your dataset group dashboard to start the workflow, once your datasets have imported:\n",
    "\n",
    "![](static/imgs/dashboard-data-imported.png \"Screenshot of dataset group dashboard with data imported\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Configure** your solution with:\n",
    "\n",
    "- **Name** `personalize-poc-userpersonalization`\n",
    "- **Recipe** `aws-user-personalization`\n",
    "- **Event type** `review` (recall from notebook 1 that that all interactions in our dataset should have been tagged with this type)\n",
    "- **Event value** as 3 (use reviews of 3 stars or more, for our example)\n",
    "\n",
    "![](static/imgs/create-solution-up.png \"Screenshot of User-Personalization solution create screen\")\n",
    "\n",
    "Once you've checked the configurations, go ahead and click **Next**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Click** 'Finish' on the next screen to *actually start your solution training*\n",
    "\n",
    "> ⚠️ **Remember**: If you don't create a *solution version*, you won't actually build a model!\n",
    "\n",
    "![](static/imgs/create-solnver-up.png \"Screenshot of 'create solution version' step\")\n",
    "\n",
    "You'll typically be returned to the dataset group dashboard, with a message that your solution version creation is now underway. Note that all tabs in the sidebar are now enabled:\n",
    "\n",
    "![](static/imgs/dashboard-solution-creating.png \"DSG dashboard with solution creating\")\n",
    "\n",
    "> ⏰ This training is kicked off *in the background* and can take a while to complete - upwards of 25 minutes and typically around 90 minutes for this recipe on our sample dataset.\n",
    "\n",
    "As with datasets and dataset import jobs, note that it's the **solution version status** that matters - not the *solution status*. You can review your solutions via the Solutions tab:\n",
    "\n",
    "![](static/imgs/solutions-up-only.png \"Solutions list showing U-P solution only\")\n",
    "\n",
    "...and select the solution of interest to see the status of its underlying versions:\n",
    "\n",
    "![](static/imgs/solution-up-training.png \"Solution detail screen showing still-in-progress version training\")\n",
    "\n",
    "**Rather than waiting here**, we'll start our other solutions training first and then wait for all together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SIMS\n",
    "\n",
    "SIMS is one of the oldest algorithms used within Amazon for recommendation systems. A core use case for it is when you have one item and you want to recommend items that have been interacted with in similar ways over your entire user base. This means the result is not personalized per user. Sometimes this leads to recommending mostly popular items, so there is a hyperparameter that can be tweaked which will reduce the popular items in your results. \n",
    "\n",
    "For our use case, using the Movielens data, let's assume we pick a particular movie. We can then use SIMS to recommend other movies based on the interaction behavior of the entire user base. The results are not personalized per user, but instead, differ depending on the movie we chose as our input.\n",
    "\n",
    "▶️ **Open** the 'Solutions' tab (as above) and **click** on the *Create solution* button to get started. You'll see the same two-step workflow as before.\n",
    "\n",
    "▶️ **Configure** your solution with:\n",
    "\n",
    "- **Name** `personalize-poc-sims`\n",
    "- **Recipe** `aws-sims`\n",
    "- **Event type** `review` (recall from notebook 1 that that all interactions in our dataset should have been tagged with this type)\n",
    "- **Event value** as 3 (use reviews of 3 stars or more, for our example)\n",
    "\n",
    "...and click **Next** when ready\n",
    "\n",
    "▶️ **Click** 'Finish' on the next screen to *actually start your solution training*\n",
    "\n",
    "> ⚠️ **Remember**: If you don't create a *solution version*, you won't actually build a model!\n",
    "\n",
    "> ⏰ This training is kicked off *in the background* and can take a while to complete - upwards of 25 minutes and typically around 35 minutes for this recipe on our sample dataset.\n",
    "\n",
    "Rather than waiting here, we'll start our other solutions training first and then wait for all together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking\n",
    "\n",
    "Personalized Ranking is an interesting application of HRNN. Instead of just recommending what is most probable for the user in question, this algorithm takes in a user and a list of items as well. The items are then rendered back in the order of most probable relevance for the user. The use case here is for filtering on unique categories that you do not have item metadata to create a filter, or when you have a broad collection that you would like better ordered for a particular user.\n",
    "\n",
    "For our use case, using the MovieLens data, we could imagine that a VOD application may want to create a shelf of comic book movies, or movies by a specific director. We most likely have these lists based title metadata we have. We would use personalized ranking to re-order the list of movies for each user, based on their previous tagging history. \n",
    "\n",
    "Just like last time, we start by selecting the recipe.\n",
    "\n",
    "▶️ **Open** the 'Solutions' tab (as above) and **click** on the *Create solution* button to get started. You'll see the same two-step workflow as before.\n",
    "\n",
    "▶️ **Configure** your solution with:\n",
    "\n",
    "- **Name** `personalize-poc-rerank`\n",
    "- **Recipe** `aws-personalized-ranking`\n",
    "- **Event type** `review` (recall from notebook 1 that that all interactions in our dataset should have been tagged with this type)\n",
    "- **Event value** as 3 (use reviews of 3 stars or more, for our example)\n",
    "\n",
    "...and click **Next** when ready\n",
    "\n",
    "▶️ **Click** 'Finish' on the next screen to *actually start your solution training*\n",
    "\n",
    "> ⚠️ **Remember**: If you don't create a *solution version*, you won't actually build a model!\n",
    "\n",
    "> ⏰ This training is kicked off *in the background* and can take a while to complete - upwards of 25 minutes and typically around 45 minutes for this recipe on our sample dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning *(Information Only)*\n",
    "\n",
    "Personalize offers the option of running *hyperparameter tuning* when creating a solution. Because of the additional computation required to perform hyperparameter tuning, this feature is turned off by default. Therefore, the solutions we created above, will simply use the default values of the hyperparameters for each recipe. For more information about hyperparameter tuning, see the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/customizing-solution-config-hpo.html).\n",
    "\n",
    "If instead you have settled on the correct recipe to use, and are ready to run hyperparameter tuning, you can enable it during the *create solution* workflow:\n",
    "\n",
    "![](static/imgs/create-solution-hpo-option.png \"Create solution screenshot showing HPO option\")\n",
    "\n",
    "If you already know the values you want to use for a specific hyperparameter, you can also set this value when you create the solution. Again, this is configured during the *create solution* workflow:\n",
    "\n",
    "![](static/imgs/create-solution-hpset-option.png \"Create solution screenshot showing manual HP configuration\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Training to Complete\n",
    "\n",
    "You should now have 3 solutions created, all in 'Active' state:\n",
    "\n",
    "![](static/imgs/solutions-all-active.png \"Solutions list view showing 3 active solutions\")\n",
    "\n",
    "▶️ **Check** the **details** of each of your solutions to see the status of the *solution version* we created inside each one.\n",
    "\n",
    "▶️ **Wait** until the **solution versions** enter 'Active' state for **all** of your solutions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Solution Versions\n",
    "\n",
    "It should not take more than ~90 minutes to train all the solutions from this notebook. While training is in progress, we recommend taking the time to read up on the various algorithms (recipes) and their behavior in detail. This is also a good time to consider alternatives to how the data was fed into the system and what kind of results you expect to see.\n",
    "\n",
    "When the solutions finish creating, the next step is to obtain the evaluation metrics. Personalize calculates these metrics based on a subset of the training data. The image below illustrates how Personalize splits the data. Given 10 users, with 10 interactions each (a circle represents an interaction), the interactions are ordered from oldest to newest based on the timestamp. Personalize uses all of the interaction data from 90% of the users (blue circles) to train the solution version, and the remaining 10% for evaluation. For each of the users in the remaining 10%, 90% of their interaction data (green circles) is used as input for the call to the trained model. The remaining 10% of their data (orange circle) is compared to the output produced by the model and used to calculate the evaluation metrics.\n",
    "\n",
    "![personalize metrics](static/imgs/personalize_metrics.png)\n",
    "\n",
    "We recommend reading [the documentation](https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html) to understand the metrics, but we have also copied parts of the documentation below for convenience.\n",
    "\n",
    "You need to understand the following terms regarding evaluation in Personalize:\n",
    "\n",
    "* *Relevant recommendation* refers to a recommendation that matches a value in the testing data for the particular user.\n",
    "* *Rank* refers to the position of a recommended item in the list of recommendations. Position 1 (the top of the list) is presumed to be the most relevant to the user.\n",
    "* *Query* refers to the internal equivalent of a GetRecommendations call.\n",
    "\n",
    "The metrics produced by Personalize are:\n",
    "* **coverage**: The proportion of unique recommended items from all queries out of the total number of unique items in the training data (includes both the Items and Interactions datasets).\n",
    "* **mean_reciprocal_rank_at_25**: The [mean of the reciprocal ranks](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) of the first relevant recommendation out of the top 25 recommendations over all queries. This metric is appropriate if you're interested in the single highest ranked recommendation.\n",
    "* **normalized_discounted_cumulative_gain_at_K**: Discounted gain assumes that recommendations lower on a list of recommendations are less relevant than higher recommendations. Therefore, each recommendation is discounted (given a lower weight) by a factor dependent on its position. To produce the [cumulative discounted gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) (DCG) at K, each relevant discounted recommendation in the top K recommendations is summed together. The normalized discounted cumulative gain (NDCG) is the DCG divided by the ideal DCG such that NDCG is between 0 - 1. (The ideal DCG is where the top K recommendations are sorted by relevance.) Amazon Personalize uses a weighting factor of 1/log(1 + position), where the top of the list is position 1. This metric rewards relevant items that appear near the top of the list, because the top of a list usually draws more attention.\n",
    "* **precision_at_K**: The number of relevant recommendations out of the top K recommendations divided by K. This metric rewards precise recommendation of the relevant items.\n",
    "\n",
    "Let's take a look at the evaluation metrics for each of the solutions produced in this notebook. *Please note, your results might differ from the results described in the text of this notebook, due to the quality of the Movielens dataset.* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User Personalization Metrics\n",
    "\n",
    "▶️ **Open** the `personalize-poc-userpersonalization` solution's detail page\n",
    "\n",
    "▶️ **Click** on the *solution version ID* to open the **solution version detail** page, as below:\n",
    "\n",
    "![](static/imgs/solnver-detail-up.png \"User-Personalization solution version details screen, showing metrics\")\n",
    "\n",
    "In our example above, the hold-out validation described above found that:\n",
    "\n",
    "- (`Coverage`) ~5% of the total item catalogue was recommended at least once across all the validation tests\n",
    "- (`Precision`) On average ~1.8% of the top 5 (and 2.2% of the top 10) recommendations were something that the user had later interacted with in the held-out validation set (orange, in the diagram above)\n",
    "- (`MRR`) The average inverse-rank of the first relevant item in presented recommendations was ~0.07 (approx 1/14) - suggesting ~14th place in the recommendation list\n",
    "- (`NDCG`) Even weighting that relevant recommendations further down the list are less valuable (because there's less chance of a user clicking on them), we see strong relative gains for presenting more recommendations (10, 25) rather than cutting off at the top 5.\n",
    "\n",
    "This is **clearly not a great model**, but keep in mind that **our source data (MovieLens ratings) was not ideal**:\n",
    "\n",
    "- **Ratings** are sparse as compared to views (who reviews every movie they watch?) so don't give us a very complete picture of what each user may be interested in\n",
    "- We further exacerbated this problem by filtering out any reviews of <3 stars: So could likely build a more \"accurate\" model by **optimizing more towards views**, rather than aiming for the high bar of \"viewed and liked\"\n",
    "- The **timestamps** of reviews may not match well with the times the movie was actually watched - which could cause additional problems because Personalize trains dynamic models which try to predict the *sequence* in which we might want to interact with items.\n",
    "- Remember, we only trained with a **small subset of the MovieLens data** for this example! Using the full dataset would give our model significantly more information to learn about patterns of user preference, and typically boosts metrics significantly in our tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SIMS Metrics\n",
    "\n",
    "Similarly, let's look at the metrics for our `personalize-poc-sims` solution version:\n",
    "\n",
    "![](static/imgs/solnver-metrics-sims.png \"Solution version details screen for SIMS model\")\n",
    "\n",
    "The metrics for this solution seem healthier pretty much across the board, both for item discoverability (recommending about twice as much of the item catalogue) and relevance.\n",
    "\n",
    "Remember that these recipes are solving **different tasks** though: These results may indicate that for this situation (where we seem to be struggling to get enough data to recommend what other movies a user might like overall) - we might be able to drive some better results by recommending other movies **from the context of one the user is already looking at**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking Metrics\n",
    "\n",
    "Similarly, let's look at the metrics for our `personalize-poc-rerank` solution version:\n",
    "\n",
    "![](static/imgs/solnver-detail-rerank.png \"Solution version details screen for re-ranking model\")\n",
    "\n",
    "Results here are more in line with the user personalization recipe, which is to be expected because in the training and validation setting the model doesn't know what the *input item list* would be, so is working in a very similar way to U-P.\n",
    "\n",
    "However note that item catalogue coverage, in particular, is much poorer than for the User-Personalization model. While the U-P recipe integrates some treatment for cold-starting recommendations for newer items with less history data, this ranking recipe did not at the time we ran our test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Evaluation Metrics\n",
    "\n",
    "It's tempting to over-focus on these evaluation metrics, but very important to consider the bigger picture:\n",
    "\n",
    "* In recommendation problems, there is a **strong feedback loop** that **favours the existing deployed system**: People don't click/purchase items they don't see, so the historical interaction data we train from is influenced by the biases of the previously deployed system(s).\n",
    "* This 'offline' validation procedure doesn't have any knowledge of **input item lists** you might supply to re-ranking models, or **filter rules** you may apply to models in general... Which might have significant impact on real-world deployed performance by filtering out known-irrelevant items (or accidentally removing important items!)\n",
    "* **Cold starting** of new items is difficult to evaluate using these metrics. The aim of cold-starting strategies is to recommend items which are new to your business. Therefore, these items will not appear in the existing user transaction data which is used to compute the evaluation metrics!\n",
    "\n",
    "Keeping in mind these factors, the evaluation metrics produced by Personalize are generally useful for two cases:\n",
    "\n",
    "1. Comparing the performance of solution versions trained on the same recipe, but with different values for the hyperparameters and features (impression data etc)\n",
    "1. Comparing the performance of solution versions trained on different recipes (except HRNN Coldstart).\n",
    "\n",
    "Properly evaluating a recommendation system is always best done through **A/B testing** while measuring **actual business outcomes**. Since recommendations generated by a system usually influence the user behavior which it is based on, it is better to run small experiments and apply A/B testing for **longer periods of time**. Over time, the bias from the existing model will fade."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Record your setup\n",
    "\n",
    "Although we did this setup in the AWS console, we'll want to use our models here in the SageMaker notebook later.\n",
    "\n",
    "To facilitate this, we need to store some values!\n",
    "\n",
    "▶️ **Check** the configurations in the cell below match your setup, and run it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_personalization_solution_name = \"personalize-poc-userpersonalization\"\n",
    "sims_solution_name = \"personalize-poc-sims\"\n",
    "rerank_solution_name = \"personalize-poc-rerank\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "▶️ **Run** the below, which will automatically determine and store the 'ARN's of your solutions, from the above and the other values we stored in previous notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "%store -r\n",
    "personalize = boto3.client(\"personalize\")\n",
    "\n",
    "def solnver_arn_from_solution_name(solution_name, dataset_group_arn):\n",
    "    # Use the dataset group ARN to build the solution ARN:\n",
    "    arn_base = dataset_group_arn.rpartition(\":\")[0]\n",
    "    solution_arn = f\"{arn_base}:solution/{solution_name}\"\n",
    "    solution_desc = personalize.describe_solution(\n",
    "        solutionArn=solution_arn\n",
    "    )\n",
    "    return solution_desc[\"solution\"][\"latestSolutionVersion\"][\"solutionVersionArn\"]\n",
    "\n",
    "up_solution_version_arn = solnver_arn_from_solution_name(\n",
    "    user_personalization_solution_name,\n",
    "    dataset_group_arn,\n",
    ")\n",
    "print(up_solution_version_arn)\n",
    "%store up_solution_version_arn\n",
    "sims_solution_version_arn = solnver_arn_from_solution_name(\n",
    "    sims_solution_name,\n",
    "    dataset_group_arn,\n",
    ")\n",
    "print(sims_solution_version_arn)\n",
    "%store sims_solution_version_arn\n",
    "rerank_solution_version_arn = solnver_arn_from_solution_name(\n",
    "    rerank_solution_name,\n",
    "    dataset_group_arn,\n",
    ")\n",
    "print(rerank_solution_version_arn)\n",
    "%store rerank_solution_version_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All set!\n",
    "\n",
    "We've now trained models for a range of different recommendation tasks, based on our historical data.\n",
    "\n",
    "In the next notebook we'll **deploy** these models to enable us to start generating real-time recommendations:\n",
    "\n",
    "- Follow along in the **AWS Console** with the instructions and screenshots in [04a_Deploying_Campaigns_and_Filters_(Console).ipynb](04a_Deploying_Campaigns_and_Filters_(Console).ipynb), *OR*\n",
    "- Run the same steps in code with the **AWS SDK for Python (Boto3)** by following [04b_Deploying_Campaigns_and_Filters_(Python_SDK).ipynb](04b_Deploying_Campaigns_and_Filters_(Python_SDK).ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}