lesson1_rulebased.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Run this every time you open the spreadsheet\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from collections import Counter\n",
    "import lib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and inspect the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the data.\n",
    "tweets, test_tweets = lib.read_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The variable \"tweets\" is a list of tweets.\n",
    "# Mini-exercise 1: Print the number of tweets in the list\n",
    "\n",
    "\n",
    "\n",
    "# Mini-exercise 2: Assign the 10th tweet to the variable \"tweet\" and print it\n",
    "\n",
    "\n",
    "\n",
    "# To view the category of a tweet, we access the attribute tweet.category\n",
    "# Mini-exercise 3: Print the category of this tweet.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function prints out a table containing all the tweets, along with their category labels\n",
    "lib.show_tweets(tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python refresher\n",
    "\n",
    "First, let's do some exercises to refresh our memory of a few Python concepts.\n",
    "\n",
    "### Functions\n",
    "A Python function is written like this:\n",
    "```\n",
    "def add_one(x):\n",
    "    return x+1\n",
    "```\n",
    "The name of the function is `add_one`, `x` is the input variable, and the `return` keyword tells us what to give as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 1. Define a function called \"square_minus_1\" that takes one variable (x), \n",
    "# squares it, subtracts 1, and returns the result.\n",
    "\n",
    "#### YOUR CODE STARTS HERE ####\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#### YOUR CODE ENDS HERE ####\n",
    "\n",
    "print \"Testing:\"\n",
    "for x in [3,-4,6.5,0]:\n",
    "    print str(x) + \" -> \" + str(square_minus_1(x)),\n",
    "    print \"CORRECT\" if square_minus_1(x)==(x**2-1) else \"INCORRECT\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### If-else statements\n",
    "\n",
    "An if/else statement looks like this:\n",
    "\n",
    "```\n",
    "if electoral_votes >= 270:\n",
    "    print \"You win the election\"\n",
    "else:\n",
    "    print \"You lose the election\"\n",
    "```\n",
    "\n",
    "The if-statement is evaluated (`electoral_votes >= 270`); if it's true then the code under the `if` is executed, if it's false then the code under the `else` is executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 2. Define a function called \"contains_ss\" that takes one variable (word) \n",
    "# and returns True if the word contains a double-s and False if it doesn't.\n",
    "# Hint: to test whether a string e.g. \"ss\" is inside another string variable e.g. word, write\n",
    "#    if \"ss\" in word: \n",
    "\n",
    "#### YOUR CODE STARTS HERE ####\n",
    "\n",
    "\n",
    "\n",
    "#### YOUR CODE ENDS HERE ####\n",
    "\n",
    "print \"Testing:\"\n",
    "for word in [\"computer\", \"science\", \"lesson\"]:\n",
    "    print \"%s ->\"%word, contains_ss(word),\n",
    "    print \"CORRECT\" if contains_ss(word)==(\"ss\" in word) else \"INCORRECT\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More complex if-else statements\n",
    "\n",
    "Maybe you want to check *several* conditions? You can use an if/elif/else statement.\n",
    "\n",
    "```\n",
    "if teamA_score > teamB_score:\n",
    "    print \"Team A wins\"\n",
    "elif teamA_score < teamB_score:\n",
    "    print \"Team B wins\"\n",
    "else:\n",
    "    print \"It's a tie!\"\n",
    "```\n",
    "\n",
    "`elif` stands for \"else if\". In fact, the above code is just a neater way of writing this:\n",
    "```\n",
    "if teamA_score > teamB_score:\n",
    "    print \"Team A wins\"\n",
    "else:\n",
    "    if teamA_score < teamB_score:\n",
    "        print \"Team B wins\"\n",
    "    else:\n",
    "        print \"It's a tie!\"\n",
    "```\n",
    "\n",
    "You can have as many `elif` statments as you like. These are useful for when you want several options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exercise 3. Define a function called \"grade\" that takes one input (score).\n",
    "# If score >= 90, return the string \"A\"\n",
    "# Otherwise, if score >= 80, return the string \"B\"\n",
    "# Otherwise, if score >= 70, return the string \"C\"\n",
    "# Otherwise, if score >= 60, return the string \"D\"\n",
    "# Otherwise, if score >= 50, return the string \"E\"\n",
    "# Otherwise, return the string \"F\"\n",
    "\n",
    "#### YOUR CODE STARTS HERE ####\n",
    "\n",
    "\n",
    "\n",
    "#### YOUR CODE ENDS HERE ####\n",
    "\n",
    "\n",
    "print \"Testing:\"\n",
    "for (score,g) in [(77,\"C\"),(80,\"B\"),(32,\"F\"),(100,\"A\"),(69,\"D\")]:\n",
    "    print \"%i -> %s\" % (score, grade(score)),\n",
    "    print \"CORRECT\" if grade(score)==g else \"INCORRECT\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write a rule-based tweet classifier\n",
    "\n",
    "Time to write our rule-based classifier!\n",
    "The function outline below uses a `if/elif/else` statement to return the predicted category of a tweet.\n",
    "\n",
    "Fill in the missing `if` and `elif` statements with something sensible (there is no one right answer)!\n",
    "\n",
    "Start with something simple; we'll build it into something more complicated later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def classify_rb(tweet):\n",
    "    \n",
    "  tweet = str(tweet).lower() # this makes the tweet lower-case, so we don't have to worry about matching case\n",
    "\n",
    "  if ________:\n",
    "    return \"Medical\"\n",
    "  elif ________:\n",
    "    return \"Energy\"\n",
    "  elif ________:\n",
    "    return \"Water\"\n",
    "  elif ________:\n",
    "    return \"Food\"\n",
    "  else:\n",
    "    return \"None\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test your rule-based classifier on some examples\n",
    "\n",
    "Run the cell below to see the results of your rule-based classifier. \n",
    "You should see a table showing each tweet, along with its true category and the category predicted by your system.\n",
    "\n",
    "Which types of tweets does your system get right? Which types of tweets does your system get wrong and why? \n",
    "\n",
    "How would you measure the accuracy of your system?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = [(tweet, classify_rb(tweet)) for tweet in test_tweets] # a list of (tweet, prediction) pairs\n",
    "\n",
    "lib.show_predictions(predictions, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Break your rule-based classifier!\n",
    "\n",
    "It's time to FOOL THE RULES!\n",
    "\n",
    "You'll be deliberately trying to break each others' rule-based classifiers by writing tricky tweets that fool your neighbor's rule-based classifier. Once your own classifier has been fooled by a tricky tweet, it's your job to amend the rules in your classifier to account for the new case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_rb_game(tweet):\n",
    "    \n",
    "    # Copy to here the body of the function classify_rb from above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet about Food that will be misclassified\n",
    "\n",
    "Below, write a disaster-scenario tweet about Food that the classification function above will get wrong (i.e. fail to recognize it's about food).\n",
    "\n",
    "Hint: think of less-obvious food-related keywords that aren't included in the rule-based system above.\n",
    "\n",
    "Then run the cell - make sure the tweet is classified as something other than Food!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(food_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet about Energy that will be misclassified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "energy_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(energy_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet about Water that will be misclassified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "water_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(water_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet about Medical that will be misclassified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "medical_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(medical_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet NOT about Food, that will be falsely classified as Food\n",
    "\n",
    "Below, write a disaster-scenario tweet that is NOT about Food, but that the classifier above will classify as Food.\n",
    "\n",
    "Hint: you want to trick the classifier into thinking you're talking about food when you're not. Look at the keywords the rule-based system associates with food. Can you find a way to use them while actually talking about not-food? \n",
    "\n",
    "* For example, if the system looks for the word \"food\" you could write *\"Waiting out #Sandy by reading Plato. Food for thought.\"* \n",
    "* If the system looks for the word \"cook\", you could write *\"I hear the power's out in Cook County.\"*\n",
    "* More simply, you could mention food incidentally but the real subject of the tweet is something else e.g. *\"Was out food shopping when I heard about the power outage on the news. Hope everyone's OK.\"*\n",
    "\n",
    "Then run the cell - make sure the tweet is classified as Food!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_food_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(not_food_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet NOT about Energy, that will be falsely classified as Energy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_energy_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(not_energy_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet NOT about Water, that will be falsely classified as Water"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_water_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(not_water_tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write a tweet NOT about Medical, that will be falsely classified as Medical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "not_medical_tweet = \"\"\n",
    "print \"This tweet is classified as: %s\\n\" % classify_rb_game(not_medical_tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}