Projects/step_1/C2/Exploring+Hacker+News+Posts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Exploring Hacker News Posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### In this project, we'll work with a data set of submissions to popular technology site Hacker News.\n",
    "#### We're specifically interested in posts whose titles begin with either Ask HN or Show HN. Users submit Ask HN posts to ask the Hacker News community a specific question.\n",
    "We'll compare these two types of posts to determine the following:\n",
    "\n",
    "* Do Ask HN or Show HN receive more comments on average?\n",
    "* Do posts created at a certain time receive more comments on average?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hacker News is a site started by the startup incubator Y Combinator, where user-submitted stories (known as \"posts\") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles, and posts that make it to the top of Hacker News' listings can get hundreds of thousands of visitors as a result.\n",
    "\n",
    "You can find the data set [here](https://www.kaggle.com/hacker-news/hacker-news-posts), but note that it has been reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]\n"
     ]
    }
   ],
   "source": [
    "from csv import reader\n",
    "\n",
    "hn=list(reader(open('hacker_news.csv')))\n",
    "print(hn[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Removing Headers from a List of Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']]\n",
      "[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]\n"
     ]
    }
   ],
   "source": [
    "headers=hn[:1]\n",
    "hn=hn[1:]\n",
    "print(headers)\n",
    "print(hn[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Extracting Ask HN and Show HN Posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've removed the headers from hn, we're ready to filter our data. Since we're only concerned with post titles beginning with Ask HN or Show HN, we'll create new lists of lists containing just the data for those titles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the posts that begin with either Ask HN or Show HN, we'll use the string method startswith\n",
    "`string.startswith('pattern')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# empty list to store 'Ask HN' posts\n",
    "ask_posts=[]\n",
    "\n",
    "# empty list to store 'Show HN' posts\n",
    "show_posts=[]\n",
    "\n",
    "# empty list to store other posts than 'Ask HN' and 'Show HN'\n",
    "other_posts=[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# loop over hn dataset \n",
    "for row in hn:\n",
    "    title=row[1]\n",
    "    if title.lower().startswith('ask hn'):\n",
    "        ask_posts.append(row)\n",
    "    elif title.lower().startswith('show hn'):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1744\n",
      "1162\n",
      "17194\n"
     ]
    }
   ],
   "source": [
    "# no posts in ask_posts\n",
    "print(len(ask_posts))\n",
    "\n",
    "#no of posts in show_posts\n",
    "print(len(show_posts))\n",
    "\n",
    "#no of posts in otherr_posts\n",
    "print(len(other_posts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['12296411', 'Ask HN: How to improve my personal website?', '', '2', '6', 'ahmedbaracat', '8/16/2016 9:55'], ['10610020', 'Ask HN: Am I the only one outraged by Twitter shutting down share counts?', '', '28', '29', 'tkfx', '11/22/2015 13:43'], ['11610310', 'Ask HN: Aby recent changes to CSS that broke mobile?', '', '1', '1', 'polskibus', '5/2/2016 10:14'], ['12210105', 'Ask HN: Looking for Employee #3 How do I do it?', '', '1', '3', 'sph130', '8/2/2016 14:20'], ['10394168', 'Ask HN: Someone offered to buy my browser extension from me. What now?', '', '28', '17', 'roykolak', '10/15/2015 16:38']]\n"
     ]
    }
   ],
   "source": [
    "print(ask_posts[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['10627194', 'Show HN: Wio Link  ESP8266 Based Web of Things Hardware Development Platform', 'https://iot.seeed.cc', '26', '22', 'kfihihc', '11/25/2015 14:03'], ['10646440', 'Show HN: Something pointless I made', 'http://dn.ht/picklecat/', '747', '102', 'dhotson', '11/29/2015 22:46'], ['11590768', 'Show HN: Shanhu.io, a programming playground powered by e8vm', 'https://shanhu.io', '1', '1', 'h8liu', '4/28/2016 18:05'], ['12178806', 'Show HN: Webscope  Easy way for web developers to communicate with Clients', 'http://webscopeapp.com', '3', '3', 'fastbrick', '7/28/2016 7:11'], ['10872799', 'Show HN: GeoScreenshot  Easily test Geo-IP based web pages', 'https://www.geoscreenshot.com/', '1', '9', 'kpsychwave', '1/9/2016 20:45']]\n"
     ]
    }
   ],
   "source": [
    "print(show_posts[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Calculating the Average Number of Comments for Ask HN and Show HN Posts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# find total no of comments in ask posts \n",
    "\n",
    "total_ask_comments=0\n",
    "\n",
    "for row in ask_posts:\n",
    "    # convert no of comments to int from string\n",
    "    n_comment=int(row[4])\n",
    "    total_ask_comments+=n_comment\n",
    "\n",
    "avg_ask_comments=total_ask_comments/len(ask_posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.038417431192661\n"
     ]
    }
   ],
   "source": [
    "print(avg_ask_comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# find tottal no of comments in show posts\n",
    "\n",
    "total_show_comments=0\n",
    "\n",
    "for row in show_posts:\n",
    "    n_comment=int(row[4])\n",
    "    total_show_comments+=n_comment\n",
    "    \n",
    "avg_show_comments=total_show_comments/len(show_posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.31669535283993\n"
     ]
    }
   ],
   "source": [
    "print(avg_show_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation:\n",
    "ask posts receieve more average comments than show posts\n",
    "Since ask posts are more likely to receive comments, we'll focus our remaining analysis just on these posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Finding the Amount of Ask Posts and Comments by Hour Created"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll determine if ask posts created at a certain time are more likely to attract comments. We'll use the following steps to perform this analysis:\n",
    "\n",
    "* Calculate the amount of ask posts created in each hour of the day, along with the number of comments received.\n",
    "* Calculate the average number of comments ask posts receive by hour created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#import datetime module to work with date and time data\n",
    "import datetime as dt\n",
    "\n",
    "# empty result_list will store time and no of comments as list of list\n",
    "result_list=[]\n",
    "\n",
    "for row in ask_posts:\n",
    "    time=row[-1]\n",
    "    n_comments=int(row[-3])\n",
    "    result_list.append([time,n_comments])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1], ['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17]]\n"
     ]
    }
   ],
   "source": [
    "print(result_list[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'00': 447,\n",
       " '01': 683,\n",
       " '02': 1381,\n",
       " '03': 421,\n",
       " '04': 337,\n",
       " '05': 464,\n",
       " '06': 397,\n",
       " '07': 267,\n",
       " '08': 492,\n",
       " '09': 251,\n",
       " '10': 793,\n",
       " '11': 641,\n",
       " '12': 687,\n",
       " '13': 1253,\n",
       " '14': 1416,\n",
       " '15': 4477,\n",
       " '16': 1814,\n",
       " '17': 1146,\n",
       " '18': 1439,\n",
       " '19': 1188,\n",
       " '20': 1722,\n",
       " '21': 1745,\n",
       " '22': 479,\n",
       " '23': 543}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# contains the number of ask posts created during each hour of the day.\n",
    "counts_by_hour={}\n",
    "\n",
    "#contains the corresponding number of comments ask posts created at each hour received.\n",
    "comments_by_hour={}\n",
    "\n",
    "date_format = \"%m/%d/%Y %H:%M\"\n",
    "\n",
    "for each_row in result_list:\n",
    "    date = each_row[0]\n",
    "    comment = each_row[1]\n",
    "    time = dt.datetime.strptime(date, date_format).strftime(\"%H\")\n",
    "    if time in counts_by_hour:\n",
    "        comments_by_hour[time] += comment\n",
    "        counts_by_hour[time] += 1\n",
    "    else:\n",
    "        comments_by_hour[time] = comment\n",
    "        counts_by_hour[time] = 1\n",
    "\n",
    "comments_by_hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Calculating the Average Number of Comments for Ask HN Posts by Hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['14', 13.233644859813085],\n",
       " ['10', 13.440677966101696],\n",
       " ['12', 9.41095890410959],\n",
       " ['13', 14.741176470588234],\n",
       " ['17', 11.46],\n",
       " ['07', 7.852941176470588],\n",
       " ['03', 7.796296296296297],\n",
       " ['21', 16.009174311926607],\n",
       " ['22', 6.746478873239437],\n",
       " ['05', 10.08695652173913],\n",
       " ['09', 5.5777777777777775],\n",
       " ['11', 11.051724137931034],\n",
       " ['23', 7.985294117647059],\n",
       " ['02', 23.810344827586206],\n",
       " ['20', 21.525],\n",
       " ['04', 7.170212765957447],\n",
       " ['18', 13.20183486238532],\n",
       " ['06', 9.022727272727273],\n",
       " ['16', 16.796296296296298],\n",
       " ['19', 10.8],\n",
       " ['08', 10.25],\n",
       " ['01', 11.383333333333333],\n",
       " ['00', 8.127272727272727],\n",
       " ['15', 38.5948275862069]]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create empty list to store hour and corresponding avg no of comments per post for crresponding hour as list of list\n",
    "avg_by_hour=[]\n",
    "\n",
    "for hr in comments_by_hour:\n",
    "    avg_by_hour.append([hr, comments_by_hour[hr] / counts_by_hour[hr]])\n",
    "\n",
    "avg_by_hour \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Sorting and Printing Values from a List of Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[13.233644859813085, '14'],\n",
       " [13.440677966101696, '10'],\n",
       " [9.41095890410959, '12'],\n",
       " [14.741176470588234, '13'],\n",
       " [11.46, '17']]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# swap columns in avg_by_hour\n",
    "swap_avg_by_hour=[]\n",
    "\n",
    "for row in avg_by_hour:\n",
    "    swap_avg_by_hour.append([row[1],row[0]])\n",
    "    \n",
    "swap_avg_by_hour[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[38.5948275862069, '15'],\n",
       " [23.810344827586206, '02'],\n",
       " [21.525, '20'],\n",
       " [16.796296296296298, '16'],\n",
       " [16.009174311926607, '21']]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort swap_avg_by_hour list \n",
    "\n",
    "sorted_swap=sorted(swap_avg_by_hour,reverse=True)\n",
    "\n",
    "sorted_swap[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 Hours for 'Ask HN' Comments\n",
      "15:00: 38.59 average comments per post\n",
      "02:00: 23.81 average comments per post\n",
      "20:00: 21.52 average comments per post\n",
      "16:00: 16.80 average comments per post\n",
      "21:00: 16.01 average comments per post\n"
     ]
    }
   ],
   "source": [
    "print(\"Top 5 Hours for 'Ask HN' Comments\")\n",
    "for avg, hr in sorted_swap[:5]:\n",
    "    print(\n",
    "        \"{}: {:.2f} average comments per post\".format(\n",
    "            dt.datetime.strptime(hr, \"%H\").strftime(\"%H:%M\"),avg)\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "In this project, we analyzed ask posts and show posts to determine which type of post and time receive the most comments on average. Based on our analysis, to maximize the amount of comments a post receives, we'd recommend the post be categorized as ask post and created between 15:00 and 16:00 (3:00 pm est - 4:00 pm est).\n",
    "\n",
    "However, it should be noted that the data set we analyzed excluded posts without any comments. Given that, it's more accurate to say that of the posts that received comments, ask posts received more comments on average and ask posts created between 15:00 and 16:00 (3:00 pm est - 4:00 pm est) received the most comments on average."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}