Data_Wrangling_Project Matthijs - Spotify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Wrangling - Spotify Top 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "In this project we will look at multiple questions regarding the Top 200 Spotify songs. For this we will look at the sentiment of the lyrics and combine this data with other datasets to hopefully derive some interesting conclusions. Every section in this notebok represents a research question regarding this topic.\n",
    "\n",
    "Group members of this project:\n",
    "<ul>\n",
    "    <li>Luuk Kaandorp (2623537)</li>\n",
    "    <li>Lucas de Geus</li>\n",
    "    <li>Ward Pennink</li>\n",
    "    <li>Matthijs Blaauw</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import\n",
    "This section imports every function and package used in the following sections, please run this before running all others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package vader_lexicon to\n",
      "[nltk_data]     C:\\Users\\lucas\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# read neccessary libraries for pandas\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "pd.options.display.max_rows = 20\n",
    "np.set_printoptions(precision = 4, suppress = True)\n",
    "\n",
    "# read neccessary libraries for getting the lyrics\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# read neccessary libraries for sentiment analysis (see cell below for acknowledgement)\n",
    "import nltk\n",
    "nltk.download('vader_lexicon')\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "# read neccessary libraries for text pre-processing\n",
    "import string\n",
    "\n",
    "# read neccessary libraries for language detection\n",
    "from langdetect import detect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgement\n",
    "Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1: Proportion of songs with positive sentiment compared to songs with negative sentiment in the Top 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This question tries to answer whether there are more songs with a positive sentiment in the Top 200 than songs with a negative sentiment. In the code field below we start by getting the weekly Top 200 from the US on Spotify in the week of 09-01-2020 to 16-01-2020."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-2-c4b56d5a8368>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# this reads in the Spotify Top 200 of the US of this week\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mthis_weeks_top200_file\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m\"regional-us-weekly-latest.csv\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mthis_weeks_top200\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mthis_weeks_top200_file\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheader\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[0mthis_weeks_top200\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mthis_weeks_top200\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mthis_weeks_top200\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "# this reads in the Spotify Top 200 of the US of this week\n",
    "this_weeks_top200_file = \"regional-us-weekly-latest.csv\"\n",
    "this_weeks_top200 = pd.read_csv(this_weeks_top200_file, header=1)\n",
    "this_weeks_top200 = pd.DataFrame(this_weeks_top200.head(50))\n",
    "this_weeks_top200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we got the songs we need, we need to establish a function which gets the lyrics for the given song. We use the Genius API to get the lyrics. We also need a function which processes the result. The last function we need is the one that scrapes the lyrics from the html page we get from the request. The below code section contains these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'  Chorus Pullin out the coupe at the lot Told em fuck 12 fuck SWAT Bustin all the bales out the box I just hit a lick with the box Had to put the stick in a box mmh Pour up the whole damn seal Ima get lazy I got the mojo deals we been trappin like the 80s She sucked a nigga soul got the Cash App Told em wipe a nigga nose say slatt slatt I wont never sell my soul and I can back that And I really wanna know where you at at  Verse 1 I was out back where the stash at Cruise the city in a bulletproof Cadillac Skrrt Cause I know these niggas after where the bag at Yeah Gotta move smarter gotta move harder Nigga try to get me for my water Ill lay his ass down on my son on my daughter I had the Draco with me Dwayne Carter Lotta niggas out here playin aint ballin I done put my whole arm in the rim Vince Carter Yeah And I know probably get a key for the quarter Shawty barely seen in double Cs I bought em Got a bitch thats looking like Aaliyah she a model I got the pink slip all my whips is keyless Compton Im about to get the key to the city Patek like the sea forgive me  Chorus Pullin out the coupe at the lot Told em fuck 12 fuck SWAT Bustin all the bales out the box I just hit a lick with the box Had to put the stick in a box mmh Pour up the whole damn seal Ima get lazy I got the mojo deals we been trappin like the 80s She sucked a nigga soul got the Cash App Told em wipe a nigga nose say slatt slatt I wont never sell my soul and I can back that And I really wanna know where you at at hah  Verse 2 Hahaha I been movin em out If Steelo with me then he got the blues in the pouch Yeah Took her to the forest put the wood in her mouth Bitch dont wear no shoes in my house The private Im flyin in I never wan fly again Ill take my chances in traffic Yeah She suckin on dick no hands with it I just made the Rollie plain like a landing strip Im a 2020 president candidate I done put a hundred bands on Zimmerman shit I been movin real gangster so thats why she picked a Crip Shawty call me Crisco cause I pop my shit Got it out the mud theres nothin you can tell me yeah When I had the drugs I was street wealthy yeah  Chorus Pullin out the coupe at the lot Told em fuck 12 fuck SWAT Bustin all the bales out the box I just hit a lick with the box Had to put the stick in a box mmh Pour up the whole damn seal Ima get lazy I got the mojo deals we been trappin like the 80s She sucked a nigga soul got the Cash App Told em wipe a nigga nose say slatt slatt I wont never sell my soul and I can back that And I really wanna know where you at at  '"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this function requests the song info based on the artist and song title\n",
    "def request_song_info(song_title, artist_name):\n",
    "    base_url = 'https://api.genius.com'\n",
    "    headers = {'Authorization': 'Bearer ' + '2wN0egWOzQ-_KQDN4XkblxZoUjG2H_Zl-xq3uXNJhVFNaIkaM0QfvSWM4pm3fWhw'}\n",
    "    search_url = base_url + '/search'\n",
    "    data = {'q': song_title + ' ' + artist_name}\n",
    "    response = requests.get(search_url, data=data, headers=headers)\n",
    "\n",
    "    return response\n",
    "\n",
    "# this function processes the result of the info request\n",
    "def process_request(response, artist_name):\n",
    "    json = response.json()\n",
    "    remote_song_info = None\n",
    "\n",
    "    for hit in json['response']['hits']:\n",
    "        if artist_name.lower() in hit['result']['primary_artist']['name'].lower():\n",
    "            remote_song_info = hit\n",
    "            break\n",
    "            \n",
    "    if remote_song_info:\n",
    "        return remote_song_info['result']['url']\n",
    "    else :\n",
    "        return \"\"\n",
    "    \n",
    "# this function scraps the lyrics from the html page\n",
    "def scrap_lyrics(url):\n",
    "    if url != \"\":\n",
    "        page = requests.get(url)\n",
    "        html = BeautifulSoup(page.text, 'html.parser')\n",
    "        lyrics = html.find('div', class_='lyrics').get_text()\n",
    "        # we detect the language of the lyrics, since the function that comes later in this document can only\n",
    "        # calculate the sentiment of english lyrics we won't process lyrics that are not english\n",
    "        if detect(lyrics) == \"en\":\n",
    "            return process_lyrics(lyrics)\n",
    "    return \"\"\n",
    "# this function pre-processes the lyrics\n",
    "def process_lyrics(lyrics):\n",
    "    # replace enter (\\n) with space\n",
    "    enter_removed = lyrics.replace('\\u2005', ' ').replace('\\n', ' ')\n",
    "    \n",
    "    # remove punctuatin\n",
    "    punctuation_removed = enter_removed.translate(str.maketrans(\"\",\"\", string.punctuation)) \n",
    "    \n",
    "    return punctuation_removed\n",
    "            \n",
    "# TEST\n",
    "test_lyrics = scrap_lyrics(process_request(request_song_info('The Box', 'Roddy Ricch'), 'Roddy Ricch'))\n",
    "test_lyrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the correct functions, we can get the lyrics for all the songs in the Top200. All we need now is a function which determines the sentiment of the lyrics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'neg': 0.133, 'neu': 0.79, 'pos': 0.077, 'compound': -0.9896}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this function calculates the sentiment of the lyrics\n",
    "def sentiment_of_lyrics(lyrics):\n",
    "    if lyrics != \"\":\n",
    "        sid = SentimentIntensityAnalyzer()\n",
    "        return sid.polarity_scores(lyrics)\n",
    "    else:\n",
    "        return False\n",
    "\n",
    "# TEST\n",
    "sentiment_of_lyrics(test_lyrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have everything we need: the songs, the lyrics and the sentiment processor. In the following section we create the final table containing the songs and their sentiment. Note that creating this table can take up to 10 minutes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Position</th>\n",
       "      <th>Track Name</th>\n",
       "      <th>Artist</th>\n",
       "      <th>Streams</th>\n",
       "      <th>URL</th>\n",
       "      <th>neg_sentiment</th>\n",
       "      <th>neu_sentiment</th>\n",
       "      <th>pos_sentiment</th>\n",
       "      <th>compound_sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>The Box</td>\n",
       "      <td>Roddy Ricch</td>\n",
       "      <td>18952305</td>\n",
       "      <td>https://open.spotify.com/track/0nbXyq5TXYPCO7p...</td>\n",
       "      <td>0.133</td>\n",
       "      <td>0.790</td>\n",
       "      <td>0.077</td>\n",
       "      <td>-0.9896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ROXANNE</td>\n",
       "      <td>Arizona Zervas</td>\n",
       "      <td>9671478</td>\n",
       "      <td>https://open.spotify.com/track/696DnlkuDOXcMAn...</td>\n",
       "      <td>0.093</td>\n",
       "      <td>0.634</td>\n",
       "      <td>0.273</td>\n",
       "      <td>0.9979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Yummy</td>\n",
       "      <td>Justin Bieber</td>\n",
       "      <td>9648561</td>\n",
       "      <td>https://open.spotify.com/track/41L3O37CECZt3N7...</td>\n",
       "      <td>0.023</td>\n",
       "      <td>0.722</td>\n",
       "      <td>0.255</td>\n",
       "      <td>0.9984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Circles</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>8244725</td>\n",
       "      <td>https://open.spotify.com/track/21jGcNKet2qwijl...</td>\n",
       "      <td>0.049</td>\n",
       "      <td>0.846</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.9630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>BOP</td>\n",
       "      <td>DaBaby</td>\n",
       "      <td>7985170</td>\n",
       "      <td>https://open.spotify.com/track/6Ozh9Ok6h4Oi1wU...</td>\n",
       "      <td>0.123</td>\n",
       "      <td>0.751</td>\n",
       "      <td>0.126</td>\n",
       "      <td>-0.8024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>46</td>\n",
       "      <td>Señorita</td>\n",
       "      <td>Shawn Mendes</td>\n",
       "      <td>3289924</td>\n",
       "      <td>https://open.spotify.com/track/6v3KW9xbzN5yKLt...</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.744</td>\n",
       "      <td>0.226</td>\n",
       "      <td>0.9979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>47</td>\n",
       "      <td>Goodbyes (Feat. Young Thug)</td>\n",
       "      <td>Post Malone</td>\n",
       "      <td>3266958</td>\n",
       "      <td>https://open.spotify.com/track/6vBdBCoOhKHiYDD...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>48</td>\n",
       "      <td>No Guidance (feat. Drake)</td>\n",
       "      <td>Chris Brown</td>\n",
       "      <td>3216358</td>\n",
       "      <td>https://open.spotify.com/track/6XHVuErjQ4XNm6n...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>49</td>\n",
       "      <td>Death</td>\n",
       "      <td>Trippie Redd</td>\n",
       "      <td>3176458</td>\n",
       "      <td>https://open.spotify.com/track/3Kd6Wp3lG5HXMB2...</td>\n",
       "      <td>0.221</td>\n",
       "      <td>0.616</td>\n",
       "      <td>0.163</td>\n",
       "      <td>-0.9966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>50</td>\n",
       "      <td>Start Wit Me (feat. Gunna)</td>\n",
       "      <td>Roddy Ricch</td>\n",
       "      <td>3156095</td>\n",
       "      <td>https://open.spotify.com/track/7A20WnLkjtZU2d7...</td>\n",
       "      <td>0.155</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.084</td>\n",
       "      <td>-0.9925</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Position                   Track Name          Artist   Streams  \\\n",
       "0          1                      The Box     Roddy Ricch  18952305   \n",
       "1          2                      ROXANNE  Arizona Zervas   9671478   \n",
       "2          3                        Yummy   Justin Bieber   9648561   \n",
       "3          4                      Circles     Post Malone   8244725   \n",
       "4          5                          BOP          DaBaby   7985170   \n",
       "..       ...                          ...             ...       ...   \n",
       "45        46                     Señorita    Shawn Mendes   3289924   \n",
       "46        47  Goodbyes (Feat. Young Thug)     Post Malone   3266958   \n",
       "47        48    No Guidance (feat. Drake)     Chris Brown   3216358   \n",
       "48        49                        Death    Trippie Redd   3176458   \n",
       "49        50   Start Wit Me (feat. Gunna)     Roddy Ricch   3156095   \n",
       "\n",
       "                                                  URL  neg_sentiment  \\\n",
       "0   https://open.spotify.com/track/0nbXyq5TXYPCO7p...          0.133   \n",
       "1   https://open.spotify.com/track/696DnlkuDOXcMAn...          0.093   \n",
       "2   https://open.spotify.com/track/41L3O37CECZt3N7...          0.023   \n",
       "3   https://open.spotify.com/track/21jGcNKet2qwijl...          0.049   \n",
       "4   https://open.spotify.com/track/6Ozh9Ok6h4Oi1wU...          0.123   \n",
       "..                                                ...            ...   \n",
       "45  https://open.spotify.com/track/6v3KW9xbzN5yKLt...          0.030   \n",
       "46  https://open.spotify.com/track/6vBdBCoOhKHiYDD...          0.000   \n",
       "47  https://open.spotify.com/track/6XHVuErjQ4XNm6n...          0.000   \n",
       "48  https://open.spotify.com/track/3Kd6Wp3lG5HXMB2...          0.221   \n",
       "49  https://open.spotify.com/track/7A20WnLkjtZU2d7...          0.155   \n",
       "\n",
       "    neu_sentiment  pos_sentiment  compound_sentiment  \n",
       "0           0.790          0.077             -0.9896  \n",
       "1           0.634          0.273              0.9979  \n",
       "2           0.722          0.255              0.9984  \n",
       "3           0.846          0.104              0.9630  \n",
       "4           0.751          0.126             -0.8024  \n",
       "..            ...            ...                 ...  \n",
       "45          0.744          0.226              0.9979  \n",
       "46          0.000          0.000              0.0000  \n",
       "47          0.000          0.000              0.0000  \n",
       "48          0.616          0.163             -0.9966  \n",
       "49          0.762          0.084             -0.9925  \n",
       "\n",
       "[50 rows x 9 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate sentiment for all songs in a Top200\n",
    "def sentiment_for_top200(top200):\n",
    "    for index, row in top200.iterrows():\n",
    "        sentiment = sentiment_of_lyrics(scrap_lyrics(process_request(request_song_info(row['Track Name'], row['Artist']), row['Artist'])))\n",
    "        if (sentiment != False):\n",
    "            top200.loc[index,'neg_sentiment'] = sentiment.get('neg')\n",
    "            top200.loc[index,'neu_sentiment'] = sentiment.get('neu')\n",
    "            top200.loc[index,'pos_sentiment'] = sentiment.get('pos')\n",
    "            top200.loc[index,'compound_sentiment'] = sentiment.get('compound')\n",
    "        else:\n",
    "            top200.loc[index,'neg_sentiment'] = 0\n",
    "            top200.loc[index,'neu_sentiment'] = 0\n",
    "            top200.loc[index,'pos_sentiment'] = 0\n",
    "            top200.loc[index,'compound_sentiment'] = 0\n",
    "    return top200\n",
    "\n",
    "processed_this_weeks_top200 = sentiment_for_top200(this_weeks_top200)\n",
    "processed_this_weeks_top200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the table containing the top 200 songs on Spotify with their corresponding sentiment based on their lyrics. We will also use this table in further sections. We will now look at the proportion of negative and positive lyrics based on their compound sentiment to answer our question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Songs with positive sentiment: 23\n",
      "Songs with negative sentiment: 27\n"
     ]
    }
   ],
   "source": [
    "# the number of positive sentiment lyrics\n",
    "this_weeks_compound_sentiment = processed_this_weeks_top200['compound_sentiment']\n",
    "this_weeks_num_positive = this_weeks_compound_sentiment[this_weeks_compound_sentiment > 0.5 ].count()\n",
    "\n",
    "# the number of negative sentiment lyrics\n",
    "this_weeks_num_negative = this_weeks_compound_sentiment[this_weeks_compound_sentiment < 0.5 ].count()\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(this_weeks_num_positive))\n",
    "print(\"Songs with negative sentiment: \" + str(this_weeks_num_negative))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to not have to repeat the code above every time we want to compute the compound sentiment for a week, we wrap the above code in a function, called compound_sentiment. This function takes the file for which the compound sentiment needs to be calculated as parameter, and returns a tuple containing the number of songs with a positive sentiment and the number of songs with a negative sentiment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compound_sentiment_function(file):    \n",
    "    top50 = pd.DataFrame(pd.read_csv(file, header=1).head(50))\n",
    "\n",
    "    # creates augmented top50 table with sentiment\n",
    "    processed_top50 = sentiment_for_top200(top50)\n",
    "    top50_compound_sentiment = processed_top50['compound_sentiment']\n",
    "    \n",
    "    # the number of positive sentiment lyrics    \n",
    "    top50_num_positive = top50_compound_sentiment[top50_compound_sentiment > 0.5 ].count()\n",
    "\n",
    "    # the number of negative sentiment lyrics\n",
    "    top50_num_negative = top50_compound_sentiment[top50_compound_sentiment < 0.5 ].count()\n",
    "    \n",
    "    return (top50_num_positive, top50_num_negative)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the result, there is only a small difference between the positive and negative sentiment. We consider this difference to be negligable. Therefore we decided to also look at the 2 weeks before this week to see if this holds for multiple weeks or was just coincidence. In the below code field we look at the week before (02-01-2020 to 09-01-2020). Note that creating this table can also take up to 10 minutes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Songs with positive sentiment: 22\n",
      "Songs with negative sentiment: 28\n"
     ]
    }
   ],
   "source": [
    "# this reads in the Spotify Top 200 of the US of the week before and calculates the sentiment\n",
    "compound_sentiment_week_before = compound_sentiment_function(\"regional-us-weekly-2019-12-27--2020-01-03.csv\")\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(compound_sentiment_week_before[0]))\n",
    "print(\"Songs with negative sentiment: \" + str(compound_sentiment_week_before[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following code field we look at the week 2 weeks before this week (26-12-2019 to 02-01-2020). Note that creating this table can also take up to 10 minutes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Songs with positive sentiment: 29\n",
      "Songs with negative sentiment: 21\n"
     ]
    }
   ],
   "source": [
    "# this reads in the Spotify Top 200 of the US of 2 weeks before\n",
    "compound_sentiment_2weeks = compound_sentiment_function(\"regional-us-weekly-2019-12-20--2019-12-27.csv\")\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(compound_sentiment_2weeks[0]))\n",
    "print(\"Songs with negative sentiment: \" + str(compound_sentiment_2weeks[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, 2 weeks before was around Christmas time and it had a lot more songs in the top 200 with a positive sentiment than songs with negative sentiment. We know from the numbers that the top 200 tends to contain more songs with positive sentiment than songs with negative sentiment. In the next code field we calculate the difference in percentage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In the last 3 weeks, we see that there are -2.63% more songs with positive sentiment in their lyrics in the Top200 than negative.\n"
     ]
    }
   ],
   "source": [
    "# we calculate the total number of positive and negative songs\n",
    "total_positive_all_weeks = this_weeks_num_positive + compound_sentiment_week_before[0] + compound_sentiment_2weeks[0]\n",
    "total_negative_all_weeks = this_weeks_num_negative + compound_sentiment_week_before[1] + compound_sentiment_2weeks[1]\n",
    "\n",
    "# we calculate the difference between positive and negative songs in percentage\n",
    "difference = ((total_positive_all_weeks - total_negative_all_weeks) / total_negative_all_weeks) * 100\n",
    "print(\"In the last 3 weeks, we see that there are \" + str(round(difference, 2)) + \"% more songs with positive sentiment in their lyrics in the Top200 than negative.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our conclusion is that it seems like there are not notably more songs with a positive sentiment than ones with negative sentiments in 'normal' weeks without holidays based on the last 3 weeks. We saw that the week around Christmas was an exception to this conclusion and therefore we investigate this further in the following section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q2: Influence of holidays on the sentiment listened to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This question tries to answer whether people tend to listen more to songs with a positive sentiment than songs with a negative sentiment during weeks around holidays. For this question we will look at the 2 weeks around the holidays Christmas, Easter and Thanksgiving. Because we have already looked at the Christmas period for 2019, this question looks at the holiday periods of 2018. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Songs with positive sentiment: 30\n",
      "Songs with negative sentiment: 20\n"
     ]
    }
   ],
   "source": [
    "christmas2018 = compound_sentiment_function(\"regional-us-weekly-2018-12-21--2018-12-28.csv\")\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(christmas2018[0]))\n",
    "print(\"Songs with negative sentiment: \" + str(christmas2018[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Songs with positive sentiment: 25\n",
      "Songs with negative sentiment: 25\n"
     ]
    }
   ],
   "source": [
    "easter2018 = compound_sentiment_function(\"regional-us-weekly-2018-03-30--2018-04-06.csv\")\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(easter2018[0]))\n",
    "print(\"Songs with negative sentiment: \" + str(easter2018[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thanksgiving2018 = compound_sentiment_function(\"regional-us-weekly-2018-11-16--2018-11-23.csv\")\n",
    "\n",
    "print(\"Songs with positive sentiment: \" + str(thanksgiving2018[0]))\n",
    "print(\"Songs with negative sentiment: \" + str(thanksgiving2018[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import csv\n",
    "\n",
    "weekLists = [f for f in listdir('dataUs') if isfile(join('dataUs', f))]\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sentiments = []\n",
    "postive_songs = []\n",
    "negative_songs = []\n",
    "for file in weekLists:\n",
    "    avg_sentiment = float(0)\n",
    "    num_records = 0\n",
    "    num_postive_songs = 0\n",
    "    num_negative_songs = 0\n",
    "    with open('dataUs/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            # compound sentiment\n",
    "            if row[7] != 'none':\n",
    "                num_records += 1\n",
    "                avg_sentiment += float(row[7])\n",
    "            # positive sentiment\n",
    "            if row[6] != 'none' and float(row[6]) > 0.3:\n",
    "                num_postive_songs += 1\n",
    "            if row[4] != 'none' and float(row[4]) > 0.3:\n",
    "                num_negative_songs += 1\n",
    "    avg_sentiment = avg_sentiment / num_records\n",
    "    sentiments.append(avg_sentiment)\n",
    "    postive_songs.append(num_postive_songs)\n",
    "    negative_songs.append(num_negative_songs)\n",
    "\n",
    "plt.plot(sentiments)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(postive_songs)\n",
    "plt.plot(negative_songs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "positions_sentiments = []\n",
    "\n",
    "for i in range(0, 200):\n",
    "    positions_sentiments.append([])\n",
    "\n",
    "print(len(positions_sentiments))\n",
    "\n",
    "for file in weekLists:\n",
    "    avg_sentiment = float(0)\n",
    "    with open('dataUs/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            # compound sentiment\n",
    "            if row[7] != 'none':\n",
    "                positions_sentiments[index - 1].append(float(row[7]))\n",
    "\n",
    "position_avg_sentiments = []\n",
    "\n",
    "for index, row in enumerate(positions_sentiments):\n",
    "    position_avg_sentiments.append(sum(row) / float(len(row)))\n",
    "\n",
    "plt.plot(position_avg_sentiments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weekLists = [f for f in listdir('dataUs') if isfile(join('dataUs', f))]\n",
    "\n",
    "usSentiments = []\n",
    "ukSentiments = []\n",
    "auSentiemnts = []\n",
    "for file in weekLists:\n",
    "    avg_sentiment = float(0)\n",
    "    num_records = 0\n",
    "\n",
    "    with open('dataUs/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            # compound sentiment\n",
    "            if row[7] != 'none':\n",
    "                num_records += 1\n",
    "                avg_sentiment += float(row[7])\n",
    "    avg_sentiment = avg_sentiment / num_records\n",
    "    usSentiments.append(avg_sentiment)\n",
    "    \n",
    "    with open('dataGb/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            # compound sentiment\n",
    "            if row[7] != 'none':\n",
    "                num_records += 1\n",
    "                avg_sentiment += float(row[7])\n",
    "    avg_sentiment = avg_sentiment / num_records\n",
    "    ukSentiments.append(avg_sentiment)\n",
    "    \n",
    "    with open('dataAu/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            # compound sentiment\n",
    "            if row[7] != 'none':\n",
    "                num_records += 1\n",
    "                avg_sentiment += float(row[7])\n",
    "    avg_sentiment = avg_sentiment / num_records\n",
    "    auSentiemnts.append(avg_sentiment)\n",
    "\n",
    "plt.plot(usSentiments)\n",
    "plt.plot(ukSentiments)\n",
    "plt.plot(auSentiemnts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "days = [f for f in listdir('dataGbDay') if isfile(join('dataGbDay', f))]\n",
    "\n",
    "totalStreamsPerDay = []\n",
    "totalPostiveStreamsPerDay = []\n",
    "totalNegativeStreamsPerDay = []\n",
    "for file in days:\n",
    "    \n",
    "    with open('dataGbDay/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        tempStreams = 0\n",
    "        tempPosStreams = 0\n",
    "        tempNegStreams = 0\n",
    "        reader = csv.reader(csvfile)\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            if row[6] != 'none' and float(row[6]) > 0.3:\n",
    "                tempPosStreams += float(row[2])\n",
    "            if row[4] != 'none' and float(row[4]) > 0.3:\n",
    "                tempNegStreams += float(row[2])\n",
    "            tempStreams += float(row[2])\n",
    "    totalStreamsPerDay.append(tempStreams)\n",
    "    totalPostiveStreamsPerDay.append(tempPosStreams)\n",
    "    totalNegativeStreamsPerDay.append(tempNegStreams)\n",
    "\n",
    "plt.plot(totalStreamsPerDay)\n",
    "# plt.plot(totalPostiveStreamsPerDay)\n",
    "# plt.plot(totalNegativeStreamsPerDay)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "days = [f for f in listdir('dataGbDay') if isfile(join('dataGbDay', f))]\n",
    "weekdays = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\n",
    "\n",
    "streamsPerWeekday = [0, 0, 0, 0, 0, 0, 0]\n",
    "for file in days:\n",
    "    with open('dataGbDay/' + file, newline='', encoding=\"utf-8\") as csvfile:\n",
    "        reader = csv.reader(csvfile)\n",
    "        totalStreams = 0\n",
    "        for index, row in enumerate(reader):\n",
    "            if (index == 0):\n",
    "                continue\n",
    "            totalStreams += float(row[2])\n",
    "        date = datetime.strptime(file.split('.')[0], '%Y-%m-%d')\n",
    "        streamsPerWeekday[weekdays.index(date.strftime(\"%A\"))] += totalStreams\n",
    "        if (date.strftime(\"%A\") == 'Monday'):\n",
    "            plt.plot(streamsPerWeekday)\n",
    "            streamsPerWeekday = [0, 0, 0, 0, 0, 0, 0]\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'setup' is not recognized as an internal or external command,\n",
      "operable program or batch file.\n"
     ]
    }
   ],
   "source": [
    "!setup install"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}