notebooks/1_EvenMoreFundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Even More Python Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Notebook settings \n",
    "\n",
    "# multiple lines of output per cell\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">### Today\n",
    ">\n",
    "> - Modules and Packages\n",
    ">\n",
    ">\n",
    "> - The Python Standard Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules and Packages\n",
    "\n",
    "Logically, Python modules are groupings of related code that are structures as to facilitate its re-use. \n",
    "\n",
    "Physically, modules are `.py` files implementing a set of **functions, classes or variables**, as well as **executable statements**, that can be accessed from other modules by using the `import` command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `import` command can be used both to import **the whole code** of a module, using the following syntax:\n",
    "\n",
    "```python\n",
    "import module\n",
    "```\n",
    "\n",
    "or just **specific attributes** (one or more functions, variables, classes or a combination of these) with the following syntax:\n",
    "\n",
    "```python\n",
    "from module import name1, name2, name3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, if order to know what is our current working directory, we can use the function `getcwd()` available `os` module (see below) in two different ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that all the other functions of the module are available as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['11_Sentiment_Analysis.ipynb', '.DS_Store', '7_WordNet.ipynb', '10_WordEmbeddings.ipynb', 'images', '8_Vector_Semantics.ipynb', '0_HelloWorld.ipynb', '3_EvenMoreFundamentals.ipynb', 'stuff', '2_MoreFundamentals.ipynb', '13_Clustering_TopicModelling.ipynb', '4_ScientificProgramming.ipynb', '6_WebScraping_APIs.ipynb', '1_Fundamentals.ipynb', '.ipynb_checkpoints', 'data', '4_RegularExpressions.ipynb', '9_ML.ipynb', '7_Distributions_in_text.ipynb', '5_NLP_pipelines.ipynb', '12_Recommender_Systems.ipynb']\n"
     ]
    }
   ],
   "source": [
    "# os.list(path) returns the names of the entries in the folder \"path\"\n",
    "print(os.listdir(os.getcwd()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important**: use the `dir(module_name)` function to list all the attributes available in a (loaded) module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ALLOW_THREADS', 'AxisError', 'BUFSIZE', 'CLIP', 'ComplexWarning', 'DataSource', 'ERR_CALL', 'ERR_DEFAULT', 'ERR_IGNORE', 'ERR_LOG']\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "print(dir(numpy)[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['In', 'InteractiveShell', 'Out', '_', '_2', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_i', '_i1', '_i2', '_i3', '_i4', '_i5', '_ih', '_ii', '_iii', '_oh', 'exit', 'foo', 'get_ipython', 'numpy', 'os', 'quit']\n"
     ]
    }
   ],
   "source": [
    "# NOTE that Without arguments, `dir()` lists all names (variables, modules, functions etc.) defined currently\n",
    "foo = 10\n",
    "print(dir())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Method 2]: importing only the needed attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from os import getcwd\n",
    "getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important**: Note that the syntax adopted in the import statement influence the way attributes are called:\n",
    "\n",
    "- if the whole module is imported, then the attribute is called as **`module.attribute`** (e.g. `os.getcwd()`)\n",
    "\n",
    "\n",
    "- if only the attribute is imported, than it can be accessed simply as **`attribute`** (e.g. `getcwd()`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Import as [local name]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.array([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Re-loading a module\n",
    "\n",
    "The first time you import a module or a function in a module (i.e. no matter the `import` syntax you adopt), all the code in the module source file is run at once. If another module, or the user in the interactive shell, import the module a second time, **nothing happens**. The Python interpreter won't load that code twice.\n",
    "\n",
    "This can be seen clearly by importing the \"Zen of Python\" twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Zen of Python, by Tim Peters\n",
      "\n",
      "Beautiful is better than ugly.\n",
      "Explicit is better than implicit.\n",
      "Simple is better than complex.\n",
      "Complex is better than complicated.\n",
      "Flat is better than nested.\n",
      "Sparse is better than dense.\n",
      "Readability counts.\n",
      "Special cases aren't special enough to break the rules.\n",
      "Although practicality beats purity.\n",
      "Errors should never pass silently.\n",
      "Unless explicitly silenced.\n",
      "In the face of ambiguity, refuse the temptation to guess.\n",
      "There should be one-- and preferably only one --obvious way to do it.\n",
      "Although that way may not be obvious at first unless you're Dutch.\n",
      "Now is better than never.\n",
      "Although never is often better than *right* now.\n",
      "If the implementation is hard to explain, it's a bad idea.\n",
      "If the implementation is easy to explain, it may be a good idea.\n",
      "Namespaces are one honking great idea -- let's do more of those!\n"
     ]
    }
   ],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the second time nothing happens\n",
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This behavior can be bypassed by importing a previously imported module with the `reload()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Zen of Python, by Tim Peters\n",
      "\n",
      "Beautiful is better than ugly.\n",
      "Explicit is better than implicit.\n",
      "Simple is better than complex.\n",
      "Complex is better than complicated.\n",
      "Flat is better than nested.\n",
      "Sparse is better than dense.\n",
      "Readability counts.\n",
      "Special cases aren't special enough to break the rules.\n",
      "Although practicality beats purity.\n",
      "Errors should never pass silently.\n",
      "Unless explicitly silenced.\n",
      "In the face of ambiguity, refuse the temptation to guess.\n",
      "There should be one-- and preferably only one --obvious way to do it.\n",
      "Although that way may not be obvious at first unless you're Dutch.\n",
      "Now is better than never.\n",
      "Although never is often better than *right* now.\n",
      "If the implementation is hard to explain, it's a bad idea.\n",
      "If the implementation is easy to explain, it may be a good idea.\n",
      "Namespaces are one honking great idea -- let's do more of those!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'this' from '/anaconda3/envs/uva/lib/python3.8/this.py'>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from importlib import reload\n",
    "reload(this)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Extra) Style Guide for Python Imports\n",
    "\n",
    "> extracted from: [PEP 8 -- Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/#imports)\n",
    "\n",
    "#### Imports should usually be on separate lines\n",
    "\n",
    "```python\n",
    "import os\n",
    "import sys\n",
    "```\n",
    "\n",
    "- **don't** do this:\n",
    "\n",
    "```python\n",
    "import os, sys\n",
    "```\n",
    "\n",
    "- while this is ok:\n",
    "\n",
    "```python\n",
    "from subprocess import Popen, PIPE\n",
    "```\n",
    "\n",
    "#### Placement\n",
    "\n",
    "Imports are always put **at the top of the file**, just after any module comments and docstrings, and before module globals and constants. They should be grouped in the following **order** (groups should be separated by blank lines):\n",
    "\n",
    "1. future-imports\n",
    "2. standard library imports\n",
    "3. related third party imports\n",
    "4. local application/library specific imports\n",
    "\n",
    "For instance:\n",
    "\n",
    "```python\n",
    "from __future__ import division\n",
    "\n",
    "import os\n",
    "import sys\n",
    "\n",
    "import numpy\n",
    "import matplotlib\n",
    "\n",
    "import my_measures\n",
    "```\n",
    "\n",
    "- Module level \"dunders\" (i.e. names with two leading and two trailing underscores) such as `__author__` and `__version__`, should be placed after the module docstring but before any import statements (except from `__future__` imports). For instance:\n",
    "\n",
    "```python\n",
    "\"\"\" This is a sample docstring\n",
    "\n",
    "It doesn't do anything, really\n",
    "\"\"\"\n",
    "\n",
    "from __future__ import division\n",
    "\n",
    "__version__ = '0.0'\n",
    "\n",
    "__author__ = \"Tom Marvolo Riddle\"\n",
    "\n",
    "import os\n",
    "import sys\n",
    "```\n",
    "\n",
    "#### Absolute imports\n",
    "\n",
    "The use of relative imports (quite common in the past) turned out to be a bad idea. As a general rules, use absolute imports whenever possible. Simply put, they are more readable and behave better:\n",
    "\n",
    "```python\n",
    "import mypkg.sibling\n",
    "from mypkg import sibling\n",
    "from mypkg.sibling import example\n",
    "```\n",
    "\n",
    "Relative imports looks like (but **don't** do it): \n",
    "\n",
    "```python\n",
    "from . import sibling\n",
    "```\n",
    "\n",
    "#### AVOID wildcard imports\n",
    "\n",
    "It is not uncommon to bump into code where modules are imported by using wildcards, such as in:\n",
    "\n",
    "```python\n",
    "from module import *\n",
    "```\n",
    "\n",
    "This practice can be harmful, in that it messes up the namespace (both for the human and the computer). \n",
    "\n",
    "While there can be a situation in which this practice can be defended (i.e. to republish an internal interface as part of a public API), in general it should be avoided."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modules Location\n",
    "\n",
    "After receiving an `import` instruction, the Python interpreter searches for the requested module in the following locations:\n",
    "\n",
    "- the current working directory\n",
    "\n",
    "\n",
    "- (if not found) each directory in the environmental variable PYTHONPATH\n",
    "\n",
    "\n",
    "- (if nothing else works) the installation-dependent default path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Want to know where the (compiled) source code of a module you've imported is located in your HD? Use the `__file__` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/anaconda3/envs/uva/lib/python3.8/os.py'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.__file__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing Modules\n",
    "\n",
    "In itself, writing a Python module is a trivial process: simply create a `[module_name].py` and import it in another file (or in the shell) by executing an `import [module_name]` statement. Try the following exercise:\n",
    "\n",
    "- In you current working directory (`%pwd`), create a text file called `exemplar_module.py`\n",
    "\n",
    "\n",
    "- Open this file and copy the following source code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python \n",
    "def schedule_meeting(week_day_numer):\n",
    "    weekdays = [\"zondag\", \"maandag\", \"dinsdag\", \"woensdag\", \"donderdag\", \"vrijdag\", \"zaterdag\"]\n",
    "    print(\"the meeting is scheduled for \" + weekdays[week_day_numer])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- import the function `schedule_meeting()` from this module and call it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we've said before, modules can contain **executable statements**. These statements are executed **only the first time** the module is imported, and are executed **also when the module is run as a script** (i.e. from the shell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When these statements are used to **initialize the module**, this behavior is not problematic (actually, that's why the interpreter works in this way)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This behavior is unwanted when we have portion of the code that are intended to be run **solely when the module is run as a script**. The solution is to have this portion of code inside an `if __name__ == \"__main__\":` statement **at the bottom** of the script.\n",
    "\n",
    "Try the following exercise:\n",
    "\n",
    "- In you current working directory (`%pwd`), create a text file called `exemplar_module_main.py`\n",
    "\n",
    "\n",
    "- Open this file and copy the following source code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python \n",
    "def schedule_meeting(week_day_numer):\n",
    "    weekdays = [\"zondag\", \"maandag\", \"dinsdag\", \"woensdag\", \"donderdag\", \"vrijdag\", \"zaterdag\"]\n",
    "    print(\"the meeting is scheduled for \" + weekdays[week_day_numer])\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    print(\"I won't show up when imported!\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- if you import the module, no message is printed..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Contrary to what happens if you execute the script from the command line: `!python exemplar_module_main.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Explanation**: when the Python interpreter reads a source file, it automatically sets the variable `__name__` to:\n",
    "\n",
    "- the module name if it is imported\n",
    "\n",
    "- `\"__main__\"` if the module is run as the main program file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing Packages\n",
    "\n",
    "*\"Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name `A.B` designates a submodule named `B` in a package named `A`\"* (source: [documentation](https://docs.python.org/2/tutorial/modules.html#packages)).\n",
    "\n",
    "Simply put, packages are hierarchical directory structures organizing an environment composed by modules and subpackages. Directories of this sort are recognized by the Python interpreter because they **have to contain** a file called `__init__.py`. The content of this file (e.g. initialization code, declaration of some sort, it may even be empty) is roughly irrelevant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try the following exercise:\n",
    "\n",
    "- In your current working directory (%pwd), create a folder called `exemplar_package`\n",
    "\n",
    "\n",
    "- In this subfolder, create an empty text file called `__init__.py`\n",
    "\n",
    "\n",
    "- In this folder, create another text file called `exemplar_module_main.py`, in which you have to copy the following source code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python \n",
    "def schedule_meeting(week_day_numer):\n",
    "    weekdays = [\"sunday\", \"monday\", \"tuesday\", \"wednesday\", \"thursday\", \"friday\", \"saturday\"]\n",
    "    print(\"the meeting is scheduled for \" + weekdays[week_day_numer])\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    print(\"I won't show up when imported!\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We can import this module and execute its only function in the following way:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import exemplar_package.exemplar_module_main\n",
    "exemplar_package.exemplar_module_main.schedule_meeting(4)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we didn't override the dutch-y version of `schedule_meeting` import from the module in our working directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions can be imported directly as well (note that in this way we override the `schedule_meeting` function from the older module)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from exemplar_package.exemplar_module_main import schedule_meeting\n",
    "schedule_meeting(4)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batteries Included: the Python Standard Library \n",
    "\n",
    "One of the Python's mostly cited strengths is the so-called \"batteries included\" philosophy, that translates into real life as the fact that the Python source distribution comes with a rich library of tools. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### A Quick Tour of (some of) the \"batteries\" we will depend on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### OS Interface\n",
    "\n",
    "The `os` module provides 500+ **platform-independent** functions that allows us to interact with the OS.\n",
    "\n",
    "Exemplar functions of this module include:\n",
    "\n",
    "- `os.getcwd()`: Return the current working directory\n",
    "- `os.chdir(path/to/directory)`: Change the current working directory to `path/to/directory`\n",
    "- `os.rename(source/path, target/path)`: Rename a file or directory `source/path` as `target/path`\n",
    "- `os.remove(path/to/file)`: Remove the `path/to/file` file\n",
    "- `os.makedirs(path/to/folder)`: Create the `path/to/folder` folder\n",
    "- `os.path.exists(path/to/file)`: Check if the `path/to/file` file exists\n",
    "- `os.path.isdir(path/to/folder)`: Check if the `path/to/folder` folder exists\n",
    "- `os.path.split(a/path)`: Split `a/path` into a pair \"(head, tail)\", where \"tail\" is the last component (e.g. a filename) and \"head\" is the rest of the path\n",
    "- `os.path.join(head/path, tail/path)`: Appends the `tail/path` to the `head/path`\n",
    "\n",
    "But the `os` module can be used also to retrieve system information, to manage processes, to create files and so forth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The Python Standard Library includes higher level alternatives to this module, among which the `tempfile` module to create temporary files, the `shutil` module for high-level file and directory handling (e.g. to copy or move a file) and `glob` to perform directory wildcard searches.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [Case 1]: Create a new folder (by overwriting existing folder)\n",
    "\n",
    "The function `os.makedirs(folder/path)` creates a new directory, but raises an error exception if the directory already exists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_name = \"tmp_folder\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "created!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.makedirs(folder_name)\n",
    "    print(\"created!\")\n",
    "except OSError as e:\n",
    "    print (e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Errno 17] File exists: 'tmp_folder'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.makedirs(folder_name)\n",
    "    print(\"created!\")\n",
    "except OSError as e:\n",
    "    print (e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `shutil` module offers the function `rmtree(path/to/folder)` to delete an entire directory, together with its files and subdirectories\n",
    "\n",
    "**Always double check!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deleted and created!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.makedirs(folder_name)\n",
    "    print(\"created!\")\n",
    "except OSError:\n",
    "    shutil.rmtree(folder_name)\n",
    "    os.makedirs(folder_name)\n",
    "    print(\"deleted and created!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTES:**\n",
    "\n",
    "- the module `os` offers the function `rmdir()` to delete a directory. Why didn't we use this? Because it only works **when the directory is empty**.\n",
    "\n",
    "\n",
    "- we could have used the `os.path.exists()` function to check if a directory exists. However, there's a chance the directory can be created between the `os.path.exists()` and the `os.makedirs()` calls, resulting in an OSError."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [Case 2]: List files and/or subfolders in a folder\n",
    "\n",
    "There are at least three functions in the python standard library that allows you to navigate a folder:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[`os.listdir()`]** returns a list of all the entries (i.e. files, links, subfolders...) in a directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tmp_folder',\n",
       " '11_Sentiment_Analysis.ipynb',\n",
       " '.DS_Store',\n",
       " '7_WordNet.ipynb',\n",
       " '10_WordEmbeddings.ipynb',\n",
       " 'images',\n",
       " '8_Vector_Semantics.ipynb',\n",
       " '0_HelloWorld.ipynb',\n",
       " '3_EvenMoreFundamentals.ipynb',\n",
       " 'stuff',\n",
       " '2_MoreFundamentals.ipynb',\n",
       " '13_Clustering_TopicModelling.ipynb',\n",
       " '4_ScientificProgramming.ipynb',\n",
       " '6_WebScraping_APIs.ipynb',\n",
       " '1_Fundamentals.ipynb',\n",
       " '.ipynb_checkpoints',\n",
       " 'data',\n",
       " '4_RegularExpressions.ipynb',\n",
       " '9_ML.ipynb',\n",
       " '7_Distributions_in_text.ipynb',\n",
       " '5_NLP_pipelines.ipynb',\n",
       " '12_Recommender_Systems.ipynb']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's see what's in our working directory\n",
    "os.listdir(os.getcwd())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11_Sentiment_Analysis.ipynb\n",
      ".DS_Store\n",
      "7_WordNet.ipynb\n",
      "10_WordEmbeddings.ipynb\n",
      "8_Vector_Semantics.ipynb\n",
      "0_HelloWorld.ipynb\n",
      "3_EvenMoreFundamentals.ipynb\n",
      "2_MoreFundamentals.ipynb\n",
      "13_Clustering_TopicModelling.ipynb\n",
      "4_ScientificProgramming.ipynb\n",
      "6_WebScraping_APIs.ipynb\n",
      "1_Fundamentals.ipynb\n",
      "4_RegularExpressions.ipynb\n",
      "9_ML.ipynb\n",
      "7_Distributions_in_text.ipynb\n",
      "5_NLP_pipelines.ipynb\n",
      "12_Recommender_Systems.ipynb\n"
     ]
    }
   ],
   "source": [
    "# it may have sense to select only the file entries:\n",
    "for entry in os.listdir(os.getcwd()):\n",
    "    if os.path.isfile(entry):\n",
    "        print (entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[`os.walk()`]** navigates the directory top-down (or bottom-up, if specified) and returns, for each directory, a 3-tuple `(dirpath, dirnames, filenames)`: `dirpath` is the path to the directory, `dirnames` is a list of the names of the subdirectories, `filenames` is a list of the names of the non-directory files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks', ['tmp_folder', 'images', 'stuff', '.ipynb_checkpoints', 'data'], ['11_Sentiment_Analysis.ipynb', '.DS_Store', '7_WordNet.ipynb', '10_WordEmbeddings.ipynb', '8_Vector_Semantics.ipynb', '0_HelloWorld.ipynb', '3_EvenMoreFundamentals.ipynb', '2_MoreFundamentals.ipynb', '13_Clustering_TopicModelling.ipynb', '4_ScientificProgramming.ipynb', '6_WebScraping_APIs.ipynb', '1_Fundamentals.ipynb', '4_RegularExpressions.ipynb', '9_ML.ipynb', '7_Distributions_in_text.ipynb', '5_NLP_pipelines.ipynb', '12_Recommender_Systems.ipynb'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/tmp_folder', [], [])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/images', [], ['supervised-classification.png', 'Indent.png', 'weighting-users.png', 'ipython.png', 'unicode.png', 'string-slicing.png', 'pathlen.png', 'notebook-ui.png', 'SA-techniques.png', 'tokens-types.png', 'array-memory.png', 'treeStructure.png', 'jupyter.png', 'batteries_included.png', 'list-slicing.png', 'sentiwordnet.png', 'nltk-downloader.png', 'vectors-similarity.png', 'thinking.png', 'weighting-item.png', '3dvectors.png', 'hr.png', 'pipeline.png', 'k-means.png', 'ic.png', 'auc.png', 'lcs.png', 'corpus-org.png', 'fibonacci_tree.png', 'zoolander2.jpg', 'DOM-table.gif', 'your_turn.jpg', 'spyder-nice-docstring-rendering.png', 'precision-recall.png'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/stuff', [], ['tweets_on_boris.csv', 'fun.py', '.DS_Store', 'output-test-1.txt', 'conf_public.conf', 'output-test-2.txt', 'conf.conf', 'tweets_from_boris.csv'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/.ipynb_checkpoints', [], ['4_RegularExpressions-checkpoint.ipynb', '4_ScientificProgramming-checkpoint.ipynb', '0_HelloWorld-checkpoint.ipynb', '3_EvenMoreFundamentals-checkpoint.ipynb', '2_MoreFundamentals-checkpoint.ipynb', '1_Fundamentals-checkpoint.ipynb'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/data', ['ml-1m', 'gutenberg-extension'], ['exemplar-lineplot.png', 'exemplar-lineplot.pdf', '.DS_Store', 'example_default_setting.csv', 'wordSim353.csv', 'ml-1m.zip', 'blogdata.txt', 'elonmusk_tweets.csv', 'example_alternative_settings.csv', 'zebo.txt', 'example_output.csv', 'output-test.txt', 'hr.png', 'k-means.png', 'tongue-twisters.txt', 'lexicon.txt', 'adams-hhgttg.txt'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/data/ml-1m', [], ['movies.dat', 'ratings.dat', 'README', 'users.dat'])\n",
      "('/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/data/gutenberg-extension', [], ['doyle-sherlock.txt', 'stoker-dracula.txt', 'README', 'shelley-frankestein.txt', 'austen-pride.txt', 'joyce-dubliners.txt'])\n"
     ]
    }
   ],
   "source": [
    "for dirpath_infos in os.walk(os.getcwd()):\n",
    "    print (dirpath_infos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `glob` module provides a function, **[`glob.glob()`]**, that returns all the directory entries that match a given pattern (without recursing into subdirectories). \n",
    "\n",
    "It can be used to find entries that have a given prefix, a given suffix or some key character sequences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `glob()` function supports two wildcards and character ranges:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `*`: **zero or more characters** in a segment of a name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/11_Sentiment_Analysis.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/7_WordNet.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/10_WordEmbeddings.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/8_Vector_Semantics.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/0_HelloWorld.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/3_EvenMoreFundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/2_MoreFundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/13_Clustering_TopicModelling.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/4_ScientificProgramming.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/6_WebScraping_APIs.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/1_Fundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/4_RegularExpressions.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/9_ML.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/7_Distributions_in_text.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/5_NLP_pipelines.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/12_Recommender_Systems.ipynb']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list all the .py files in the current working directory\n",
    "glob.glob(os.path.join(os.getcwd(), \"*.ipynb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `?`: any **single character** in a segment of a name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/7_WordNet.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/8_Vector_Semantics.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/0_HelloWorld.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/3_EvenMoreFundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/2_MoreFundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/4_ScientificProgramming.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/6_WebScraping_APIs.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/1_Fundamentals.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/4_RegularExpressions.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/9_ML.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/7_Distributions_in_text.ipynb',\n",
       " '/Users/giovannicolavizza/Dropbox/db_projects/Teaching/AUC_TMCI_2021/notebooks/5_NLP_pipelines.ipynb']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list all the tuesday notebook files\n",
    "glob.glob(os.path.join(os.getcwd(), \"?_*.ipynb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- selected characters can be specified by means of **character sets**, i.e. by placing the characters of choice between square braces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Command Line Arguments\n",
    "\n",
    "When scripts are launched from the shell, command line arguments are stored in the `argv` attribute of the [`sys`](https://docs.python.org/2.7/library/sys.html) module. \n",
    "\n",
    "`sys.argv` is a list composed by the script name followed by the ordered sequence of command line arguments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try the following exercise:\n",
    "\n",
    "- In your current working directory (`%pwd`), create a text file called `exemplar_module_args.py`.\n",
    "\n",
    "\n",
    "- Open this file and copy the following source code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python \n",
    "import sys\n",
    "\n",
    "print(sys.argv)\n",
    "    \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- launch the following script from the command line: `!python exemplar_module_args.py arg_1 arg_2 arg_3`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> See the **[getopt](https://docs.python.org/2.7/library/getopt.html)** and the **[argparse](https://docs.python.org/2.7/library/argparse.html)** modules to build more powerful and flexible command line processing tools\n",
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Reading and Writing CSV files\n",
    "\n",
    "The **CSV** (Comma Separated Values) is a **text file** format often used to import and export spreadsheet and databases. \n",
    "\n",
    "Notwithstanding its widespread use, the CSV is not a fully standardized file format, but it follows some conventions:\n",
    "\n",
    "- each spreadsheet row is a record. Most of the times a row correspond to an input file line, but sometimes it can span multiple lines;\n",
    "\n",
    "\n",
    "- each record consists of fields (you can think of them as column cells), separated by a **delimiter**. This is often a comma “`,`”, but tabs “`\\t`”, white spaces “` `” and semicolons “`;`” are widely used;\n",
    "\n",
    "\n",
    "- sometimes the first line of the text file reports the column **headers**, but this is optional;\n",
    "\n",
    "\n",
    "- **quote characters** (usually double quotes “`\"`”) are used **at least** to handle ambiguous cases (e.g. strings containing the separator character), but many options are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV files are usually parsed by looping through the input file and splitting each row, while writing to csv requires to loop through an iterator and join all the fields. The Python `csv` module takes over the handling of these tasks, by supplying specific objects that can be parameterized at will, and even some common settings called *dialects*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To **read** a csv file, use the function `reader()` to generate an object that can be used to process the rowd of the input file:   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Perry Cox', 'M.D.', 'John C. McGinley', '45']\n",
      "['Carla Espinosa-Turk', 'RN', 'Judy Reyes', '36']\n",
      "['Christopher Turk', 'M.D.', 'Donald Faison', '31']\n",
      "['Bob Kelso', 'M.D.', 'Ken Jenkins', '70']\n",
      "['Elliot Reid', 'M.D.', 'Sarah Chalke', '29']\n",
      "['J.D.', 'M.D.', 'Zach Braff', '31']\n",
      "['Janitor', 'unknown', 'Neil Flynn', '40']\n"
     ]
    }
   ],
   "source": [
    "# read & text mode\n",
    "with open(\"data/example_default_setting.csv\", \"r\") as infile:\n",
    "    scrubs_reader = csv.reader(infile)\n",
    "    for line in scrubs_reader:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure flexibility, the `reader()` functions accepts a wide range of parameters, among which the **delimiter** character,  the character used for **quoting** and the quoting behavior (see the [documentation](https://docs.python.org/3/library/csv.html) for the full list)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Perry Cox', 'M.D.', 'John C. McGinley', 45.0]\n",
      "['Carla Espinosa-Turk', 'RN', 'Judy Reyes', 36.0]\n",
      "['Christopher Turk', 'M.D.', 'Donald Faison', 31.0]\n",
      "['Bob Kelso', 'M.D.', 'Ken Jenkins', 70.0]\n",
      "['Elliot Reid', 'M.D.', 'Sarah Chalke', 29.0]\n",
      "['J.D.', 'M.D.', 'Zach Braff', 31.0]\n",
      "['Janitor', 'unknown', 'Neil Flynn', 40.0]\n"
     ]
    }
   ],
   "source": [
    "with open(\"data/example_alternative_settings.csv\", \"r\") as infile:\n",
    "    scrubs_reader = csv.reader(infile, delimiter = '\\t', quotechar = \"'\", quoting = csv.QUOTE_NONNUMERIC)\n",
    "    header = next(scrubs_reader)  # let's extract the header\n",
    "    for line in scrubs_reader:\n",
    "        print (line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['character', 'credentials', 'actor', 'age']\n"
     ]
    }
   ],
   "source": [
    "# to extract the header we used the .next() method to perform the first step of the iteration\n",
    "print (header)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV files can be written by using the `writer()` function to create an object for writing, and using `writerow()` to print each single row. For instance, to write the following list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "list2write = [['Perry Cox', 'M.D.', 'John C. McGinley', 45.0],\n",
    "              ['Carla Espinosa-Turk', 'RN', 'Judy Reyes', 36.0],\n",
    "              ['Christopher Turk', 'M.D.', 'Donald Faison', 31.0],\n",
    "              ['Bob Kelso', 'M.D.', 'Ken Jenkins', 70.0],\n",
    "              ['Elliot Reid', 'M.D.', 'Sarah Chalke', 29.0],\n",
    "              ['J.D.', 'M.D.', 'Zach Braff', 31.0],\n",
    "              ['Janitor', 'unknown', 'Neil Flynn', 40.0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "48"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "41"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "35"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "41"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(\"data/example_output.csv\", \"w\") as outfile:\n",
    "    scrubs_writer = csv.writer(outfile, quoting = csv.QUOTE_ALL)\n",
    "    for row in list2write:\n",
    "        scrubs_writer.writerow(row)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Counting\n",
    "\n",
    "A `Counter` is a container that records how many times a given object is added. As implemented in the `collections` module, this data type is as a subclass of `dict` where objects are dictionary keys and counts are values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Counters can be initialized in four different ways (note that the result is identical):\n",
    "\n",
    "- from a sequence: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'t': 2, 'e': 1, 'x': 1, 'm': 1, 'i': 2, 'n': 2, 'g': 1})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections.Counter(['t', 'e', 'x', 't', 'm', 'i', 'n', 'i', 'n', 'g'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- from a dictionary containing values and counts: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'e': 1, 'g': 1, 'i': 2, 'm': 1, 'n': 2, 't': 2, 'x': 1})"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections.Counter({'e': 1, 'g': 1, 'i': 2, 'm': 1, 'n': 2, 't': 2, 'x': 1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- using keyword arguments mapping string names to counts: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'e': 1, 'g': 1, 'i': 2, 'm': 1, 'n': 2, 't': 2, 'x': 1})"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections.Counter(e = 1, g = 1, i = 2, m = 1, n = 2, t = 2, x = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- by creating and empty counter and populating it via the `update()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'t': 2, 'i': 2, 'n': 2, 'e': 1, 'x': 1, 'm': 1, 'g': 1})\n"
     ]
    }
   ],
   "source": [
    "tm_counter = collections.Counter()\n",
    "tm_counter.update(\"textmining\")\n",
    "print(tm_counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Counters works as dictionaries, except that they return a **zero count for missing items** (rather than an error):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm_counter[\"y\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Counters support the following three methods:\n",
    "\n",
    "- The `elements()` method returns an iterator over keys repeating each keys as many times as its count:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['t', 't', 'e', 'x', 'm', 'i', 'i', 'n', 'n', 'g']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(tm_counter.elements())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `most_common(n)` method returns a sorted list of the most common *n* (key, count) pairs (when *n* is omitted, all the keys are returned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('t', 2), ('i', 2), ('n', 2)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm_counter.most_common(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('t', 2), ('i', 2), ('n', 2), ('e', 1), ('x', 1), ('m', 1), ('g', 1)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm_counter.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `subtract()` method subtract elements from an iterable or from another mapping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'i': 2, 'n': 2, 'm': 1, 'g': 1, 't': 0, 'e': 0, 'x': 0})\n"
     ]
    }
   ],
   "source": [
    "# from an iterable\n",
    "tm_counter.subtract(\"text\")\n",
    "print (tm_counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'i': 2, 'n': 2, 'm': 1, 'g': 1, 'e': -1, 'x': -1, 't': -2})\n"
     ]
    }
   ],
   "source": [
    "# from another mapping\n",
    "tm_counter.subtract(collections.Counter(e = 1, t = 2, x = 1))\n",
    "print (tm_counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Further Reading:\n",
    ">\n",
    "> Other relevant data structures in the `collections` module include `defaultdict` and `OrderedDict`.\n",
    ">\n",
    "> The best way to become familiar with the Python Standard Library is to read Doug Hellmann's [Python Module of the Week](https://pymotw.com/3/) series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1.\n",
    "\n",
    "Browse functions offered by the  the `random` module as described in the [official documentation](https://docs.python.org/3/library/random.html) to look for a way to organize the following students in 2 mutually exclusive groups of size $2 \\leq s  \\leq3$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "students = [\"Katrien\", \"Daniel\", \"Bob\", \"York\", \"James\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.\n",
    "\n",
    "The following code doesn't work. Find the bug and fix it:\n",
    "\n",
    "```python\n",
    "from random import choice\n",
    "\n",
    "outcomes = {'heads':0, 'tails':0}\n",
    "outcomes_keys = list(outcomes.keys())\n",
    "\n",
    "\n",
    "for i in range(10000):\n",
    "    outcomes[random.choice(outcomes_keys)] += 1\n",
    "\n",
    "print('Heads:', outcomes['heads'])\n",
    "print('Tails:', outcomes['tails'])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3.\n",
    "\n",
    "Read the file data/adams-hhgttg.txt and:\n",
    "\n",
    "\n",
    "- in a separate module, write a function that counts how many times a given word -- independently of its case -- occurs in a text\n",
    "\n",
    "\n",
    "- import this function and count the number of words in `adams-hhgttg.txt` (hint: use the set operation to get the words you want to apply the function on. You may filter this list somewhat to expedite calculations).\n",
    "\n",
    "\n",
    "- save these frequencies in a csv file called `data/hhgttg-frqs.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}