docs/cookbook/notebooks/Minet in a Jupyter notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "hispanic-version",
   "metadata": {},
   "source": [
    "# Using minet in a Jupyter notebook\n",
    "\n",
    "`minet` being first and foremost a python CLI tool and library, it can naturally be used in a Jupyter notebook and this makes it a very good tool to experiment with. As such it can for instance very easily be used by students in digital humanities, data science or webmining classes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "improved-torture",
   "metadata": {},
   "source": [
    "*Summary*\n",
    "\n",
    "* [Installing minet in a Jupyter notebook](#Installing-minet-in-a-Jupyter-notebook)\n",
    "* [Using minet as command line tool from a notebook](#Using-minet-as-command-line-tool-from-a-notebook)\n",
    "* [Processing the results from a notebook](#Using-minet-as-command-line-tool-from-a-notebook)\n",
    "* [Using minet as a python library](#Using-minet-as-a-python-library)\n",
    "* [Parting words](#Parting-words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numeric-eclipse",
   "metadata": {},
   "source": [
    "## Installing minet in a Jupyter notebook\n",
    "\n",
    "On a machine that supports python, `minet` can naturally be installed using `pip` in your favorite python environment running before starting your notebook:\n",
    "\n",
    "```bash\n",
    "pip install minet\n",
    "```\n",
    "\n",
    "But sometimes, the notebooks won't be run from your machine, for instance if you use some notebooks hosted by a university for their students or when using [Google Colab](https://colab.research.google.com/) (which we wholeheartedly recommend as a way to let anyone experiment with python notebooks very easily without the drag of installing a working python environment on a wide variety of computers, devices and OSes).\n",
    "\n",
    "But don't worry. There is an easy solution to this, you see, because one feature of Jupyter notebooks which is often overlooked is its ability to run shell commands when prefixing one of your code cells by `!` as in this example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "awful-lounge",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install minet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "introductory-publicity",
   "metadata": {},
   "source": [
    "Now that `minet` seems to be installed, let's run it to make sure of this and assess which version we will be working with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "specialized-basic",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "minet 0.47.0\n"
     ]
    }
   ],
   "source": [
    "!minet --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dental-trade",
   "metadata": {},
   "source": [
    "## Using minet as command line tool from a notebook\n",
    "\n",
    "Let's use minet to scrape the tweets from [@medialab_ScPo](https://twitter.com/medialab_ScPo)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "complicated-helicopter",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching for \"from:medialab_ScPo\"                                  \n",
      "Collecting tweets: 437 tweets [00:05, 74.60 tweets/s, queries=1, tokens=1]\n"
     ]
    }
   ],
   "source": [
    "!minet twitter scrape tweets \"from:medialab_ScPo\" -o tweets.csv --simple-progress"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tested-point",
   "metadata": {},
   "source": [
    "This should work quickly and when the command ends, you should have a `tweets.csv` file in your working directory containing all the relevant tweets.\n",
    "\n",
    "If you are working on Google Colab, this file will be accessible by clicking the \"folder\" icon on your left revealing your virtual filesystem. Be sure to download the file before exiting the notebook because Google Colab will delete any file you produced when shutting down your notebook to free resources. Note also that you can read and write files from Google Drive directly from a Google Colab notebook by importing and using `google.colab.drive`. More details about this [here](https://colab.research.google.com/notebooks/io.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70821b2e-414e-401f-9245-d45859b47d1d",
   "metadata": {},
   "source": [
    "*Regarding the --simple-progress flag*\n",
    "\n",
    "As `minet` was designed for the command line first and foremost, its complex progress bars, albeit very useful, can mess up with simplistic terminal emulation systems such as the one Jupyter offers.\n",
    "\n",
    "This means you should probably always use the `--simple-progress` flag when running commands from a Jupyter cell to avoid  a flurry of progress bar lines to flood your screen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "approximate-grave",
   "metadata": {},
   "source": [
    "If at one point you are lost or don't remember how to use `minet` as a command line tool, be sure to either use the `-h/--help` flag or go check the relevant [docs](https://github.com/medialab/minet/blob/master/docs/cli.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "homeless-signal",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;2;250;194;43mUsage:\u001b[0m minet twitter scrape [\u001b[36m-h\u001b[0m] [\u001b[36m--silent\u001b[0m]\n",
      "                            [\u001b[36m--refresh-per-second\u001b[0m \u001b[38;5;36mREFRESH_PER_SECOND\u001b[0m]\n",
      "                            [\u001b[36m--simple-progress\u001b[0m] [\u001b[36m--include-refs\u001b[0m] [\u001b[36m-l\u001b[0m \u001b[38;5;36mLIMIT\u001b[0m]\n",
      "                            [\u001b[36m--query-template\u001b[0m \u001b[38;5;36mQUERY_TEMPLATE\u001b[0m] [\u001b[36m-c\u001b[0m \u001b[38;5;36mCOOKIE\u001b[0m]\n",
      "                            [\u001b[36m--rcfile\u001b[0m \u001b[38;5;36mRCFILE\u001b[0m] [\u001b[36m--timezone\u001b[0m \u001b[38;5;36mTIMEZONE\u001b[0m] [\u001b[36m-i\u001b[0m \u001b[38;5;36mINPUT\u001b[0m]\n",
      "                            [\u001b[36m--explode\u001b[0m \u001b[38;5;36mEXPLODE\u001b[0m] [\u001b[36m-s\u001b[0m \u001b[38;5;36mSELECT\u001b[0m] [\u001b[36m--total\u001b[0m \u001b[38;5;36mTOTAL\u001b[0m]\n",
      "                            [\u001b[36m-o\u001b[0m \u001b[38;5;36mOUTPUT\u001b[0m]\n",
      "                            \u001b[36m{tweets,users}\u001b[0m \u001b[36mquery_or_query_column\u001b[0m\n",
      "\n",
      "\u001b[1;38;2;212;42;32m# Minet Twitter Scrape Command\u001b[0m\n",
      "\n",
      "\u001b[39mScrape Twitter's public facing search API to collect tweets or users.\u001b[0m\n",
      "\n",
      "\u001b[39mBe sure to check Twitter's advanced search to check what kind of\u001b[0m\n",
      "\u001b[39moperators you can use to tune your queries (time range, hashtags,\u001b[0m\n",
      "\u001b[39mmentions, boolean etc.):\u001b[0m\n",
      "\u001b[36mhttps://twitter.com/search-advanced?f=live\u001b[0m\n",
      "\n",
      "\u001b[39mUseful operators include \"\u001b[0m\u001b[38;5;36msince\u001b[0m\u001b[39m\" and \"\u001b[0m\u001b[38;5;36muntil\u001b[0m\u001b[39m\" to search specific\u001b[0m\n",
      "\u001b[39mtime ranges like so: \"\u001b[0m\u001b[38;5;36msince:2014-01-01 until:2017-12-31\u001b[0m\u001b[39m\", or \"\u001b[0m\u001b[38;5;36mfrom\u001b[0m\u001b[39m\" that\u001b[0m\n",
      "\u001b[39mlet you search all tweets from a given user like so: \"\u001b[0m\u001b[38;5;36mfrom:medialab_ScPo\u001b[0m\u001b[39m\".\u001b[0m\n",
      "\n",
      "\u001b[39mNote that since 2023-04-21, Twitter search is not available to\u001b[0m\n",
      "\u001b[39mvisitors of the website without being logged in. This means that\u001b[0m\n",
      "\u001b[39mthe scraper now requires some cookies to be able to work. So, by\u001b[0m\n",
      "\u001b[39mdefault, the command will grab the relevant cookies from your\u001b[0m\n",
      "\u001b[39mlocal Firefox browser (be sure to be logged in). But you can also\u001b[0m\n",
      "\u001b[39mask the command to grab the cookie from another browser or provide\u001b[0m\n",
      "\u001b[39mthe cookie string directly instead. Check the \u001b[0m\u001b[36m-c\u001b[0m\u001b[39m/\u001b[0m\u001b[36m--cookie\u001b[0m\u001b[39m docs to\u001b[0m\n",
      "\u001b[39mfigure out how to proceed.\u001b[0m\n",
      "\n",
      "\u001b[39mFinally, be advised that since you are actually using your account\u001b[0m\n",
      "\u001b[39mto scrape data, there is nothing preventing Twitter from banning your\u001b[0m\n",
      "\u001b[39maccount and it might very well happen in the future (who knows?).\u001b[0m\n",
      "\n",
      "\u001b[1;38;2;212;42;32mBEWARE\u001b[0m\u001b[39m: the web search results seem to become inconsistent when\u001b[0m\n",
      "\u001b[39mqueries return vast amounts of tweets. In which case you are\u001b[0m\n",
      "\u001b[39mstrongly advised to segment your queries using temporal filters.\u001b[0m\n",
      "\n",
      "\u001b[39mNOTE 2023-07-12: scraping users does not work anymore for now.\u001b[0m\n",
      "\u001b[39mTweet scraping was fixed but He-Who-Must-Not-Be-Named breaks things\u001b[0m\n",
      "\u001b[39mon a daily basis so be warned this might not be very stable nor\u001b[0m\n",
      "\u001b[39mconsistent.\u001b[0m\n",
      "\n",
      "\u001b[38;2;250;194;43mPositional Arguments:\u001b[0m\n",
      "  \u001b[36m{tweets,users}\u001b[0m                \u001b[39mWhat to scrape. Currently only allows for\u001b[0m\n",
      "                                \u001b[39m`\u001b[0m\u001b[38;5;36mtweets\u001b[0m\u001b[39m` or `\u001b[0m\u001b[38;5;36musers\u001b[0m\u001b[39m`.\u001b[0m\n",
      "  \u001b[36mquery_or_query_column\u001b[0m         \u001b[39mSingle query to process or name of the CSV\u001b[0m\n",
      "                                \u001b[39mcolumn containing queries when using \u001b[0m\u001b[36m-i\u001b[0m\u001b[39m/\u001b[0m\u001b[36m--input\u001b[0m\u001b[39m.\u001b[0m\n",
      "\n",
      "\u001b[38;2;250;194;43mOptional Arguments:\u001b[0m\n",
      "  \u001b[36m-c\u001b[0m, \u001b[36m--cookie\u001b[0m \u001b[38;5;36mCOOKIE\u001b[0m           \u001b[39mAuthenticated cookie to use or browser from\u001b[0m\n",
      "                                \u001b[39mwhich to extract it (supports \"\u001b[0m\u001b[38;5;36mfirefox\u001b[0m\u001b[39m\",\u001b[0m\n",
      "                                \u001b[39m\"\u001b[0m\u001b[38;5;36mchrome\u001b[0m\u001b[39m\", \"\u001b[0m\u001b[38;5;36mchromium\u001b[0m\u001b[39m\", \"\u001b[0m\u001b[38;5;36mopera\u001b[0m\u001b[39m\" and \"\u001b[0m\u001b[38;5;36medge\u001b[0m\u001b[39m\").\u001b[0m\n",
      "                                \u001b[39mDefaults to `\u001b[0m\u001b[38;5;36mfirefox\u001b[0m\u001b[39m`. Can also be configured in\u001b[0m\n",
      "                                \u001b[39ma .minetrc file as \"\u001b[0m\u001b[38;5;36mtwitter.cookie\u001b[0m\u001b[39m\" or read from\u001b[0m\n",
      "                                \u001b[39mthe MINET_TWITTER_COOKIE env variable.\u001b[0m\n",
      "  \u001b[36m--include-refs\u001b[0m                \u001b[39mWhether to emit referenced tweets (quoted,\u001b[0m\n",
      "                                \u001b[39mretweeted & replied) in the CSV output. Note\u001b[0m\n",
      "                                \u001b[39mthat it consumes a memory proportional to the\u001b[0m\n",
      "                                \u001b[39mtotal number of unique tweets retrieved.\u001b[0m\n",
      "  \u001b[36m-l\u001b[0m, \u001b[36m--limit\u001b[0m \u001b[38;5;36mLIMIT\u001b[0m             \u001b[39mMaximum number of tweets or users to collect per\u001b[0m\n",
      "                                \u001b[39mquery.\u001b[0m\n",
      "  \u001b[36m--query-template\u001b[0m \u001b[38;5;36mQUERY_TEMPLATE\u001b[0m\n",
      "                                \u001b[39mQuery template. Can be useful for instance to\u001b[0m\n",
      "                                \u001b[39mchange a column of twitter user screen names\u001b[0m\n",
      "                                \u001b[39minto from:@user queries.\u001b[0m\n",
      "  \u001b[36m--timezone\u001b[0m \u001b[38;5;36mTIMEZONE\u001b[0m           \u001b[39mTimezone for dates, for example 'Europe/Paris'.\u001b[0m\n",
      "                                \u001b[39mDefaults to UTC.\u001b[0m\n",
      "  \u001b[36m-s\u001b[0m, \u001b[36m--select\u001b[0m \u001b[38;5;36mSELECT\u001b[0m           \u001b[39mColumns of \u001b[0m\u001b[36m-i\u001b[0m\u001b[39m/\u001b[0m\u001b[36m--input\u001b[0m\u001b[39m CSV file to include in the\u001b[0m\n",
      "                                \u001b[39moutput (separated by `\u001b[0m\u001b[38;5;36m,\u001b[0m\u001b[39m`). Use an empty string\u001b[0m\n",
      "                                \u001b[39mif you don't want to keep anything: \u001b[0m\u001b[36m--select\u001b[0m\u001b[39m ''.\u001b[0m\n",
      "  \u001b[36m--explode\u001b[0m \u001b[38;5;36mEXPLODE\u001b[0m             \u001b[39mUse to indicate the character used to separate\u001b[0m\n",
      "                                \u001b[39mmultiple values in a single CSV cell. Defaults\u001b[0m\n",
      "                                \u001b[39mto none, i.e. CSV cells having a single values,\u001b[0m\n",
      "                                \u001b[39mwhich is usually the case.\u001b[0m\n",
      "  \u001b[36m--total\u001b[0m \u001b[38;5;36mTOTAL\u001b[0m                 \u001b[39mTotal number of items to process. Might be\u001b[0m\n",
      "                                \u001b[39mnecessary when you want to display a finite\u001b[0m\n",
      "                                \u001b[39mprogress indicator for large files given as\u001b[0m\n",
      "                                \u001b[39minput to the command.\u001b[0m\n",
      "  \u001b[36m-i\u001b[0m, \u001b[36m--input\u001b[0m \u001b[38;5;36mINPUT\u001b[0m             \u001b[39mCSV file (potentially gzipped) containing all\u001b[0m\n",
      "                                \u001b[39mthe queries you want to process. Will consider\u001b[0m\n",
      "                                \u001b[39m`\u001b[0m\u001b[38;5;36m-\u001b[0m\u001b[39m` as stdin.\u001b[0m\n",
      "  \u001b[36m-o\u001b[0m, \u001b[36m--output\u001b[0m \u001b[38;5;36mOUTPUT\u001b[0m           \u001b[39mPath to the output file. Will consider `\u001b[0m\u001b[38;5;36m-\u001b[0m\u001b[39m` as\u001b[0m\n",
      "                                \u001b[39mstdout. If not given, results will also be\u001b[0m\n",
      "                                \u001b[39mprinted to stdout.\u001b[0m\n",
      "  \u001b[36m--rcfile\u001b[0m \u001b[38;5;36mRCFILE\u001b[0m               \u001b[39mCustom path to a minet configuration file. More\u001b[0m\n",
      "                                \u001b[39minfo about this here:\u001b[0m\n",
      "                                \u001b[36mhttps://github.com/medialab/minet/blob/master/do\u001b[0m\n",
      "                                \u001b[36mcs/cli.md#minetrc\u001b[0m\n",
      "  \u001b[36m--refresh-per-second\u001b[0m \u001b[38;5;36mREFRESH_PER_SECOND\u001b[0m\n",
      "                                \u001b[39mNumber of times to refresh the progress bar per\u001b[0m\n",
      "                                \u001b[39msecond. Can be a float e.g. `\u001b[0m\u001b[38;5;36m0.5\u001b[0m\u001b[39m` meaning once\u001b[0m\n",
      "                                \u001b[39mevery two seconds. Use this to limit CPU usage\u001b[0m\n",
      "                                \u001b[39mwhen launching multiple commands at once.\u001b[0m\n",
      "                                \u001b[39mDefaults to `\u001b[0m\u001b[38;5;36m10\u001b[0m\u001b[39m`.\u001b[0m\n",
      "  \u001b[36m--simple-progress\u001b[0m             \u001b[39mWhether to simplify the progress bar and make it\u001b[0m\n",
      "                                \u001b[39mfit on a single line. Can be useful in terminals\u001b[0m\n",
      "                                \u001b[39mwith partial ANSI support, e.g. a Jupyter\u001b[0m\n",
      "                                \u001b[39mnotebook cell.\u001b[0m\n",
      "  \u001b[36m--silent\u001b[0m                      \u001b[39mWhether to suppress all the log and progress\u001b[0m\n",
      "                                \u001b[39mbars. Can be useful when piping.\u001b[0m\n",
      "  \u001b[36m-h\u001b[0m, \u001b[36m--help\u001b[0m                    \u001b[39mshow this help message and exit\u001b[0m\n",
      "\n",
      "\u001b[38;2;250;194;43mExamples:\u001b[0m\n",
      "\n",
      "\u001b[39m. Collecting the latest 500 tweets of a given Twitter user:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet tw scrape tweets \"\u001b[0m\u001b[38;5;36mfrom:@jack\u001b[0m\u001b[36m\" \u001b[0m\u001b[36m--limit\u001b[0m\u001b[36m 500 > tweets.csv\u001b[0m\n",
      "\n",
      "\u001b[39m. Collecting the tweets from multiple Twitter queries listed in a CSV file:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet tw scrape tweets query \u001b[0m\u001b[36m-i\u001b[0m\u001b[36m queries.csv > tweets.csv\u001b[0m\n",
      "\n",
      "\u001b[39m. Templating the given CSV column to query tweets by users:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet tw scrape tweets user \u001b[0m\u001b[36m-i\u001b[0m\u001b[36m users.csv \u001b[0m\u001b[36m--query-template\u001b[0m\u001b[36m 'from:@{value}' > tweets.csv\u001b[0m\n",
      "\n",
      "\u001b[39m. Collecting users with \"\u001b[0m\u001b[38;5;36madam\u001b[0m\u001b[39m\" in their user_name or user_description:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet tw scrape users adam > users.csv\u001b[0m\n",
      "\n",
      "\u001b[38;2;250;194;43mTips:\u001b[0m\n",
      "\n",
      "\u001b[39m- You can add a \"\u001b[0m\u001b[38;5;36mOR @aNotExistingHandle\u001b[0m\u001b[39m\" to your query to avoid searching\u001b[0m\n",
      "\u001b[39m  for your query terms in usernames or handles. Note that this is a temporary hack\u001b[0m\n",
      "\u001b[39m  which might stop working at any time so be sure to double check before relying on\u001b[0m\n",
      "\u001b[39m  this trick.\u001b[0m\n",
      "\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet tw scrape tweets \"\u001b[0m\u001b[38;5;36mkeyword OR @anObviouslyNotExistingHandle\u001b[0m\u001b[36m\"\u001b[0m\n",
      "\n",
      "\u001b[39m  For more information see the related discussion here:\u001b[0m\n",
      "\u001b[39m  \u001b[0m\u001b[36mhttps://webapps.stackexchange.com/questions/127425/how-to-exclude-usernames-and-handles-while-searching-twitter\u001b[0m\n",
      "\n",
      "\u001b[3;38;2;250;194;43mhow to use the command with a CSV file?\u001b[0m\n",
      "\n",
      "\u001b[2;39m> A lot of minet commands, including this one, can both be\u001b[0m\n",
      "\u001b[2;39m> given a single value to process or a bunch of them if\u001b[0m\n",
      "\u001b[2;39m> given the column of a CSV file passed to \u001b[0m\u001b[2;36m-i\u001b[0m\u001b[2;39m/\u001b[0m\u001b[2;36m--input\u001b[0m\u001b[2;39m instead.\u001b[0m\n",
      "\n",
      "\u001b[2;39m> Note that when given a CSV file as input, minet will\u001b[0m\n",
      "\u001b[2;39m> concatenate the input file columns with the ones added\u001b[0m\n",
      "\u001b[2;39m> by the command. You can always restrict the input file\u001b[0m\n",
      "\u001b[2;39m> columns to keep by using the \u001b[0m\u001b[2;36m-s\u001b[0m\u001b[2;39m/\u001b[0m\u001b[2;36m--select\u001b[0m\u001b[2;39m flag.\u001b[0m\n",
      "\n",
      "\u001b[39m. Here is how to use a command with a single value:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet twitter scrape \"\u001b[0m\u001b[38;5;36mvalue\u001b[0m\u001b[36m\"\u001b[0m\n",
      "\n",
      "\u001b[39m. Here is how to use a command with a CSV file:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet twitter scrape column_name \u001b[0m\u001b[36m-i\u001b[0m\u001b[36m file.csv\u001b[0m\n",
      "\n",
      "\u001b[39m. Here is how to read CSV file from stdin using `\u001b[0m\u001b[38;5;36m-\u001b[0m\u001b[39m`:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mxsv search \u001b[0m\u001b[36m-s\u001b[0m\u001b[36m col . | minet twitter scrape column_name \u001b[0m\u001b[36m-i\u001b[0m\u001b[36m -\u001b[0m\n",
      "\n",
      "\u001b[39m. Here is how to indicate that the CSV column may contain multiple\u001b[0m\n",
      "\u001b[39m  values separated by a special character:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet twitter scrape column_name \u001b[0m\u001b[36m-i\u001b[0m\u001b[36m file.csv \u001b[0m\u001b[36m--explode\u001b[0m\u001b[36m \"\u001b[0m\u001b[38;5;36m|\u001b[0m\u001b[36m\"\u001b[0m\n",
      "\n",
      "\u001b[39m. This also works with single values:\u001b[0m\n",
      "\u001b[39m    $ \u001b[0m\u001b[36mminet twitter scrape \"\u001b[0m\u001b[38;5;36mvalue1,value2\u001b[0m\u001b[36m\" \u001b[0m\u001b[36m--explode\u001b[0m\u001b[36m \"\u001b[0m\u001b[38;5;36m,\u001b[0m\u001b[36m\"\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!minet twitter scrape --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorporate-bacon",
   "metadata": {},
   "source": [
    "So now you should be able to compose your own commands to scrape tweets you might need. Twitter's search is quite complete and also supports a variety of search operators which you can discover by using the advanced search widget [here](https://twitter.com/search-advanced?f=live).\n",
    "\n",
    "Never forget to use the `--limit` flag when testing a query or when searching for generic things or your command will never end and at one point your hard drive will probably choke if you attempt to retrieve billions of tweets :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incorporate-anime",
   "metadata": {},
   "source": [
    "## Processing the results from a notebook\n",
    "\n",
    "As you are in a python environment, you can of course use the capabilities of the language to explore the resulting CSV files further.\n",
    "\n",
    "Here is an example printing our `tweets.csv` file's headers and displaying the lab's most retweeted tweet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "recent-quarterly",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Headers: ['query', 'id', 'timestamp_utc', 'local_time', 'user_screen_name', 'text', 'possibly_sensitive', 'retweet_count', 'like_count', 'reply_count', 'lang', 'to_username', 'to_userid', 'to_tweetid', 'source_name', 'source_url', 'user_location', 'lat', 'lng', 'user_id', 'user_name', 'user_verified', 'user_description', 'user_url', 'user_image', 'user_tweets', 'user_followers', 'user_friends', 'user_likes', 'user_lists', 'user_created_at', 'user_timestamp_utc', 'collected_via', 'match_query', 'retweeted_id', 'retweeted_user', 'retweeted_user_id', 'retweeted_timestamp_utc', 'quoted_id', 'quoted_user', 'quoted_user_id', 'quoted_timestamp_utc', 'collection_time', 'url', 'place_country_code', 'place_name', 'place_type', 'place_coordinates', 'links', 'media_urls', 'media_files', 'media_types', 'mentioned_names', 'mentioned_ids', 'hashtags', 'intervention_type', 'intervention_text', 'intervention_url']\n",
      "\n",
      "Best tweet with 63 retweets: Ça sort aujourd'hui en libraire : l'ouvrage \"Culture #Numérique\" de @Karmacoma pour décoder les nouveaux usages apportés par la révolution numérique. #digitalrevolution #digitalculture\n",
      "https://medialab.sciencespo.fr/fr/blog/culture-numerique/ https://twitter.com/medialab_ScPo/status/1103588839740919808/photo/1\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "\n",
    "with open('./tweets.csv') as f:\n",
    "    reader = csv.DictReader(f)\n",
    "    print('Headers:', reader.fieldnames)\n",
    "    print()\n",
    "    \n",
    "    best_tweet = max(reader, key=lambda x: int(x['retweet_count']))\n",
    "    print('Best tweet with %s retweets:' % best_tweet['retweet_count'], best_tweet['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "independent-joyce",
   "metadata": {},
   "source": [
    "Here is another example using the `pandas` library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "single-custom",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>id</th>\n",
       "      <th>timestamp_utc</th>\n",
       "      <th>local_time</th>\n",
       "      <th>user_screen_name</th>\n",
       "      <th>text</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>like_count</th>\n",
       "      <th>reply_count</th>\n",
       "      <th>...</th>\n",
       "      <th>links</th>\n",
       "      <th>media_urls</th>\n",
       "      <th>media_files</th>\n",
       "      <th>media_types</th>\n",
       "      <th>mentioned_names</th>\n",
       "      <th>mentioned_ids</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>intervention_type</th>\n",
       "      <th>intervention_text</th>\n",
       "      <th>intervention_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>from:medialab_ScPo</td>\n",
       "      <td>1371864480540397573</td>\n",
       "      <td>1615909354</td>\n",
       "      <td>2021-03-16T16:42:34</td>\n",
       "      <td>medialab_ScPo</td>\n",
       "      <td>Mardi 23/03, @brunopatino est l'invité du sémi...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>https://medialab.sciencespo.fr/actu/lien-entre...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>brunopatino</td>\n",
       "      <td>11227042</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>from:medialab_ScPo</td>\n",
       "      <td>1371408961740673027</td>\n",
       "      <td>1615800750</td>\n",
       "      <td>2021-03-15T10:32:30</td>\n",
       "      <td>medialab_ScPo</td>\n",
       "      <td>Digital Growth Strategies : is there a new way...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>https://medialab.sciencespo.fr/en/news/digital...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>from:medialab_ScPo</td>\n",
       "      <td>1367834292966068226</td>\n",
       "      <td>1614948483</td>\n",
       "      <td>2021-03-05T13:48:03</td>\n",
       "      <td>medialab_ScPo</td>\n",
       "      <td>La Silicon Valley : un écosystème d’innovation...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>https://sciencespo.zoom.us/meeting/register/tJ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>from:medialab_ScPo</td>\n",
       "      <td>1367750981585354754</td>\n",
       "      <td>1614928620</td>\n",
       "      <td>2021-03-05T08:17:00</td>\n",
       "      <td>medialab_ScPo</td>\n",
       "      <td>Aujourd'hui, 14h30 -&gt; TransNum s'intéresse à l...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>https://medialab.sciencespo.fr/actu/silicon-va...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>from:medialab_ScPo</td>\n",
       "      <td>1366433462723436550</td>\n",
       "      <td>1614614499</td>\n",
       "      <td>2021-03-01T17:01:39</td>\n",
       "      <td>medialab_ScPo</td>\n",
       "      <td>Qu'est-ce qui fait de la Silicon Valley un mod...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>https://medialab.sciencespo.fr/actu/silicon-va...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                query                   id  timestamp_utc  \\\n",
       "0  from:medialab_ScPo  1371864480540397573     1615909354   \n",
       "1  from:medialab_ScPo  1371408961740673027     1615800750   \n",
       "2  from:medialab_ScPo  1367834292966068226     1614948483   \n",
       "3  from:medialab_ScPo  1367750981585354754     1614928620   \n",
       "4  from:medialab_ScPo  1366433462723436550     1614614499   \n",
       "\n",
       "            local_time user_screen_name  \\\n",
       "0  2021-03-16T16:42:34    medialab_ScPo   \n",
       "1  2021-03-15T10:32:30    medialab_ScPo   \n",
       "2  2021-03-05T13:48:03    medialab_ScPo   \n",
       "3  2021-03-05T08:17:00    medialab_ScPo   \n",
       "4  2021-03-01T17:01:39    medialab_ScPo   \n",
       "\n",
       "                                                text  possibly_sensitive  \\\n",
       "0  Mardi 23/03, @brunopatino est l'invité du sémi...                 0.0   \n",
       "1  Digital Growth Strategies : is there a new way...                 0.0   \n",
       "2  La Silicon Valley : un écosystème d’innovation...                 0.0   \n",
       "3  Aujourd'hui, 14h30 -> TransNum s'intéresse à l...                 0.0   \n",
       "4  Qu'est-ce qui fait de la Silicon Valley un mod...                 0.0   \n",
       "\n",
       "   retweet_count  like_count  reply_count  ...  \\\n",
       "0              7          13            0  ...   \n",
       "1              7          11            0  ...   \n",
       "2              0           1            1  ...   \n",
       "3              4           9            0  ...   \n",
       "4             10          13            0  ...   \n",
       "\n",
       "                                               links media_urls  media_files  \\\n",
       "0  https://medialab.sciencespo.fr/actu/lien-entre...        NaN          NaN   \n",
       "1  https://medialab.sciencespo.fr/en/news/digital...        NaN          NaN   \n",
       "2  https://sciencespo.zoom.us/meeting/register/tJ...        NaN          NaN   \n",
       "3  https://medialab.sciencespo.fr/actu/silicon-va...        NaN          NaN   \n",
       "4  https://medialab.sciencespo.fr/actu/silicon-va...        NaN          NaN   \n",
       "\n",
       "   media_types mentioned_names mentioned_ids hashtags  intervention_type  \\\n",
       "0          NaN     brunopatino      11227042      NaN                NaN   \n",
       "1          NaN             NaN           NaN      NaN                NaN   \n",
       "2          NaN             NaN           NaN      NaN                NaN   \n",
       "3          NaN             NaN           NaN      NaN                NaN   \n",
       "4          NaN             NaN           NaN      NaN                NaN   \n",
       "\n",
       "   intervention_text  intervention_url  \n",
       "0                NaN               NaN  \n",
       "1                NaN               NaN  \n",
       "2                NaN               NaN  \n",
       "3                NaN               NaN  \n",
       "4                NaN               NaN  \n",
       "\n",
       "[5 rows x 58 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./tweets.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seven-contrary",
   "metadata": {},
   "source": [
    "## Using minet as a python library\n",
    "\n",
    "Having been developed in python, `minet` can also be used as a python library if you ever need to. It can be useful if you need to integrate `minet` schemes into your own python workflow or just need to customize things. Just know that often the command line tool handles a lot of things for you that you will, as a result, need to handle yourself when working from python.\n",
    "\n",
    "The relevant documentation can be found [here](https://github.com/medialab/minet/blob/master/docs/lib.md).\n",
    "\n",
    "Here is an example of Twitter scraping directly from python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "further-plymouth",
   "metadata": {},
   "outputs": [],
   "source": [
    "from minet.twitter import TwitterAPIScraper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "useful-virgin",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mardi 23/03, @brunopatino est l'invité du séminaire du médialab. A travers ses derniers ouvrages, il abordera les liens entre journalisme, information et environment techno économique.\n",
      "Ouvert à tous, sur inscription : https://medialab.sciencespo.fr/actu/lien-entre-journalisme-information-et-environnement-techno-economique-le-cas-de-la-revolution-numerique/\n"
     ]
    }
   ],
   "source": [
    "scraper = TwitterAPIScraper()\n",
    "\n",
    "for tweet in scraper.search_tweets('from:medialab_ScPo'):\n",
    "    print(tweet['text'])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "requested-threshold",
   "metadata": {},
   "source": [
    "## Parting words\n",
    "\n",
    "Now you should have a clear idea of how to use `minet` from a Jupyter notebook, for your personal use or for students in a class, for instance.\n",
    "\n",
    "However, one should note that Jupyter notebooks, as wonderful as they are, are not a good way to industrialize data collection processes and are not meant to be long-running (e.g. when collecting Tweets, you might want to let your command run for days). For this kind of endeavours, traditional tools such as a Unix shells and some sysadmin skills can't be beaten.\n",
    "\n",
    "Finally, some `minet` commands might require that you given credentials to access some APIs (such as YouTube's one). Be sure not to publish those, for instance if you work on Google Colab, so that people won't steal them from you on the web.\n",
    "\n",
    "If you need to feed those commands with credentials differently, check out `minet` docs about config files and environment variables [here](https://github.com/medialab/minet/blob/master/docs/cli.md#minetrc). Note that both those options can be easy footguns on Google Colab also because leaving a config file in the open or having a cell setting a environment variable are not good solutions :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}