earnings_filter_by_region.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pF8beWv0lrgc"
      },
      "source": [
        "# Web Scraping Yahoo! Finance using Python\n",
        "\n",
        "A detailed guide for web scraping https://finance.yahoo.com/ using **Selenium**, **HTML tags**\n",
        "\n",
        "![](https://i.imgur.com/V1bzyMs.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FULGhEjlrge"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "**What is Web scraping?**<br>\n",
        "Web scraping is the process of extracting and parsing data from websites in an automated fashion using a computer program. It's a useful technique for creating datasets for research and learning.\n",
        "\n",
        "\n",
        "**Objective**<br>\n",
        "The main objective of this tutorial is to showcase different web scraping methods which can be applied to any web page. \n",
        "This is for educational purposes only. Please read the Terms & Conditions carefully for any website to see whether you can legally use the data. \n",
        "\n",
        "In this project, we will perform web scraping using the following 3 techniques based on the problem statement.\n",
        "* use `Selenium` to scrape data from dynamically loading websites \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kbMRHIhblrgh"
      },
      "source": [
        "**How to run the Code**<br>\n",
        "You can execute the code using \"Run\" button on the top of this page and selecting **\"Run on Colab\"** or **\"Run Locally\"** \n",
        "<br>\n",
        "<br>\n",
        "**Setup and Tools**<br>\n",
        "<u>Run on Colab :</u> \n",
        "    You will need to provide the Google login to run this notebook on Colab.<br>\n",
        "<u>Run Locally :</u> Download and install [Anaconda](https://www.anaconda.com/) framework, We will be using Jupyter Notebook for writing & executing code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gaOUKkUOlrgt"
      },
      "source": [
        "## 2. Web Scraping Earnings \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IuViJR0Elrgt"
      },
      "source": [
        "\n",
        "Here's an outline of the steps we'll follow<br>\n",
        "**2.1 Introduction of selenium**<br>\n",
        "**2.2 Downloads & Installation**<br>\n",
        "**2.3 Install & Import libraries**<br>\n",
        "**2.4 Create Web Driver**<br>\n",
        "**2.5 Exploring and locating Elements**<br>\n",
        "**2.6 Extract & Compile the information into a python list**<br>\n",
        "**2.7 Save the extracted information to a CSV file**<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvUPJgUEQWZo"
      },
      "source": [
        "### 2.1 Introduction of selenium\n",
        "\n",
        "**[Selenium](https://www.selenium.dev/)** is an open-source web-based automation tool. Python language and other languages are used with Selenium for testing as well as web scraping. Here we will use Chrome browser, but you can try on any browser.<br>\n",
        "\n",
        "**Why you should use Selenium?**\n",
        "- Clicking on buttons\n",
        "- Filling forms\n",
        "- Scrolling\n",
        "- Taking a screen-shot\n",
        "- Refreshing the page\n",
        "\n",
        "You can find proper documentation on selenium [here](https://selenium-python.readthedocs.io/)<br>\n",
        "\n",
        "The following methods will help to find elements in a webpage (these methods will return a list):\n",
        "- `find_elements_by_name`\n",
        "- `find_elements_by_xpath`\n",
        "- `find_elements_by_link_text`\n",
        "- `find_elements_by_partial_link_text`\n",
        "- `find_elements_by_tag_name`\n",
        "- `find_elements_by_class_name`\n",
        "- `find_elements_by_css_selector`\n",
        "\n",
        "In this tutorial we will use only `find_elements_by_xpath` and `find_elements_by_tag_name` You can find complete documentation of these methods [here](https://selenium-python.readthedocs.io/locating-elements.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y5IXSU3_QWZs"
      },
      "source": [
        "### 2.2 Downloads & Installation \n",
        "\n",
        "Unlike the previous section, here we'll have to do some prep work to implement this method. We will need to install Selenium & proper web browser driver<br>\n",
        "\n",
        "If you are using **Google Colab** platform then execute following code to perform Initial installation. This piece of code `'google.colab' in str(get_ipython())` is used to identify the Google Colab platform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 160,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ot7iEH0QQWZs",
        "outputId": "3113b334-68a6-45b7-89fa-a3bffd87c746"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Google CoLab Installation\n",
            "Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
            "Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
            "Hit:4 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease\n",
            "Hit:5 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease\n",
            "Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB]\n",
            "Hit:10 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "Hit:11 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease\n",
            "Hit:12 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "Fetched 261 kB in 3s (87.7 kB/s)\n",
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "27 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "chromium-chromedriver is already the newest version (105.0.5195.102-0ubuntu0.18.04.1).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'apt autoremove' to remove it.\n",
            "0 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.\n"
          ]
        }
      ],
      "source": [
        "if 'google.colab' in str(get_ipython()):\n",
        "    print('Google CoLab Installation')\n",
        "    !apt update --quiet\n",
        "    !apt install chromium-chromedriver --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5kj9N6faQWZs"
      },
      "source": [
        "To run it on **Locally** you will need **Webdriver for Chrome** on your machine. You can download it from this link https://chromedriver.chromium.org/downloads and just copy the file in the folder where we will create the python file (No need of installation). But make sure that the driver‘s version matches the Chrome browser version installed on the local machine.\n",
        "\n",
        "![](https://i.imgur.com/FvQ586e.gif)\n",
        "![](https://i.imgur.com/wQbjRIU.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGOzu85gQWZs"
      },
      "source": [
        "### 2.3 Install & Import libraries\n",
        "\n",
        "Installation of the required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 161,
      "metadata": {
        "id": "WmDBSb_qQWZt"
      },
      "outputs": [],
      "source": [
        "!pip install selenium --quiet\n",
        "!pip install pandas --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pn1J-U1PQWZt"
      },
      "source": [
        "Once the Libraries installation is done, next step is to import all the required modules / libraries. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 162,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PaHNd1hSQWZt",
        "outputId": "2b856f6d-43b1-427f-c4d1-e7b0db8a97b8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Library Import\n",
            "Running on CoLab\n",
            "Common Library Import\n"
          ]
        }
      ],
      "source": [
        "print('Library Import')\n",
        "if 'google.colab' not in str(get_ipython()):\n",
        "    print('Not running on CoLab')\n",
        "    from selenium.webdriver.chrome.options import Options\n",
        "    from selenium.webdriver.chrome.service import Service\n",
        "    import os\n",
        "else:\n",
        "    print('Running on CoLab')\n",
        "    \n",
        "print('Common Library Import')\n",
        "from selenium import webdriver\n",
        "from selenium.webdriver.common.by import By\n",
        "from selenium.webdriver.support.ui import WebDriverWait\n",
        "from selenium.webdriver.support import expected_conditions as EC\n",
        "from selenium.webdriver.common.keys import Keys\n",
        "from selenium.webdriver.common.action_chains import ActionChains\n",
        "import pandas as pd \n",
        "import time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlfwVrnVQWZt"
      },
      "source": [
        "So all the necessary prep work is done. Let's  move ahead to implement this method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ms87q9huQWZu"
      },
      "source": [
        "### 2.4 Create Web Driver\n",
        "\n",
        "In this step first we will create the instance of Chrome WebDriver using `webdriver.Chrome()` method. and then the `driver.get()` method will navigate to a page given by the URL. In this case also there is slight variation based on platform. Also we have used `options` parameters for e.g. `--headless` option will load the driver in background. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 163,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cAn7GekCQWZu",
        "outputId": "68f80315-57fe-4ad1-af47-7a4dd5d829e8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Running on CoLab\n"
          ]
        }
      ],
      "source": [
        "if 'google.colab' in str(get_ipython()):\n",
        "    print('Running on CoLab')\n",
        "    def get_driver(url):\n",
        "        \"\"\"Return web driver\"\"\"\n",
        "        colab_options = webdriver.ChromeOptions()\n",
        "        colab_options.add_argument('--no-sandbox')\n",
        "        colab_options.add_argument('--disable-dev-shm-usage')\n",
        "        colab_options.add_argument('--headless')\n",
        "        colab_options.add_argument('--start-maximized') \n",
        "        colab_options.add_argument('--start-fullscreen')\n",
        "        colab_options.add_argument('--single-process')\n",
        "        driver = webdriver.Chrome(options=colab_options)\n",
        "        driver.get(url)\n",
        "        return driver\n",
        "else:\n",
        "    print('Not running on CoLab')\n",
        "    def get_driver(url):\n",
        "        \"\"\"Return web driver\"\"\"\n",
        "        chrome_options = Options()\n",
        "        chrome_options.add_argument('--no-sandbox')\n",
        "        chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "        chrome_options.add_argument('--headless')\n",
        "        chrome_options.add_argument('--start-maximized') \n",
        "        chrome_options.add_argument('--start-fullscreen')\n",
        "        chrome_options.add_argument('--single-process')\n",
        "        serv = Service(os.getcwd()+'/chromedriver')\n",
        "        driver = webdriver.Chrome(options=chrome_options, service=serv)\n",
        "        driver.get(url)\n",
        "        return driver"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qnCtyw66QWZu"
      },
      "source": [
        "Test run of `get_driver`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 184,
      "metadata": {
        "id": "rd7DJYjaQWZu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d68768b2-0bd6-457a-dcf9-f1752ecbbf74"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Company Earnings Calendar - Yahoo Finance\n"
          ]
        }
      ],
      "source": [
        "EARNINGS_URL = 'https://finance.yahoo.com/calendar/earnings?from=2022-10-16&to=2022-10-22&day=2022-10-21' ## CHANGE THE DATE \n",
        "driver = get_driver(EARNINGS_URL)\n",
        "print(driver.title)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "filter_arrow = driver.find_element(By.XPATH, value='//*[@id=\"screener-criteria\"]/div/header/button/span[1]/span')\n",
        "filter_arrow.click()\n",
        "time.sleep(5)"
      ],
      "metadata": {
        "id": "xjOpy42LMZrw"
      },
      "execution_count": 185,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "unclick_us = driver.find_element(By.XPATH, value='//*[@id=\"screener-criteria\"]/div/div[1]/div[1]/div[1]/div/div[2]/ul/li[1]/button/span')\n",
        "unclick_us.click()\n",
        "time.sleep(5)"
      ],
      "metadata": {
        "id": "N9aU_aijOzRQ"
      },
      "execution_count": 186,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "add_region = driver.find_element(By.XPATH, value='//*[@id=\"screener-criteria\"]/div/div[1]/div[1]/div[1]/div/div[2]/ul/li/div/div/span')\n",
        "add_region.click()\n",
        "time.sleep(5)"
      ],
      "metadata": {
        "id": "gm2MgoaBPUok"
      },
      "execution_count": 187,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "find_filter = driver.find_element(By.XPATH, value='//*[@id=\"dropdown-menu\"]/div/div[1]/div/input')\n",
        "find_filter.send_keys(\"Brazil\") ## you can replace brazil with any other country \n",
        "time.sleep(2)\n",
        "find_filter.send_keys(Keys.TAB, Keys.TAB, Keys.SPACE)\n",
        "#Keys.TAB\n",
        "#Keys.TAB\n",
        "#Keys.SPACE"
      ],
      "metadata": {
        "id": "z15sknGDQFu_"
      },
      "execution_count": 188,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "find_button = driver.find_element(By.XPATH, value='//*[@id=\"screener-criteria\"]/div/div[1]/div[3]/button/span')\n",
        "find_button.click()\n",
        "time.sleep(3)"
      ],
      "metadata": {
        "id": "hDeGSU9oRN2y"
      },
      "execution_count": 189,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pOF6149XQWZu"
      },
      "source": [
        "### 2.5 Exploring and locating Elements\n",
        "\n",
        "This is almost similar step that we have done in phase 1. We will try to identify relevant information like `<tags>`, `class` , `XPath` etc from the web page. Right-click and select the \"Inspect\" to do further analysis.\n",
        "\n",
        "As the webpage showing cryptocurrency information in the Table form. We can grab the table header by using tag `<th>`, we will use find_elements by TAG to get the table headers. These headers can be used as columns for a CSV file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 190,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zR29b9xH8di9",
        "outputId": "f33efb7d-7fc5-4d6b-9d56-5e08f255c7cb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Symbol\n",
            "Earnings Call Time\n"
          ]
        }
      ],
      "source": [
        "header = driver.find_elements(By.TAG_NAME, value= 'th')\n",
        "print(header[0].text)\n",
        "print(header[2].text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lNZ0kwDhQWZv"
      },
      "source": [
        "Creating a helper function to get first 10 columns from header, we have used List comprehension with conditions. You can also check out usage of `enumerate` method. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 191,
      "metadata": {
        "id": "QTfDpiWrQWZv"
      },
      "outputs": [],
      "source": [
        "def get_table_header(driver):\n",
        "    \"\"\"Return Table columns in list form \"\"\"\n",
        "    header = driver.find_elements(By.TAG_NAME, value= 'th')\n",
        "    header_list = [item.text for index, item in enumerate(header) ]\n",
        "    return header_list"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "header_list = get_table_header(driver)\n",
        "print(header_list)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uid9byVXYI_w",
        "outputId": "cfa940d9-54c1-46fb-875e-0861b642a5ea"
      },
      "execution_count": 192,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Symbol', 'Company', 'Earnings Call Time', 'EPS Estimate', 'Reported EPS', 'Surprise(%)']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vqW7UgNwQWZv"
      },
      "source": [
        "Next we find out number of rows available in a Page, you can see table rows are placed in `<tr>` tag, we can capture the `XPath` by selection `<tr>` tag the Right Click $\\rightarrow$ Copy $\\rightarrow$ Copy XPath.\n",
        "\n",
        "![](https://i.imgur.com/DVAYMzY.gif)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 193,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "U3FVWjftQWZv",
        "outputId": "eb306386-dbf3-4496-f47a-2e442df1aab1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'WHRL4.SA\\nWhirlpool SA TAS - - -'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 193
        }
      ],
      "source": [
        "txt=driver.find_element(By.XPATH, value='//*[@id=\"cal-res-table\"]/div[1]/table/tbody/tr[1]').text\n",
        "txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g6JcCdb1Trrp"
      },
      "source": [
        "Above `XPath` points to first row, we can get rid of row number part from XPath and use it with `find_elements` to get hold of all the available rows. Let's implement this in a function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 194,
      "metadata": {
        "id": "dd9kJccjToHK"
      },
      "outputs": [],
      "source": [
        "def get_table_rows(driver):\n",
        "    \"\"\"Get number of rows available on the page \"\"\"\n",
        "    tablerows = len(driver.find_elements(By.XPATH, value='//*[@id=\"cal-res-table\"]/div[1]/table/tbody/tr'))\n",
        "    return tablerows    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 195,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0v1phXVMRn2E",
        "outputId": "67af1561-a3ef-4fbe-b918-84fe0ae9ecb4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n"
          ]
        }
      ],
      "source": [
        "print(get_table_rows(driver))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A45Ekufq8di9"
      },
      "source": [
        "Similarly, we can take the XPath for any column value.\n",
        "\n",
        "![](https://i.imgur.com/aT3I3Ur.gif)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 196,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "1MNZ9bGilrgt",
        "outputId": "4ccc119c-d5fc-4d80-a205-d7d89f0dae09"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Whirlpool SA'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 196
        }
      ],
      "source": [
        "driver.find_element(By.XPATH, value='//*[@id=\"cal-res-table\"]/div[1]/table/tbody/tr[1]/td[2]').text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2KhgeZAW7d8"
      },
      "source": [
        "So we can change the `row_number` & `column_number` in `XPath` and loop it through row count and column count to get all the available column values. Let's generalize this and put it in a function. We will get the data for one row at a time and return column values in the form of a dictionary "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 197,
      "metadata": {
        "id": "pEorS2flYikH"
      },
      "outputs": [],
      "source": [
        "def parse_table_rows(rownum, driver, header_list):\n",
        "    \"\"\"get the data for one row at a time and return column value in the form of dictionary\"\"\"\n",
        "    row_dictionary = {}\n",
        "    #time.sleep(1/3)\n",
        "    for index , item in enumerate(header_list):\n",
        "        time.sleep(1/20)\n",
        "        column_xpath = '//*[@id=\"cal-res-table\"]/div[1]/table/tbody/tr[{}]/td[{}]'.format(rownum, index+1)\n",
        "        row_dictionary[item] = driver.find_element(By.XPATH, value=column_xpath).text\n",
        "    return row_dictionary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 198,
      "metadata": {
        "id": "s45SMjxligLN"
      },
      "outputs": [],
      "source": [
        "#button_element = driver.find_element(By.XPATH, value = '//*[@id=\"cal-res-table\"]/div[2]/button[3]')\n",
        "#button_element.click()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 199,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "917rbrdZKH13",
        "outputId": "ea97aaec-beee-40dd-ac7f-035172b9885e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'WHRL4.SA\\nWhirlpool SA TAS - - -'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 199
        }
      ],
      "source": [
        "txt=driver.find_element(By.XPATH, value='//*[@id=\"cal-res-table\"]/div[1]/table/tbody/tr[1]').text\n",
        "txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osU8bvFYeIgu"
      },
      "source": [
        "In this section we have learned how to get required data points, and perform events on webpage. "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## parsing each row\n",
        "table_data = []\n",
        "table_rows = get_table_rows(driver)\n",
        "print('Found {} rows'.format(table_rows))\n",
        "table_data += [parse_table_rows(i, driver, header_list) for i in range (1, table_rows + 1)]\n",
        "total_count = len(table_data)\n",
        "print('Total rows scraped : {}'.format(total_count))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jzMUYAzXaLYy",
        "outputId": "88e9bb98-9bd2-4e82-ff37-838822fd5c2f"
      },
      "execution_count": 200,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 1 rows\n",
            "Total rows scraped : 1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## save data\n",
        "print('Save the data to a CSV')\n",
        "table_df = pd.DataFrame(table_data)\n",
        "table_df.to_csv('earnings_selenium.csv', index=None)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1_p66pm0bOpc",
        "outputId": "4b956cdb-1ac8-48d6-8850-802ca8d156c3"
      },
      "execution_count": 201,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Save the data to a CSV\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "SSybN-EHbn_9",
        "outputId": "004a2c77-be88-49a3-8e83-f1a1bb2aa08b"
      },
      "execution_count": 202,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     Symbol       Company Earnings Call Time EPS Estimate Reported EPS  \\\n",
              "0  WHRL4.SA  Whirlpool SA                TAS            -            -   \n",
              "\n",
              "  Surprise(%)  \n",
              "0           -  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5b5c18be-d69c-46c8-a640-af9be97d8cda\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Symbol</th>\n",
              "      <th>Company</th>\n",
              "      <th>Earnings Call Time</th>\n",
              "      <th>EPS Estimate</th>\n",
              "      <th>Reported EPS</th>\n",
              "      <th>Surprise(%)</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>WHRL4.SA</td>\n",
              "      <td>Whirlpool SA</td>\n",
              "      <td>TAS</td>\n",
              "      <td>-</td>\n",
              "      <td>-</td>\n",
              "      <td>-</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5b5c18be-d69c-46c8-a640-af9be97d8cda')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5b5c18be-d69c-46c8-a640-af9be97d8cda button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5b5c18be-d69c-46c8-a640-af9be97d8cda');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 202
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 203,
      "metadata": {
        "id": "J0CKtbmTf5aF"
      },
      "outputs": [],
      "source": [
        "#terminating driver \n",
        "driver.close()\n",
        "driver.quit() "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ion8RYbZ8di_"
      },
      "source": [
        "**Summary** : Hope you've enjoyed this tutorial. Selenium enables us to perform multiple actions on the web browser, which is really very handy for scraping different types of data from any webpage.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "language_info": {
      "name": "python"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}