merged.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3604af7",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "<script src=\"https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js\"></script>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52988e89",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"text-align: left;\">\n",
    "    <h1>Webscraping</h1>\n",
    "    <h4>Applications of Cloud Computing and Big Data - ECON 446</h3>\n",
    "    <div style=\"padding: 20px 0;\">\n",
    "        <hr style=\"border: 0; height: 1px; background-image: linear-gradient(to right, rgba(0, 0, 0, 0), rgba(0, 0, 0, 0.75), rgba(0, 0, 0, 0));\">\n",
    "        <p><em>Bella Rakhlina</em><br>\n",
    "        <em>Lora Yovcheva</em><br>\n",
    "        <em>Mauricio Vargas-Estrada</em><br>\n",
    "        <br>Master Of Quantitative Economics<br>\n",
    "        University of California - Los Angeles</p>\n",
    "        <hr style=\"border: 0; height: 1px; background-image: linear-gradient(to right, rgba(0, 0, 0, 0), rgba(0, 0, 0, 0.75), rgba(0, 0, 0, 0));\">\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0399a55e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import string\n",
    "import re\n",
    "import toolz\n",
    "import time\n",
    "import pickle as pkl\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "from multiprocessing import Pool\n",
    "from tqdm import tqdm\n",
    "from rich import inspect\n",
    "from pprint import pprint\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.firefox.options import Options\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.common.exceptions import NoSuchElementException\n",
    "\n",
    "from stem import Signal\n",
    "from stem.control import Controller\n",
    "\n",
    "import tbselenium.common as cm\n",
    "from tbselenium.utils import launch_tbb_tor_with_stem\n",
    "from tbselenium.tbdriver import TorBrowserDriver\n",
    "\n",
    "import pickle as pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caecd454",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Web Crawling Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaaae8c8",
   "metadata": {},
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       a.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Create a list of links for all the wikipedia pages for NYSE traded companies A-Z and 0-9.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d4b8204f",
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {\n",
    "    'Accept-Encoding': 'gzip, deflate, br',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "65204745",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "url = lambda x: f'https://en.wikipedia.org/wiki/Companies_listed_on_the_New_York_Stock_Exchange_({x})'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "22f82802",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def get_table_info(soup):\n",
    "    links = []\n",
    "    names = []\n",
    "    tickers = []\n",
    "\n",
    "    # For each row in the table of companies.\n",
    "    for r in soup.find_all('tr'):\n",
    "        # Get the columns for the row `r`.\n",
    "        row = r.find_all('td')\n",
    "        # Check if the row contains data.\n",
    "        if row:\n",
    "            # Extract the link of the company in the row `r`.\n",
    "            tmp_link = row[0].find_all('a')\n",
    "            # Check if the company has a link.\n",
    "            if tmp_link:\n",
    "                names.append(tmp_link[0].text)\n",
    "                # Adding the root to the relative path.\n",
    "                links.append('https://www.wikipedia.org' + tmp_link[0].get('href'))\n",
    "                tickers.append(row[1].text.strip())\n",
    "    return names, tickers, links        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b00ae2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "links = []\n",
    "names = []\n",
    "tickers = []\n",
    "for letter in tqdm(list(string.ascii_uppercase) + ['0-9']):\n",
    "    req = requests.get(url(letter), headers=headers, timeout=20)\n",
    "    # Pass if the request is not successful or time out.\n",
    "    if not req.ok:\n",
    "        continue\n",
    "    soup = BeautifulSoup(req.content, 'html.parser')\n",
    "    tmp_n, tmp_t, tmp_l= get_table_info(soup)\n",
    "    links.extend(tmp_l)\n",
    "    names.extend(tmp_n)\n",
    "    tickers.extend(tmp_t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24905206",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "companies = pd.DataFrame({'company': names, 'ticker': tickers, 'link': links})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "56f59937",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "#companies.to_csv('data/companies_links.csv', index=False)\n",
    "#companies = pd.read_csv('data/companies_links.csv')\n",
    "companies = companies.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7fbfb8e8",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   company ticker  \\\n",
      "0  A. O. Smith Corporation    AOS   \n",
      "1       A10 Networks, Inc.   ATEN   \n",
      "2          AAR Corporation    AIR   \n",
      "3             Aaron's Inc.    AAN   \n",
      "4                 ABB LTD.    ABB   \n",
      "\n",
      "                                             link  \n",
      "0      https://www.wikipedia.org/wiki/A._O._Smith  \n",
      "1     https://www.wikipedia.org/wiki/A10_Networks  \n",
      "2         https://www.wikipedia.org/wiki/AAR_Corp  \n",
      "3  https://www.wikipedia.org/wiki/Aaron%27s,_Inc.  \n",
      "4        https://www.wikipedia.org/wiki/ABB_Group  \n"
     ]
    }
   ],
   "source": [
    "print(companies.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942e9b61",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       b.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Crawl through all the URLs and make 1 DF with all the NYSE publically traded companies.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db03ca95",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def get_type(url):\n",
    "    headers = {\n",
    "        'Accept-Encoding': 'gzip, deflate, br',\n",
    "        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246'\n",
    "    }\n",
    "    # Crawling the wiki page of the company. Rise and exception if the url\n",
    "    # is broken.\n",
    "    try:\n",
    "        req = requests.get(url, headers=headers, timeout=20)\n",
    "    except:\n",
    "        print(f'Request failed in {url}')\n",
    "        return ''\n",
    "    # Pass if the request is not successful or time out.\n",
    "    if not req.ok:\n",
    "        return ''\n",
    "    soup = BeautifulSoup(req.content, 'html.parser')\n",
    "    # Extracting the type of the company from the infobox vcard. Rise an\n",
    "    \n",
    "    try:\n",
    "        vcard = soup.find_all('table', {'class': 'infobox vcard'})[0]\n",
    "        comp_type = vcard.find('a', {'title': 'Public company'}).text\n",
    "    except:\n",
    "        print(f'No company type in {url}')\n",
    "        return ''\n",
    "    return comp_type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "465ad9b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiprocessing to crawl the websites in parallel. \n",
    "with Pool(100) as p:\n",
    "    comp_type = p.map(get_type, companies['link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2994e581",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Type of companies in lowercase.\n",
    "companies['type'] = [x.lower() for x in comp_type]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0416853c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because the format is not uniform between wikis, extract the word 'public'\n",
    "# from those companies that have that pattern in their type.\n",
    "pattern = re.compile(r'.*public.*')\n",
    "companies['type'] = [\n",
    "    pattern.sub('public', s) \n",
    "    if pattern.match(s) else s for s in companies.type\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3da41eb6",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "#companies.to_csv('data/companies_links_with_type.csv', index=False)\n",
    "#companies = pd.read_csv('data/companies_links_with_type.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4ec30447",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           company ticker  \\\n",
      "0          A. O. Smith Corporation    AOS   \n",
      "1               A10 Networks, Inc.   ATEN   \n",
      "2                  AAR Corporation    AIR   \n",
      "3                     Aaron's Inc.    AAN   \n",
      "4                         ABB LTD.    ABB   \n",
      "...                            ...    ...   \n",
      "1522  Zimmer Biomet Holdings, Inc.    ZBH   \n",
      "1524                   Zoetis Inc.    ZTS   \n",
      "1525                   Zuora, Inc.    ZUO   \n",
      "1526        3D Systems Corporation    DDD   \n",
      "1527                    3M Company    MMM   \n",
      "\n",
      "                                                link    type  \n",
      "0         https://www.wikipedia.org/wiki/A._O._Smith  public  \n",
      "1        https://www.wikipedia.org/wiki/A10_Networks  public  \n",
      "2            https://www.wikipedia.org/wiki/AAR_Corp  public  \n",
      "3     https://www.wikipedia.org/wiki/Aaron%27s,_Inc.  public  \n",
      "4           https://www.wikipedia.org/wiki/ABB_Group  public  \n",
      "...                                              ...     ...  \n",
      "1522    https://www.wikipedia.org/wiki/Zimmer_Biomet  public  \n",
      "1524           https://www.wikipedia.org/wiki/Zoetis  public  \n",
      "1525            https://www.wikipedia.org/wiki/Zuora  public  \n",
      "1526       https://www.wikipedia.org/wiki/3D_Systems  public  \n",
      "1527               https://www.wikipedia.org/wiki/3M  public  \n",
      "\n",
      "[1015 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "# Companies that are public according to their wiki page.\n",
    "print(companies.query(\"type=='public'\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dedd0386",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       c.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        What is the percetages of companies that contain 1 letter, 2 letters, 3 letters, 4 letters, 5 letters,... in the ticker (drop punctuation)?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "81c83a79",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Cleaning the ticker column.\n",
    "companies['clean_ticker'] = [\n",
    "    re.match(r\"[^,.!? ]+\", x).group(0)\n",
    "    for x in companies.ticker\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f52bb4f6",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Calculating the length of the ticker.\n",
    "companies['len_ticker'] = [len(x) for x in companies.clean_ticker]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "10cbc024",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Counting the percentage of companies with each length of ticker.\n",
    "len_ticker_pct = toolz.pipe(\n",
    "    companies[['clean_ticker', 'len_ticker']],\n",
    "    lambda x: x.drop_duplicates(),\n",
    "    lambda x: x.groupby('len_ticker'),\n",
    "    lambda x: x.count(),\n",
    "    lambda x: x * 100 / x.sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "21f8a324",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    -----------------------------------------\n",
      "        Length of the ticker: 1\n",
      "        % in the total of companies: 1.52\n",
      "    -----------------------------------------\n",
      "    \n",
      "\n",
      "    -----------------------------------------\n",
      "        Length of the ticker: 2\n",
      "        % in the total of companies: 10.13\n",
      "    -----------------------------------------\n",
      "    \n",
      "\n",
      "    -----------------------------------------\n",
      "        Length of the ticker: 3\n",
      "        % in the total of companies: 72.58\n",
      "    -----------------------------------------\n",
      "    \n",
      "\n",
      "    -----------------------------------------\n",
      "        Length of the ticker: 4\n",
      "        % in the total of companies: 15.76\n",
      "    -----------------------------------------\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "for i, row in len_ticker_pct.iterrows():\n",
    "    tmp = f'''\n",
    "    -----------------------------------------\n",
    "        Length of the ticker: {i}\n",
    "        % in the total of companies: {row.clean_ticker:0.2f}\n",
    "    -----------------------------------------\n",
    "    '''\n",
    "    print(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192d97d8",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Web Scraping Using Beautiful Soup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b359afbe",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       a.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Using Beautiful soup .findAll method you will webscrape the front page of Reddit. Get a list of all of the \"timestamps\"\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fc666a69",
   "metadata": {},
   "outputs": [],
   "source": [
    "# User agent to simulate a browser in selenium.\n",
    "options = Options()\n",
    "options.set_preference(\"general.useragent.override\", \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f80ac246",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Initialize the browser.\n",
    "driver = webdriver.Firefox(options=options)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db811e6d",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Open the website in the browser.\n",
    "driver.get(\"https://www.reddit.com\")\n",
    "time.sleep(5)\n",
    "\n",
    "# Verifying the numbers of posts extracted from the page.\n",
    "num_posts_before = 0\n",
    "\n",
    "# posts inside the website to scroll down.\n",
    "body = driver.find_element(By.TAG_NAME, 'body')\n",
    "\n",
    "start_time = time.time()\n",
    "time_limit_scroll = 600 # 10 minutes\n",
    "\n",
    "# Scrolling down the page until the time limit is reached.\n",
    "print('Scrolling down the page...')\n",
    "while time.time() - start_time < time_limit_scroll:\n",
    "    print(f'Time elapsed: {time.time() - start_time:.2f} seconds')\n",
    "    # Extracting the articles from the page.\n",
    "    posts = driver.find_elements(By.XPATH, '//article[@class=\"w-full m-0\"]')\n",
    "    \n",
    "    # Verifying that the number of posts is changing in each iteration.\n",
    "    if len(posts) == num_posts_before:\n",
    "        print('Something went wrong. The number of posts is not changing. Stopping..')\n",
    "        break  \n",
    "    else:\n",
    "        num_posts_before = len(posts)\n",
    "\n",
    "    # Simulate pressing the END key to scroll down the page.\n",
    "    body.send_keys(Keys.END)\n",
    "\n",
    "    # Wait for the page to load.\n",
    "    time.sleep(6)\n",
    "print('Time limit reached.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be161e15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracting the html for each post.\n",
    "html_content = [x.get_attribute('outerHTML') for x in posts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf9e352",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the html content to a pickle file.\n",
    "with open('data/reddit_posts.pkl', 'wb') as f:\n",
    "    pickle.dump(html_content, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95a1ea31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Close the browser.\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c735653d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the html content from the pickle file to avoid running the code again.\n",
    "#with open('data/reddit_posts.pkl', 'rb') as f:\n",
    "    #html_content = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4fc2b102",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Parsing each post with BeautifulSoup.\n",
    "html_content = [BeautifulSoup(x, 'html.parser') for  x in html_content]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f8cba774",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Extracting only the timestamps for each post.\n",
    "timestamps = [x.find('time').get_attribute_list('datetime')[0] for x in html_content]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ab6f58e8",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of posts extracted: 2478\n",
      "First 10 timestamps:\n",
      "['2024-04-19T07:10:50.758Z',\n",
      " '2024-04-19T17:25:09.146Z',\n",
      " '2024-04-19T13:10:37.611Z',\n",
      " '2024-04-19T11:20:55.005Z',\n",
      " '2024-04-19T15:02:59.261Z',\n",
      " '2024-04-19T19:17:01.403Z',\n",
      " '2024-04-19T02:00:27.276Z',\n",
      " '2024-04-19T11:38:59.637Z',\n",
      " '2024-04-19T19:45:27.032Z']\n"
     ]
    }
   ],
   "source": [
    "print(f'Total number of posts extracted: {len(timestamps)}')\n",
    "print('First 10 timestamps:')\n",
    "pprint(timestamps[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee84c5d3",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       b.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Using the functions findChild, descendents, etc. locate the post title, text and post time into a dataframe.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3c876db1",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "titles = [x.find('article').get_attribute_list('aria-label')[0] for x in html_content]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2a617e8a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "text_div_class = re.compile(r'.*feed-card-text-preview.*')\n",
    "texts = []\n",
    "for post in html_content:\n",
    "    tmp = post.find_all(\n",
    "        'div',\n",
    "        class_ = lambda x : x and text_div_class.match(x)    \n",
    "    )\n",
    "    if tmp:\n",
    "        tmp = tmp[0].text\n",
    "    else:\n",
    "        tmp = np.nan\n",
    "    texts.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bd66e732",
   "metadata": {},
   "outputs": [],
   "source": [
    "post_df = pd.DataFrame({\n",
    "    'timestamp': timestamps,\n",
    "    'title': titles,\n",
    "    'text': texts\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "19f532d5",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First 5 posts with text:\n",
      "                   timestamp  \\\n",
      "2   2024-04-19T13:10:37.611Z   \n",
      "21  2024-04-19T15:55:01.669Z   \n",
      "23  2024-04-19T16:35:15.247Z   \n",
      "28  2024-04-19T05:30:53.159Z   \n",
      "34  2024-04-19T15:22:26.286Z   \n",
      "\n",
      "                                                title  \\\n",
      "2   My husband won't let me take more than two sho...   \n",
      "21  AITAH for breaking up with my bf after he alle...   \n",
      "23  After years of tipping 20-25% I’m DONE. I’m ti...   \n",
      "28  Orbital cooldowns are way too long compared to...   \n",
      "34  Is it just me or is the new Taylor swift album...   \n",
      "\n",
      "                                                 text  \n",
      "2   \\n\\n    This is the weirdest thing my husband ...  \n",
      "21  \\n\\n    I’m not a clubbing kind of girl. My bf...  \n",
      "23  \\n\\n    I have always “over tipped” according ...  \n",
      "28  \\n\\n    Orbital Precision Strike has a 100s co...  \n",
      "34  \\n\\n    I'm not here to be a hater but I felt ...  \n"
     ]
    }
   ],
   "source": [
    "# Printing the first 10 posts with text.\n",
    "print('First 5 posts with text:')\n",
    "print(post_df.query('text.notna()').head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c90f56f",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## RegEx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215da9db",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       a.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Using RegEx, get all the urls of ladder faculty profiles for UCLA Economics\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "99aa443f",
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"https://economics.ucla.edu/faculty/ladder\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0be23f9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting the flex table of the ladder faculty.\n",
    "req_ladder = requests.get(URL, headers=headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01777e86",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Extracting the name and url for each profile in the ladder faculty.\n",
    "tmp = re.compile(r'flex_column av_one_fourth.*')\n",
    "profiles_info = toolz.pipe(\n",
    "    req_ladder,\n",
    "    lambda x: x.content,\n",
    "    lambda x: BeautifulSoup(x, 'html.parser'),\n",
    "    lambda x : x.find(\n",
    "        'div',\n",
    "        {'id': 'wpv-view-layout-974-CATTR0494cfbfb8d3e1b3152203680573333f'}\n",
    "    ),\n",
    "    lambda x: x.find_all(\n",
    "        'div',\n",
    "        class_ = lambda x : x and tmp.match(x)\n",
    "    ),\n",
    "    lambda x: [y.find('h3') for y in x],\n",
    "    lambda x: {\n",
    "        'name': [y.text for y in x],\n",
    "        'url': [y.find('a')['href'] for y in x]\n",
    "    },\n",
    "    lambda x: pd.DataFrame(x)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5201c24e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       b.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Webcrawl the links from A and use RegEx to get all the emails and phone numbers of ladder faculty profiles.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e1f550b",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "# Crawling the profiles urls.\n",
    "crawled_info = []\n",
    "for url in tqdm(profiles_info['url']):\n",
    "    tmp = requests.get(url, headers=headers).content\n",
    "    crawled_info.append(tmp)\n",
    "    time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d42af54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracting the email and phone from the crawled info using regex.\n",
    "regex_email = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+')\n",
    "regex_phone = re.compile(r'([\\+]?[(]?[0-9]{3}[)]?[-\\s\\.][0-9]{3}[-\\s\\.][0-9]{4,6})')\n",
    "email = []\n",
    "phone = []\n",
    "for raw in crawled_info:\n",
    "    tmp = BeautifulSoup(raw, 'html.parser')\n",
    "    tmp_email = tmp.find(string=regex_email)\n",
    "    tmp_phone = tmp.find(string=regex_phone)\n",
    "\n",
    "    if tmp_email:\n",
    "        email.append(tmp_email)\n",
    "    else:\n",
    "        email.append(np.nan)\n",
    "    \n",
    "    if tmp_phone:\n",
    "        phone.append(tmp_phone)\n",
    "    else:\n",
    "        phone.append(np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84a7b2ee",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "profiles_info['email'] = email\n",
    "profiles_info['phone'] = phone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fdb4d363",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                    name                                                url  \\\n",
      "8         Ariel Burstein  https://economics.ucla.edu/person/ariel-burstein/   \n",
      "12      Pablo Fajgelbaum  https://economics.ucla.edu/person/pablo-fajgel...   \n",
      "17    Martin B. Hackmann  https://economics.ucla.edu/person/martin-b-hac...   \n",
      "18          Jinyong Hahn    https://economics.ucla.edu/person/jinyong-hahn/   \n",
      "20        Hugo Hopenhayn  https://economics.ucla.edu/person/hugo-hopenhayn/   \n",
      "23  Adriana Lleras-Muney  https://economics.ucla.edu/person/adriana-ller...   \n",
      "25                Jay Lu          https://economics.ucla.edu/person/jay-lu/   \n",
      "26          Rosa Matzkin  https://economics.ucla.edu/person/rosa-liliana...   \n",
      "27     Maurizio Mazzocco  https://economics.ucla.edu/person/maurizio-maz...   \n",
      "28      Kathleen McGarry  https://economics.ucla.edu/person/kathleen-mcg...   \n",
      "\n",
      "                        email            phone  \n",
      "8        arielb@econ.ucla.edu   (310) 206-6732  \n",
      "12  pfajgelbaum@econ.ucla.edu   (310) 794-7241  \n",
      "17     hackmann@econ.ucla.edu   (310) 825-1011  \n",
      "18         hahn@econ.ucla.edu   (310) 825-1011  \n",
      "20        hopen@econ.ucla.edu   (310) 206-8896  \n",
      "23      alleras@econ.ucla.edu   (310) 825-3925  \n",
      "25          jay@econ.ucla.edu   (310) 825-7380  \n",
      "26      matzkin@econ.ucla.edu   (310) 825-7371  \n",
      "27     mmazzocc@econ.ucla.edu   (310) 825-6682  \n",
      "28           mcgarry@ucla.edu   (310) 825-1011  \n"
     ]
    }
   ],
   "source": [
    "# Printing those profiles with phone number.\n",
    "print(profiles_info.query('phone.notna()').head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a56f295e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Number of profiles: 45\n",
      "Number of profiles with phone number: 16\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tmp = f\"\"\"\n",
    "Number of profiles: {profiles_info.shape[0]}\n",
    "Number of profiles with phone number: {profiles_info.query('phone.notna()').shape[0]}\n",
    "\"\"\"\n",
    "print(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14d1182b",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "## Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e875d32b",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       a.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        <p>Pick a website that has useful data to a business or economic question. Put your website you plan to scrape here:</p>\n",
    "        <p>https://docs.google.com/spreadsheets/d/1PJ2DOTCVCh51fn0ry1yB7qTyccR33_IXFpkYdd58MFs/edit?usp=sharing</p>\n",
    "        <p>You must have use website that no other group has. First come first serve.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f1925f",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "The selected website, [www.realtor.com](https://www.realtor.com/), offers listings for properties for sale and rent, including those around the UCLA campus, which is our area of interest. We are particularly focused on gathering rental information from the following neighborhoods:\n",
    "\n",
    "- Bel Air\n",
    "- Brentwood\n",
    "- Culver City\n",
    "- Encino\n",
    "- Mar Vista\n",
    "- Mid Wilshire\n",
    "- Pacific Palisades\n",
    "- Palms\n",
    "- Playa del Rey\n",
    "- Playa Vista\n",
    "- Santa Monica\n",
    "- Sawtelle\n",
    "- Sherman Oaks\n",
    "- Studio City\n",
    "- Venice\n",
    "- West Los Angeles\n",
    "- Westwood\n",
    "\n",
    "To facilitate webscraping, we are integrating `Selenium` with `Tor`. The following code snippet is utilized to extract the required information from the website."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d00f7a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       b.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Use Selenium to scrape valuable information from your website and store in a dataframe.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59bc7ee",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "> In this section it is defined the functions that will be used to scrape the website. The functions are defined in the following order:\n",
    "> - `save_data`: Function to save data to a pickle file.\n",
    "> - `random_sleep`: Function to generate a random sleep time to mimic human behavior.\n",
    "> - `human_like_scroll`: Function to simulate human-like scrolling behavior more slowly.\n",
    "> - `switch_tor_circuit`: Function to switch the Tor circuit.\n",
    "> - `get_tor_version`: Function to get the Tor version.\n",
    "> - `get_current_circuit`: Function to retrieve and print the current Tor circuit.\n",
    "> - `zipcode_scrapping`: Function to scrape the website using Tor and Selenium.\n",
    "> After defining the functions, the neighborhoods of interest are scraped using the `zipcode_scrapping` function. The results are saved to pickle files for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c4238d9",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def save_data(data, filename):\n",
    "    script_directory = os.path.dirname(os.path.abspath(__file__))\n",
    "    file_path = os.path.join(script_directory, filename)\n",
    "    with open(file_path, 'wb') as file:\n",
    "        pkl.dump(data, file)\n",
    "    print(f\"Data saved to {file_path}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfbdaa13",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def random_sleep(minimum=2, maximum=5):\n",
    "    \"\"\"Generate a random sleep time to mimic human behavior.\"\"\"\n",
    "    time.sleep(random.uniform(minimum, maximum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99c66a41",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def human_like_scroll(driver):\n",
    "    \"\"\"Simulate human-like scrolling behavior more slowly.\"\"\"\n",
    "    total_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
    "    current_scroll_position = 0\n",
    "    increment = total_height / 20  # Divide the scroll into smaller steps\n",
    "\n",
    "    while current_scroll_position <= total_height:\n",
    "        # Scroll down to the next increment\n",
    "        driver.execute_script(f\"window.scrollTo(0, {current_scroll_position});\")\n",
    "        current_scroll_position += increment\n",
    "\n",
    "        # Wait a random time between scrolls to mimic human behavior\n",
    "        time.sleep(random.uniform(0.5, 3))  # Adjust timing as needed\n",
    "\n",
    "    # Finally, scroll to the very bottom to ensure all lazy loaded items are triggered\n",
    "    driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
    "    time.sleep(random.uniform(15, 20))  # A final pause at the bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "482aa90e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def switch_tor_circuit():\n",
    "    \"\"\"Use Stem to switch Tor circuit.\"\"\"\n",
    "    with Controller.from_port(port=9251) as controller:\n",
    "        controller.authenticate()\n",
    "        controller.signal(Signal.NEWNYM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9394ad88",
   "metadata": {
    "lines_to_next_cell": 1
   },
   "outputs": [],
   "source": [
    "def get_tor_version():\n",
    "    try:\n",
    "        with Controller.from_port(port=9251) as controller:\n",
    "            controller.authenticate()  # Asume que no se necesita contraseña\n",
    "            version = controller.get_version()\n",
    "            print(\"Connected to Tor version:\", version)\n",
    "    except Exception as e:\n",
    "        print(f\"Error connecting to Tor control port: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed8cec9f",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def get_current_circuit():\n",
    "    \"\"\"Retrieve and print the current Tor circuit.\"\"\"\n",
    "    try:\n",
    "        with Controller.from_port(port=9251) as controller:\n",
    "            controller.authenticate()  # Assumes no password is needed\n",
    "            for circ in controller.get_circuits():\n",
    "                if circ.status == 'BUILT':\n",
    "                    print(\"Circuit ID: {}\".format(circ.id))\n",
    "                    print(\"Circuit Path:\")\n",
    "                    for i, entry in enumerate(circ.path):\n",
    "                        desc = controller.get_network_status(entry[0], None)\n",
    "                        fingerprint, nickname = entry\n",
    "                        address = desc.address if desc else 'unknown'\n",
    "                        print(f\"  {i+1}: {nickname} ({fingerprint}) at {address}\")\n",
    "                    print(\"\\n\")\n",
    "                    break  # Just show the first 'BUILT' circuit\n",
    "    except Exception as e:\n",
    "        print(f\"Error retrieving current circuit: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4e68029",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "def zipcode_scrapping(zipcode):\n",
    "    tor_path = '/home/m4wnn/tor-browser-linux-x86_64-13.0.14/tor-browser'\n",
    "    while True:\n",
    "        try:\n",
    "            tor_process = launch_tbb_tor_with_stem(tbb_path=tor_path)\n",
    "            break\n",
    "        except Exception as e:\n",
    "            print(f\"Error initializing Tor process: {e}\")\n",
    "            print(\"Retrying to initialize.\")\n",
    "    print(\"Tor process initialized.\")\n",
    "            \n",
    "\n",
    "    ## Setting a random user-agent using pref_dict\n",
    "    user_agent_list = [\n",
    "        \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36\",\n",
    "        \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15\",\n",
    "        \"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko\",\n",
    "        \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0\",\n",
    "        \"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\",\n",
    "        \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\",\n",
    "        \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36\",\n",
    "        \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36\",\n",
    "        \"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0\",\n",
    "        \"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36\"\n",
    "    ]\n",
    "\n",
    "    random_user_agent = random.choice(user_agent_list)\n",
    "    pref_dict = {\"general.useragent.override\": random_user_agent}\n",
    "\n",
    "    # Create a TorBrowserDriver instance with customized user-agent\n",
    "    try:\n",
    "        driver = TorBrowserDriver(\n",
    "            tor_path,\n",
    "            tor_cfg=cm.USE_STEM,\n",
    "            pref_dict=pref_dict,\n",
    "            headless=True\n",
    "        )\n",
    "    except Exception as e:\n",
    "        print(f\"Error creating TorBrowserDriver: {e}\")\n",
    "        return []\n",
    "    \n",
    "    time.sleep(3)\n",
    "    #driver.get(f'https://www.realtor.com/realestateandhomes-search/{zipcode}')\n",
    "    driver.get(f'https://www.realtor.com/apartments/{zipcode}')\n",
    "    tmp_current_url = driver.current_url\n",
    "    print(tmp_current_url)\n",
    "    \n",
    "    random_sleep(5, 10)\n",
    "    \n",
    "    \n",
    "    \n",
    "    regex_property = re.compile(r'placeholder_property_\\d*')\n",
    "    property_info = []\n",
    "    try:\n",
    "        refresh_count = 0\n",
    "        while True:\n",
    "            human_like_scroll(driver)\n",
    "\n",
    "            try:\n",
    "                property_section = driver.find_element(\n",
    "                    By.CSS_SELECTOR,\n",
    "                    'section[class^=\"PropertiesList_propertiesContainer\"]'\n",
    "                )\n",
    "                section_content = BeautifulSoup(\n",
    "                    property_section.get_attribute('innerHTML'),\n",
    "                    'html.parser'\n",
    "                )\n",
    "\n",
    "                properties = section_content.find_all(\n",
    "                    'div',\n",
    "                    id=lambda x: x and regex_property.match(x)\n",
    "                )\n",
    "                \n",
    "                property_htmls = [str(prop) for prop in properties]\n",
    "                property_info.extend(property_htmls)\n",
    "                \n",
    "\n",
    "            except NoSuchElementException:\n",
    "                if refresh_count == 2:\n",
    "                    print(\"Refreshing limit reached. Exiting loop.\")\n",
    "                    break\n",
    "                print(\"Property section not found.\")\n",
    "                print(\"Changing circuit.\")\n",
    "                switch_tor_circuit()\n",
    "                print(\"Refreshing Website.\")\n",
    "                refresh_count += 1\n",
    "                driver.refresh()\n",
    "                continue \n",
    "            \n",
    "            refresh_count = 0\n",
    "            try:\n",
    "                tmp_current_url = driver.current_url\n",
    "                switch_tor_circuit()  # Switch Tor circuit before loading the page\n",
    "                time.sleep(15)  # Wait for the new circuit to be established\n",
    "                next_button = driver.find_element(By.LINK_TEXT, \"Next\")\n",
    "                next_button.click()\n",
    "                random_sleep(3, 6)\n",
    "                tmp_new_url = driver.current_url\n",
    "                if tmp_current_url == tmp_new_url:\n",
    "                    print(\"Same page detected. Exiting loop.\")\n",
    "                    break\n",
    "                print(tmp_new_url)\n",
    "            except NoSuchElementException:\n",
    "                print(\"Next button not found. Exiting loop.\")\n",
    "                break\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Unexpected error: {e}\")\n",
    "    finally:\n",
    "        driver.quit()\n",
    "        tor_process.kill()\n",
    "\n",
    "    return property_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7b1d575",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "neighborhoods = [\n",
    "    'Bel-Air_Los-Angeles_CA',\n",
    "    'Brentwood_Los-Angeles_CA',\n",
    "    'Culver-City_CA',\n",
    "    'Encino_Los-Angeles_CA',\n",
    "    'Mar-Vista_Los-Angeles_CA',\n",
    "    'Mid-Wilshire_Los-Angeles_CA',\n",
    "    'Pacific-Palisades_Los-Angeles_CA',\n",
    "    'Palms_Los-Angeles_CA',\n",
    "    'Playa-del-Rey_Los-Angeles_CA',\n",
    "    'Playa-Vista_Los-Angeles_CA',\n",
    "    'Santa-Monica_CA',\n",
    "    'Sawtelle_Los-Angeles_CA',\n",
    "    'Sherman-Oaks_Los-Angeles_CA',\n",
    "    'Studio-City_Los-Angeles_CA',\n",
    "    'Venice_CA',\n",
    "    'West-Los-Angeles_CA',\n",
    "    'Westwood_Los-Angeles_CA'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c8393b6",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "for n in neighborhoods:\n",
    "    results = zipcode_scrapping(n)\n",
    "    # Saving the results to a pickle file for further analysis.\n",
    "    save_data(results, f'results/{n}_results.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec2e9ff4",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "> In this section, the information extracted from the website is joined and cleaned. The data is then saved to a CSV file for further analysis.\n",
    "> Some properties have a range for the number of bedrooms, bathrooms, and area. In these cases, the minimum and maximum values are extracted and stored in separate columns. The same is done for the rent.\n",
    "> Even though the previous section extracted the information, for simplicity of development, the following conde loads the information from the pickle files, this way the code can be run without the need to scrape the website again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f4abb38",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "FILES_PATH = os.path.join(\n",
    "    'results'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8237bafb",
   "metadata": {
    "lines_to_next_cell": 0,
    "title": "Joining the information for each neighborhood."
   },
   "outputs": [],
   "source": [
    "raw = []\n",
    "ngh = []\n",
    "\n",
    "re_ngh = re.compile(r'\\S+(?=_results\\.pkl)')\n",
    "\n",
    "for file in os.listdir(FILES_PATH):\n",
    "    tmp_file = os.path.join(FILES_PATH, file)\n",
    "    \n",
    "    with open(tmp_file, 'rb') as f:\n",
    "        tmp_info = pkl.load(f)\n",
    "    \n",
    "    raw.extend(tmp_info)\n",
    "    ngh.extend(re_ngh.findall(file) * len(tmp_info))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7faf1da9",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "raw_soup = [BeautifulSoup(x, 'html.parser') for x in raw]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94bb1e7f",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "rent = [\n",
    "    x.find('div', {'data-testid':'card-price'}).text\n",
    "    for x in raw_soup\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60505ddd",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "rent_min = [\n",
    "    re.findall(r'(?<=^\\$)\\d+,\\d+(?=$|\\s\\-)', x)\n",
    "    for x in rent\n",
    "]\n",
    "rent_min = [x[0] if x else '' for x in rent_min]\n",
    "rent_min = [x.replace(',', '') if x != '' else '' for x in rent_min]\n",
    "rent_min = [float(x) if x != '' else np.nan for x in rent_min]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a45379d",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "rent_max = [\n",
    "    re.findall(r'(?<=\\s\\-\\s\\$)\\d+,\\d+', x)\n",
    "    for x in rent\n",
    "]\n",
    "rent_max = [x[0] if x else '' for x in rent_max]\n",
    "rent_max = [x.replace(',', '') if x != '' else '' for x in rent_max]\n",
    "rent_max = [float(x) if x != '' else np.nan for x in rent_max]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f73c0b",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_beds = [\n",
    "    x.find('li', {'data-testid':'property-meta-beds'})\n",
    "    for x in raw_soup\n",
    "]\n",
    "n_beds = [x.text if x else '' for x in n_beds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8449e5fc",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_beds_min = [\n",
    "    re.findall(r'(?<=^)(\\d+|Studio)', x)\n",
    "    for x in n_beds\n",
    "]\n",
    "n_beds_min = [x[0] if x else np.nan for x in n_beds_min]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "756e4f1e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_beds_max = [\n",
    "    re.findall(r'(?<=\\-\\s)\\d+', x)\n",
    "    for x in n_beds\n",
    "]\n",
    "n_beds_max = [x[0] if x else np.nan for x in n_beds_max]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db6a3c2a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_baths = [\n",
    "    x.find('li', {'data-testid':'property-meta-baths'})\n",
    "    for x in raw_soup\n",
    "]\n",
    "n_baths = [x.text if x else '' for x in n_baths]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b3f8abf",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_baths_min = [\n",
    "    re.findall(r'(?<=^)\\d+', x)\n",
    "    for x in n_baths\n",
    "]\n",
    "n_baths_min = [x[0] if x else np.nan for x in n_baths_min]\n",
    "n_baths_min = [float(x) if x != np.nan else np.nan for x in n_baths_min]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f997f90",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "n_baths_max = [\n",
    "    re.findall(r'(?<=\\-\\s)\\d+', x)\n",
    "    for x in n_baths\n",
    "]\n",
    "n_baths_max = [x[0] if x else np.nan for x in n_baths_max]\n",
    "n_baths_max = [float(x) if x != np.nan else np.nan for x in n_baths_max]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbf2e925",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "area = [\n",
    "    x.find('li', {'data-testid':'property-meta-sqft'})\n",
    "    for x in raw_soup\n",
    "]\n",
    "area = [x.text if x else '' for x in area]\n",
    "area =[\n",
    "    re.findall(r'(?<=sqft).+(?=\\ssquare\\sfeet)', x)\n",
    "    if x != '' else [] for x in area\n",
    "\n",
    "]\n",
    "area = [x[0] if x else '' for x in area]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fafd30f",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "area_min = [\n",
    "    re.findall(r'(?<=^)(\\d{3}|\\d+,\\d+)', x)\n",
    "    for x in area\n",
    "]\n",
    "area_min = [x[0] if x else '' for x in area_min]\n",
    "area_min = [x.replace(',', '') if x != '' else '' for x in area_min]\n",
    "area_min = [float(x) if x != '' else np.nan for x in area_min]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75b84ed7",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "area_max = [\n",
    "    re.findall(r'(?<=\\-\\s)(\\d{3}|\\d+,\\d+)', x)\n",
    "    for x in area\n",
    "]\n",
    "area_max = [x[0] if x else '' for x in area_max]\n",
    "area_max = [x.replace(',', '') if x != '' else '' for x in area_max]\n",
    "area_max = [float(x) if x != '' else np.nan for x in area_max]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7db9fb8",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "addss_1 = [\n",
    "    x.find('div', {'data-testid':'card-address-1'})\n",
    "    for x in raw_soup\n",
    "]\n",
    "addss_1 = [x.text if x else '' for x in addss_1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb6be6e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "addss_2 = [\n",
    "    x.find('div', {'data-testid':'card-address-2'})\n",
    "    for x in raw_soup\n",
    "]\n",
    "addss_2 = [x.text if x else '' for x in addss_2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d70206",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\n",
    "    'neighborhood': ngh,\n",
    "    'rent_min': rent_min,\n",
    "    'rent_max': rent_max,\n",
    "    'n_beds_min': n_beds_min,\n",
    "    'n_beds_max': n_beds_max,\n",
    "    'n_baths_min': n_baths_min,\n",
    "    'n_baths_max': n_baths_max,\n",
    "    'area_min': area_min,\n",
    "    'area_max': area_max,\n",
    "    'address_1': addss_1,\n",
    "    'address_2': addss_2\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22632c30",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data.loc[data.rent_max.isna(), 'rent_max'] = data.loc[data.rent_max.isna(), 'rent_min']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7407482",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data.loc[data.n_beds_max.isna(), 'n_beds_max'] = data.loc[data.n_beds_max.isna(), 'n_beds_min']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9b4706b",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data.loc[data.n_baths_max.isna(), 'n_baths_max'] = data.loc[data.n_baths_max.isna(), 'n_baths_min']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1317378f",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data.loc[data.area_max.isna(), 'area_max'] = data.loc[data.area_max.isna(), 'area_min']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa7b987a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "data.to_csv(os.path.join(FILES_PATH, '..', 'neighborhoods_around_ucla.csv'), index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a7ffc1a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neighborhood</th>\n",
       "      <th>rent_min</th>\n",
       "      <th>rent_max</th>\n",
       "      <th>n_beds_min</th>\n",
       "      <th>n_beds_max</th>\n",
       "      <th>n_baths_min</th>\n",
       "      <th>n_baths_max</th>\n",
       "      <th>area_min</th>\n",
       "      <th>area_max</th>\n",
       "      <th>address_1</th>\n",
       "      <th>address_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Studio-City_Los-Angeles_CA</td>\n",
       "      <td>7600.0</td>\n",
       "      <td>7600.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2150.0</td>\n",
       "      <td>2150.0</td>\n",
       "      <td>11734 Sunshine Ter</td>\n",
       "      <td>Studio City, CA 91604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Studio-City_Los-Angeles_CA</td>\n",
       "      <td>1789.0</td>\n",
       "      <td>3276.0</td>\n",
       "      <td>Studio</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>426.0</td>\n",
       "      <td>1567.0</td>\n",
       "      <td>Ava Studio City</td>\n",
       "      <td>10979 Bluffside Dr, Los Angeles, CA 91604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Studio-City_Los-Angeles_CA</td>\n",
       "      <td>1895.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>750.0</td>\n",
       "      <td>750.0</td>\n",
       "      <td>Arch Apartments</td>\n",
       "      <td>4151 Arch Dr, Studio City, CA 91604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Studio-City_Los-Angeles_CA</td>\n",
       "      <td>2695.0</td>\n",
       "      <td>2695.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1150.0</td>\n",
       "      <td>1150.0</td>\n",
       "      <td>4418 Colfax Ave Apt 3</td>\n",
       "      <td>Studio City, CA 91602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Studio-City_Los-Angeles_CA</td>\n",
       "      <td>1850.0</td>\n",
       "      <td>2895.0</td>\n",
       "      <td>Studio</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>470.0</td>\n",
       "      <td>710.0</td>\n",
       "      <td>Alhambra at Studio City</td>\n",
       "      <td>10937 Fruitland Dr, Los Angeles, CA 91604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3856</th>\n",
       "      <td>Playa-del-Rey_Los-Angeles_CA</td>\n",
       "      <td>6590.0</td>\n",
       "      <td>6590.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2332.0</td>\n",
       "      <td>2332.0</td>\n",
       "      <td>7230 W 90th St</td>\n",
       "      <td>Los Angeles, CA 90045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3857</th>\n",
       "      <td>Playa-del-Rey_Los-Angeles_CA</td>\n",
       "      <td>8000.0</td>\n",
       "      <td>8000.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1923.0</td>\n",
       "      <td>1923.0</td>\n",
       "      <td>7301 Vista Del Mar Apt B105</td>\n",
       "      <td>Playa Del Rey, CA 90293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3858</th>\n",
       "      <td>Playa-del-Rey_Los-Angeles_CA</td>\n",
       "      <td>16500.0</td>\n",
       "      <td>16500.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6975 Trolleyway</td>\n",
       "      <td>Playa Del Rey, CA 90293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3859</th>\n",
       "      <td>Playa-del-Rey_Los-Angeles_CA</td>\n",
       "      <td>3650.0</td>\n",
       "      <td>3650.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>785.0</td>\n",
       "      <td>785.0</td>\n",
       "      <td>8650 Gulana Ave Unit 1172</td>\n",
       "      <td>Los Angeles, CA 90293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3860</th>\n",
       "      <td>Playa-del-Rey_Los-Angeles_CA</td>\n",
       "      <td>3889.0</td>\n",
       "      <td>3889.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1039.0</td>\n",
       "      <td>1039.0</td>\n",
       "      <td>8300 Manitoba St Apt 216</td>\n",
       "      <td>Playa Del Rey, CA 90293</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3861 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      neighborhood  rent_min  rent_max n_beds_min n_beds_max  \\\n",
       "0       Studio-City_Los-Angeles_CA    7600.0    7600.0          3          3   \n",
       "1       Studio-City_Los-Angeles_CA    1789.0    3276.0     Studio          3   \n",
       "2       Studio-City_Los-Angeles_CA    1895.0    1995.0          1          1   \n",
       "3       Studio-City_Los-Angeles_CA    2695.0    2695.0          2          2   \n",
       "4       Studio-City_Los-Angeles_CA    1850.0    2895.0     Studio          2   \n",
       "...                            ...       ...       ...        ...        ...   \n",
       "3856  Playa-del-Rey_Los-Angeles_CA    6590.0    6590.0          3          3   \n",
       "3857  Playa-del-Rey_Los-Angeles_CA    8000.0    8000.0          4          4   \n",
       "3858  Playa-del-Rey_Los-Angeles_CA   16500.0   16500.0          2          2   \n",
       "3859  Playa-del-Rey_Los-Angeles_CA    3650.0    3650.0          1          1   \n",
       "3860  Playa-del-Rey_Los-Angeles_CA    3889.0    3889.0          2          2   \n",
       "\n",
       "      n_baths_min  n_baths_max  area_min  area_max  \\\n",
       "0             3.0          3.0    2150.0    2150.0   \n",
       "1             1.0          2.0     426.0    1567.0   \n",
       "2             1.0          1.0     750.0     750.0   \n",
       "3             2.0          2.0    1150.0    1150.0   \n",
       "4             1.0          1.0     470.0     710.0   \n",
       "...           ...          ...       ...       ...   \n",
       "3856          3.0          3.0    2332.0    2332.0   \n",
       "3857          2.0          2.0    1923.0    1923.0   \n",
       "3858          2.0          2.0       NaN       NaN   \n",
       "3859          1.0          1.0     785.0     785.0   \n",
       "3860          2.0          2.0    1039.0    1039.0   \n",
       "\n",
       "                        address_1                                  address_2  \n",
       "0              11734 Sunshine Ter                      Studio City, CA 91604  \n",
       "1                 Ava Studio City  10979 Bluffside Dr, Los Angeles, CA 91604  \n",
       "2                 Arch Apartments        4151 Arch Dr, Studio City, CA 91604  \n",
       "3           4418 Colfax Ave Apt 3                      Studio City, CA 91602  \n",
       "4         Alhambra at Studio City  10937 Fruitland Dr, Los Angeles, CA 91604  \n",
       "...                           ...                                        ...  \n",
       "3856               7230 W 90th St                      Los Angeles, CA 90045  \n",
       "3857  7301 Vista Del Mar Apt B105                    Playa Del Rey, CA 90293  \n",
       "3858              6975 Trolleyway                    Playa Del Rey, CA 90293  \n",
       "3859    8650 Gulana Ave Unit 1172                      Los Angeles, CA 90293  \n",
       "3860     8300 Manitoba St Apt 216                    Playa Del Rey, CA 90293  \n",
       "\n",
       "[3861 rows x 11 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "401053f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Number of properties: 3861\n",
      "Number of neighborhoods: 17\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tmp = f\"\"\"\n",
    "Number of properties: {data.shape[0]}\n",
    "Number of neighborhoods: {data.neighborhood.nunique()}\n",
    "\"\"\"\n",
    "print(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acac15cb",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "<div style=\"border: 1px solid black; border-radius: 5px; overflow: hidden;\">\n",
    "    <div style=\"background-color: black; color: white; padding: 5px; text-align: left;\">\n",
    "       c.) \n",
    "    </div>\n",
    "    <div style=\"padding: 10px;\">\n",
    "        Write a short paragraph about the businesses or research that would use the data you scraped. Describe it's value and what it can be used for.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae395737",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "We picked `Realtor.com` as the website to web scrape. We find great purpose in our task is there are many opportunities to capitalize on the information that can be accessed from the website. To begin with, it contains real estate listings with homes for sale or rents in and specific filters based on the clients' preferences, providing insights for an informed decision-making process. Some important features of interest that can be pulled from the website are:\n",
    "\n",
    "- Location (zip code, address) \n",
    "- Price range\n",
    "- Type of property (house, apartment, condo, commercial)  \n",
    "- Number of bedrooms and bathrooms \n",
    "- Amenities (pet friendly, in-unit laundry, pool, gym, parking, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3115852",
   "metadata": {
    "cell_marker": "\"\"\"",
    "lines_to_next_cell": 0
   },
   "source": [
    "Overall, the information that can be scraped is very valuable and can be applied to many scenarios, such as: \n",
    "\n",
    "1. **UCLA - Market and Academic Research:** The information can be used to access the details of properties in the neighboring areas near UCLA. In the context of a market research the price levels and home size can be utilized to show trends and fluctuations in pricing based on proximity to the campus. The university can provide help for informed decision making to the students who do not have much experience in renting a space, and average expectation to avoid fraudulent situations. In terms of academia, some topics of interest might be seasonality in demand and prices based on the quarter/semester school year structure, demand of a certain type or size of a home and etc. \n",
    "2. **Real Estate Market Analysis:** Can aid real estate agencies and investors on the properties in demand. For example, if there are many large houses in the area and a shortage of affordable apartment buildings, which are of preference near a university, how can market opportunities be identified both for renters and investors in properties, often represented by real estate agencies. \n",
    "3. **Urban Planning and Development:** This is another application that differentiates more from the previous two as the main benefit is in terms of public services, development, and infrastructure projects. Based on housing density and types, regulators can make informed decisions to upgrade the conditions in the area. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "634b1c99",
   "metadata": {
    "cell_marker": "\"\"\""
   },
   "source": [
    "It is important to note that the information in Realtor.com is detailed and valuable for the above said uses. As their source of business, the owners of the website have used various levels of protection in order to prevent scraping from bots, which made our goal much more complex. "
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "title,-all",
   "formats": "py:percent,ipynb",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "webscrapping",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}