0.2-api-calls-users.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03dd9fed-fd7f-414b-b883-a17bb71b7aa4",
   "metadata": {},
   "source": [
    "## Data for Colaborative Model Utility Matrix: (User-Item)\n",
    "\n",
    "Using ravelry.com's api.  Limit of 100k per call. There are over 10 million users and over 18 million projects (items).  Need to reduce the size. After some exploration, have decided on getting only projects for patterns that have over threshold of projects (600), and use those to get the users.  Will filter down users after I have the data in order to ensure the utlity matrix is not too sparse. \n",
    "\n",
    "https://www.ravelry.com/api#index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c89fe5c2-e542-4531-aeab-9f1414dfbb28",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import json\n",
    "import requests\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from pprint import pprint\n",
    "\n",
    "from config import basic_auth_username, basic_auth_password\n",
    "from config import basic_auth_username_read_only, basic_auth_password_read_only"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0e64278-a6e4-4632-9811-9644bdf01302",
   "metadata": {},
   "source": [
    "#### Get a the list of patterns from which to pull projects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c6e15e0a-ccda-4b4c-b050-865e30aa04bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data\n",
    "df = pd.read_csv('data/consolidated_patterns.csv', low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f8a0d2-bb36-49be-8601-2132bc7bd042",
   "metadata": {},
   "source": [
    "After experimenting with the project count, I decided to start with 600 as the cut off as some of these patterns have over 30k projects completed, and can only pull in 500 at a time, so start high and can work to include more patterns after I get a minimum viable product. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2278f441-6b0d-4015-a01b-7118e929b6b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3197, 24)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['projects_count'] >= 601].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a2eeec79-d257-455e-a3d6-eb29d9c6be63",
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_out_bad_data(df):\n",
    "    \"\"\" Remove any patterns that will result in a more sparse matrix \"\"\"\n",
    "    # extra indicies from .csv appending and accidental douplicates\n",
    "    df = df.drop(df[df['pattern_id']=='pattern_id'].index)\n",
    "    df = df.drop_duplicates()\n",
    "    \n",
    "    # not enough projects (43451 rows)\n",
    "    df = df.drop(df[df['projects_count'] <= 600].index)\n",
    "    \n",
    "    # drop rows with too many nulls (15795 rows)\n",
    "    df = df.drop(df[df.isnull().sum(axis=1) >3].index)\n",
    "    \n",
    "    # drop if no category\n",
    "    df = df.drop(df[df['categories'].isna()].index)\n",
    "\n",
    "    return df\n",
    "\n",
    "df = clean_out_bad_data(df)\n",
    "df = df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf1ae936-2573-48cc-8657-5f40ab42a3ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_search_results(pattern_id, page):\n",
    "    \"\"\" Get json response for page of projects for a particular pattern\"\n",
    "    try:\n",
    "        response = requests.get(f\"https://api.ravelry.com/patterns/{pattern_id}/projects.json?photoless=0&page_size=500&page={page}\",auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "        search_result = response.json()\n",
    "    except:\n",
    "        print(f'page number {page} failed')\n",
    "#         print(response)\n",
    "    return search_result\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8275b10c-c395-456c-9c4c-e26b5294caea",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_projects_for_pattern(pattern_id, saving_lower,saving_upper):\n",
    "    \"\"\"Gets all pages of projects for a particular pattern and saves to .csv file. \"\"\"\n",
    "    \n",
    "    #initialize page\n",
    "    page = 1\n",
    "    \n",
    "    # get number of pages to pull\n",
    "    try:\n",
    "        search_result = get_search_results(pattern_id,page)\n",
    "        last_page = search_result['paginator']['last_page']\n",
    "        print(f'starting new pattern - number of pages {last_page}')\n",
    "    except:\n",
    "        print(\"failed to get first page or page numbers\")\n",
    "    \n",
    "    while page < last_page+1:\n",
    "        try:\n",
    "            search_result = get_search_results(pattern_id, page)\n",
    "            \n",
    "            # for each page\n",
    "            user_id = []\n",
    "            pattern_ids =[]\n",
    "            date_completed = []\n",
    "            project_id =[]\n",
    "\n",
    "            for i in range(len(search_result['projects'])):\n",
    "                try:\n",
    "                    user_id.append(search_result['projects'][i]['user_id'])\n",
    "                    pattern_ids.append(search_result['projects'][i]['pattern_id'])\n",
    "                    date_completed.append(search_result['projects'][i]['completed'])\n",
    "                    project_id.append(search_result['projects'][i]['id'])\n",
    "                    \n",
    "                except Exception as e:\n",
    "                    print(e)\n",
    "                    \n",
    "            # assemble dictionary          \n",
    "            data = {'user_id':user_id,\n",
    "                    'pattern_ids':pattern_ids,\n",
    "                    'date_completed':date_completed,\n",
    "                    'project_id': project_id,  \n",
    "                   }\n",
    "            \n",
    "            #convert and save\n",
    "            df = pd.DataFrame(data)   \n",
    "\n",
    "            df.to_csv(f'data/users_projects_{saving_lower}-{saving_upper}.csv', mode =\"a\", index=False)\n",
    "            print(f\"yay pattern  {pattern_id}, page {page} saved!\")\n",
    "            \n",
    "            page += 1\n",
    "\n",
    "\n",
    "        except Exception as e:\n",
    "                print(e, 'Stopped on page {} -retrying!'.format(page))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "876f49c0-4efe-43ed-bd6e-d26e7f5d6460",
   "metadata": {},
   "source": [
    "I was worried about the file sizes so split them apart by 200 patterns (remember each patter has anywhere from 600 - 30000 projects each.)  The **saving upper**, and **saving lower** helped automate the separation and saving.  This ran for more than 16 hours at a time.  (There were some missing page errors that had to be rerun around after.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "af3bd637-14d5-4796-becb-c3ddf2bf5a73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving_lower = [100,300,500,700,900,1100,1300,1500,1700,1900,2100,2300,2500,2700]\n",
    "# saving_upper = [300,500,700,900,1100,1300,1500,1700,1900,2100,2300,2500,2700,2900]\n",
    "# saving_lower = [1900,2100,2300,2500,2700] # will need to go back and do 17-1900 and 2900+\n",
    "# saving_upper = [2100,2300,2500,2700,2900]\n",
    "# saving_lower = [1700,2900] \n",
    "# saving_upper = [1900,2962]\n",
    "saving_lower = [1887,2900] \n",
    "saving_upper = [1900,2962]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddb9faf3-6beb-45b6-acf8-cff645157753",
   "metadata": {},
   "source": [
    "The cell below was run to collect the projects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "67a057d5-1ece-4bd6-9c37-5b137e6e444f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern_id_df = df[['pattern_id']]\n",
    "\n",
    "# Get projects for a particular pattern\n",
    "for j in range (0,len(saving_lower)):\n",
    "    print(f'starting new file #{j}')\n",
    "    for i in range(saving_lower[j],saving_upper[j]):#range(len(pattern_id_df)):\n",
    "        pattern_id = pattern_id_df['pattern_id'][i]\n",
    "        print(f'file {j}, Step {i} Getting patterns for {pattern_id}')\n",
    "        get_all_projects_for_pattern(pattern_id, saving_lower[j],saving_upper[j])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_env",
   "language": "python",
   "name": "test_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}