0.1-api-calls-patterns.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03dd9fed-fd7f-414b-b883-a17bb71b7aa4",
   "metadata": {},
   "source": [
    "## Data for Content Based System (Pattern Metadata) \n",
    "\n",
    "Using ravelry.com's api.  Limit of 100k per call.  As of Friday November 19, 665,465 knitting patterns. Api does not currently allow for sort-by searches directly.  Will need to segment and pull out in sequential chunks - then filter down after the fact. \n",
    "\n",
    "**get_all_patterns()** function found at the bottom of this notebook consolidates all of the other functions - in order to call and save each segmented portion of the data. \n",
    "\n",
    "* get pattern list (ids) fitting within query criteria\n",
    "* then iterate through through patterns ids and pull metadata for each \n",
    "* if <100k, split into segments\n",
    "* save as .csv's to later combine into one monster dataframe "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "c89fe5c2-e542-4531-aeab-9f1414dfbb28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import json\n",
    "import requests\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from pprint import pprint\n",
    "\n",
    "from config import basic_auth_username, basic_auth_password\n",
    "from config import basic_auth_username_read_only, basic_auth_password_read_only"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c54b85-f475-4302-bb92-a81f0979c3a5",
   "metadata": {},
   "source": [
    "It's a two step process to get the knitting pattern metadata.  The pattern id's can be pulled from **get_search_results** using a query, then using the pattern_ids from the json response, **get_pattern_details** is used to access the details such as project count (how many of each pattern were knit), average rating, needle sizes. \n",
    "\n",
    "The get_search_results function below was used 3 times: \n",
    "* First for all patterns that were 5 star rated and clothing\n",
    "* Then for all patterns that were 5 stars not clothing (as there were over 100k of 5 star patterhs)\n",
    "* 4 stars (just over 75k patterns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "600e1fde-b526-44a3-b49e-c284d1587a9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_search_results(page):\n",
    "    \"\"\" gets json response for patterns with search parameters - (will be used to \n",
    "    can extract pattern ids from this list, in order to get details in subsequent functions) \"\"\"\n",
    "    try:\n",
    "#         #5stars and NOT clothing (worked!)\n",
    "#         response =requests.get(f'https://api.ravelry.com/patterns/search.json?craft=knitting&ratings=5&pc=accessories%7Cmedical%7Chome%7Ctoysandhobbies%7Cpet&photo=yes&sort=best&page_size=500&page={page}',auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "        \n",
    "#         #5starrs and IS clothing \n",
    "#         response =requests.get(f'https://api.ravelry.com/patterns/search.json?craft=knitting&ratings=5&pc=clothing&photo=yes&sort=best&page_size=100&page={page}',auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "\n",
    "        #4stars all patterhns\n",
    "        response =requests.get(f'https://api.ravelry.com/patterns/search.json?craft=knitting&ratings=4&photo=yes&sort=best&page_size=100&page={page}',auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "\n",
    "        search_results = response.json()\n",
    "    except:\n",
    "        print(f'page number {page} failed')\n",
    "        print(response)\n",
    "    return search_results\n",
    "\n",
    "\n",
    "def get_pattern_details(search_results):\n",
    "    \"\"\" parses pattern id's out of the json request from get_search_results and uses to call for pattern details (where the \n",
    "    pattern metadata can be found \"\"\"\n",
    "    ids = []\n",
    "    for i in range(len(search_results['patterns'])):\n",
    "        ids.append(search_results['patterns'][i]['id'])\n",
    "\n",
    "    id_list = [str(x) for x in ids]\n",
    "    spaced_ids = '+'.join(id_list)\n",
    "    spaced_ids\n",
    "\n",
    "    response =requests.get(f'https://api.ravelry.com/patterns.json?ids={spaced_ids}', auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "    return response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145a9e55-5dea-4707-b63f-a180da8faf07",
   "metadata": {},
   "source": [
    "### Parse out Pattern Details\n",
    "#### For a singular result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "id": "38b4490a-19ec-4b6c-beae-69bb3fb8c4b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # For one result:  \n",
    "# response =requests.get('https://api.ravelry.com/patterns/573.json',auth=HTTPBasicAuth(basic_auth_username, basic_auth_password))\n",
    "# patterns = response.json()\n",
    "# # pprint(patterns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7eef763-eb5c-41a7-a491-73428e55eba7",
   "metadata": {},
   "source": [
    "#### For multiple results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "409e28ba-659b-4bab-80b1-9223932e10c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_json_response_into_df(pattern):\n",
    "    \"\"\" takes response from get_pattern_details and parses out metadata details, returns as a dataframe.  Note this \n",
    "    only works for multiple pattern calls at once \"\"\"\n",
    "    \n",
    "#     instantiate lists\n",
    "    id_ = []\n",
    "    name = []\n",
    "    favorites_count = []\n",
    "    projects_count = []\n",
    "    difficulty_average = []\n",
    "    difficulty_count = []\n",
    "    rating_average = []\n",
    "    rating_count = []\n",
    "    pattern_type_id = []\n",
    "    pattern_type_names = []\n",
    "    pattern_type_clothing = []\n",
    "    yarn_weight = []\n",
    "    photos_url =[]\n",
    "    craft_id = []\n",
    "    yarn_weight = []\n",
    "    pattern_needle_sizes = []\n",
    "    yardage = []\n",
    "    yarn_weight_description = []\n",
    "    yardage_max = []\n",
    "    yardage = []\n",
    "    gauge = []\n",
    "    gauge_divisor = []\n",
    "    free = []\n",
    "    downloadable = []\n",
    "    queued_projects_count =[]\n",
    "    attributes = []\n",
    "    generally_available = []\n",
    "    yarn_weight_name = []\n",
    "    yardage_description =[]\n",
    "    pattern_attributes = []\n",
    "    categories = []\n",
    "    name_permalink =[]\n",
    "\n",
    "    # parse json response \n",
    "    for i in pattern['patterns']:\n",
    "        id_.append(pattern['patterns'][i]['id'])\n",
    "        name.append(pattern['patterns'][i]['name'])\n",
    "        name_permalink.append(pattern['patterns'][i]['permalink'])\n",
    "        favorites_count.append(pattern['patterns'][i]['favorites_count'])\n",
    "        projects_count.append(pattern['patterns'][i]['projects_count'])\n",
    "        difficulty_average.append(pattern['patterns'][i]['difficulty_average'])\n",
    "        difficulty_count.append(pattern['patterns'][i]['difficulty_count'])\n",
    "        queued_projects_count.append(pattern['patterns'][i]['queued_projects_count'])\n",
    "        rating_average.append(pattern['patterns'][i]['rating_average'])\n",
    "        rating_count.append(pattern['patterns'][i]['rating_count'])\n",
    "        downloadable.append(pattern['patterns'][i]['downloadable'])\n",
    "        free.append(pattern['patterns'][i]['free'])\n",
    "        gauge_divisor.append(pattern['patterns'][i]['gauge_divisor'])\n",
    "        gauge.append(pattern['patterns'][i]['gauge'])\n",
    "        yardage.append(pattern['patterns'][i]['yardage'])\n",
    "        yardage_max.append(pattern['patterns'][i]['yardage_max'])\n",
    "        yarn_weight_description.append(pattern['patterns'][i]['yarn_weight_description'])\n",
    "        generally_available.append(pattern['patterns'][i]['generally_available'])\n",
    "        yardage_description.append(pattern['patterns'][i]['yardage_description'])\n",
    "        try:\n",
    "            photos_url.append(pattern['patterns'][i]['photos'][0]['square_url'])\n",
    "        except:\n",
    "            photos_url.append(None)  \n",
    "        try:   \n",
    "            pattern_type_clothing.append(pattern['patterns'][i]['pattern_type']['clothing'])\n",
    "            pattern_type_names.append(pattern['patterns'][i]['pattern_type']['permalink'])\n",
    "        except:\n",
    "            pattern_type_clothing.append(None)\n",
    "            pattern_type_names.append(None)\n",
    "        try:\n",
    "            pattern_needle_sizes.append(pattern['patterns'][i]['pattern_needle_sizes'])\n",
    "        except:\n",
    "            pattern_needle_sizes.append(None)\n",
    "\n",
    "        attributes = []\n",
    "        try:\n",
    "            for j in range(len(pattern['patterns'][i]['pattern_attributes'])):\n",
    "                attributes.append(pattern['patterns'][i]['pattern_attributes'][j]['permalink'])\n",
    "            pattern_attributes.append(attributes)\n",
    "        except:\n",
    "            pattern_attributes.append('None')\n",
    "\n",
    "        try:\n",
    "            category_dict = pattern['patterns'][i]['pattern_categories'][0]\n",
    "            category_list = []\n",
    "            category_list = [category_dict['permalink']]\n",
    "            new_dict = category_dict['parent']\n",
    "            while 'parent' in new_dict.keys():\n",
    "                category_list.append(new_dict['permalink'])\n",
    "                new_dict = new_dict['parent']\n",
    "            categories.append(category_list)\n",
    "        except:\n",
    "            categories.append(None)\n",
    "            print(\"uhoh - check out categories!\")\n",
    "            \n",
    "    # assemble dictionary          \n",
    "    data = {'pattern_id':id_,\n",
    "            'name':name,\n",
    "            'name_permalink':name_permalink,\n",
    "            'favorites_count': favorites_count,\n",
    "            'projects_count': projects_count, \n",
    "            'difficulty_average' : difficulty_average, \n",
    "            'difficulty_count': difficulty_count, \n",
    "            'rating_average': rating_average,\n",
    "            'queued_projects_count': queued_projects_count,\n",
    "            'rating_count':rating_count,\n",
    "            'pattern_type_names' :pattern_type_names,\n",
    "            'pattern_type_clothing' :pattern_type_clothing,\n",
    "            'photos_url' :photos_url,\n",
    "            'pattern_needle_sizes' :pattern_needle_sizes,\n",
    "            'pattern_attributes':pattern_attributes,\n",
    "            'yardage_max' :yardage_max,\n",
    "            'yardage' :yardage,\n",
    "            'generally_available':generally_available,\n",
    "            'gauge' :gauge,\n",
    "            'gauge_divisor' :gauge_divisor,\n",
    "            'free' :free,\n",
    "            'downloadable': downloadable,\n",
    "            'categories':categories,\n",
    "            'yarn_weight_description' :yarn_weight_description,   \n",
    "           }\n",
    "    \n",
    "    return pd.DataFrame(data)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "845fd4e0-a7e9-4b29-ac31-5f5146114b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_and_save_csv(df):\n",
    "    \"\"\" doesn't currently filter, just saves (appends) csv at the moment \"\"\"\n",
    "    df.to_csv('data/patterns_4star.csv', mode =\"a\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "282e79e6-fdea-4163-86bf-8574d068ac47",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_patterns():\n",
    "    \"\"\" Consolidates other fuctions - iterates through pages of paginator to get pattern ids, then call for get \n",
    "    pattern details, parse response, and apend to .csv \"\"\"\n",
    "    page = 1\n",
    "    while page < 784:\n",
    "        try:\n",
    "            search_results = get_search_results(page)\n",
    "            json_data = get_pattern_details(search_results)\n",
    "            new_parsed_patterns = parse_json_response_into_df(json_data)\n",
    "            print('yay parsed pattterns page {}!'.format(page))\n",
    "\n",
    "            filter_and_save_csv(new_parsed_patterns)\n",
    "            print('yay saved!')\n",
    "\n",
    "            page += 1\n",
    "\n",
    "        except Exception as e:\n",
    "                print(e, 'Stopped on page {} -retrying!'.format(page))\n",
    "\n",
    "# get_all_patterns()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_env",
   "language": "python",
   "name": "test_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}