Skip to content

Releases: Crinibus/scraper

Version 2.0.1

10 Sep 20:46
4f05b67
Compare
Choose a tag to compare

Just a quick patch:

  • Add missing visualize color for website HiFi Klubben

Version 2.0.0

06 Sep 05:00
798af5b
Compare
Choose a tag to compare

Rename some of the arguments:

  • --visualize-all to --all
  • --visualize-id to --id
  • --visualize-name to --name

Delete argument --visualize-category


Update method BaseWebsiteHandler._request_product_data:

  • Does not return return request data, now saves it in the instance variable self.request_data

Simplify class Scraper to be more readable.

Add tests for website handlers and add product.

Add support for website HiFiKlubben

Move the setting REQUEST_DELAY to file settings.ini

Released version 1.5.1

30 May 16:45
b925420
Compare
Choose a tag to compare
  • Add missing website color for Newegg
  • Add an image of an example graph in README

Released version 1.5

11 May 15:44
e20cd4e
Compare
Choose a tag to compare
  • Add a new argument --up-to-date or -utd for short.
  • When visualizing all product or all products in a category there is now an option to only show graph for the products that are up to date with the argument --up-to-date

Released version 1.4

13 Apr 03:53
43e96ee
Compare
Choose a tag to compare
  • Add support for scraping prices from Newegg
  • Fix problems with scraping from Amazon

Released version 1.3

12 Mar 09:07
d0a7bca
Compare
Choose a tag to compare

Add argument --search

  • Search for product and category names

Released version 1.2

16 Dec 23:21
a448cd6
Compare
Choose a tag to compare

scraper/constants.py

  • update REQUEST_HEADER user agent to new version of Chrome

scaper/filemanager.py

  • add class variables with paths to files like records.json and products.csv

scaper/add_product.py

  • fix typo and add the website name to the log string when adding product from a website that is not supported
  • move function "add_products" from main.py to scraper/add_product.py
  • don't add product category and link to products.csv if already in the csv-file
  • make some functions more readable

scraper/clean_data.py

  • rename function "clean_data" to "clean_records_data"

scraper/visualize.py

  • move function "visualize" from main.py to scraper/visualize.py
  • rename function "visualize" to "visualize_data"

Add scraper/reset_data.py

  • move functions "reset" and "hard_reset" from main.py to scraper/reset_data.py

scraper/__init__.py

  • import the functions that was moved from main.py to scraper module

Released version 1.1.8.1

12 Dec 23:11
e05edf0
Compare
Choose a tag to compare
  • Add some more error handling when visualizing when no name or id is found

Released version 1.1.8

06 Jun 22:03
8e8fbdc
Compare
Choose a tag to compare
  • Add a sleep between each scrape of a product (default is 0 seconds, can be changed in scraper/constants.py)
  • Update graph titles to show if the graph is up to date

Released version 1.1.7

19 May 17:26
76b65c1
Compare
Choose a tag to compare

Class Scraper:

  • Add timeout to requests.get in method "request_url"
  • Simplify try except in method "get_info"

General:

  • Format with Black