Releases: Crinibus/scraper
Releases · Crinibus/scraper
Version 2.0.1
Just a quick patch:
- Add missing visualize color for website HiFi Klubben
Version 2.0.0
Rename some of the arguments:
--visualize-all
to--all
--visualize-id
to--id
--visualize-name
to--name
Delete argument --visualize-category
Update method BaseWebsiteHandler._request_product_data
:
- Does not return return request data, now saves it in the instance variable
self.request_data
Simplify class Scraper
to be more readable.
Add tests for website handlers and add product.
Add support for website HiFiKlubben
Move the setting REQUEST_DELAY
to file settings.ini
Released version 1.5.1
- Add missing website color for Newegg
- Add an image of an example graph in README
Released version 1.5
- Add a new argument --up-to-date or -utd for short.
- When visualizing all product or all products in a category there is now an option to only show graph for the products that are up to date with the argument --up-to-date
Released version 1.4
- Add support for scraping prices from Newegg
- Fix problems with scraping from Amazon
Released version 1.3
Add argument --search
- Search for product and category names
Released version 1.2
scraper/constants.py
- update REQUEST_HEADER user agent to new version of Chrome
scaper/filemanager.py
- add class variables with paths to files like records.json and products.csv
scaper/add_product.py
- fix typo and add the website name to the log string when adding product from a website that is not supported
- move function "add_products" from main.py to scraper/add_product.py
- don't add product category and link to products.csv if already in the csv-file
- make some functions more readable
scraper/clean_data.py
- rename function "clean_data" to "clean_records_data"
scraper/visualize.py
- move function "visualize" from main.py to scraper/visualize.py
- rename function "visualize" to "visualize_data"
Add scraper/reset_data.py
- move functions "reset" and "hard_reset" from main.py to scraper/reset_data.py
scraper/__init__.py
- import the functions that was moved from main.py to scraper module
Released version 1.1.8.1
- Add some more error handling when visualizing when no name or id is found
Released version 1.1.8
- Add a sleep between each scrape of a product (default is 0 seconds, can be changed in scraper/constants.py)
- Update graph titles to show if the graph is up to date
Released version 1.1.7
Class Scraper:
- Add timeout to requests.get in method "request_url"
- Simplify try except in method "get_info"
General:
- Format with Black