Code

Add database#226

Code

Merged

Crinibus merged 97 commits intomasterCrinibus/scraper:masterfrom add-databaseCrinibus/scraper:add-databaseCopy head branch name to clipboardNov 17, 2023

+890-608

Commits

Commits on Sep 22, 2023

• Add subpackage 'database' - create and connect to database and database models

  Crinibus

  committed
  e1eb1c1

  Copy full SHA for e1eb1c1

• Add method to class 'Format' in 'format_to_new.py' - migrate from json to database

  Crinibus

  committed
  8042c5b

  Copy full SHA for 8042c5b

• Replace function 'clean_records_data' with 'clean_datapoints'

  Crinibus

  committed
  83dfd00

  Copy full SHA for 83dfd00

• Ignore lint error E501 for line with api link to Elgiganten

  Crinibus

  committed
  7cd5d2c

  Copy full SHA for 7cd5d2c

• Create file 'database/functions' - add function 'get_product_by_product_id'

  Crinibus

  committed
  7a95a6f

  Copy full SHA for 7a95a6f

Commits on Sep 28, 2023

• Don't import Product and DataPoint as ...DB in format_to_new.py

  Crinibus

  committed
  22a1fd3

  Copy full SHA for 22a1fd3

• Rename Product and Datapoint to ProductInfo and DataPointInfo

  Show description for f0dd3fe
  Crinibus

  committed
  f0dd3fe

  Copy full SHA for f0dd3fe

• Add category to database model Product

  Crinibus

  committed
  d253f92

  Copy full SHA for d253f92

• Rename columns from 'productId' to 'product_code' for database Product and DataPoint

  Crinibus

  committed
  d9d991f

  Copy full SHA for d9d991f

Commits on Oct 1, 2023

• Add sqlmodel to requirements.txt

  Crinibus

  committed
  6960524

  Copy full SHA for 6960524

• Delete methods in class Filemanager and create class FilemanagerLegacy in format_to_new.py

  Crinibus

  committed
  3fb5343

  Copy full SHA for 3fb5343

• Add short_url column to database Product

  Crinibus

  committed
  6437784

  Copy full SHA for 6437784

• Update method 'from_json_to_db' in class Format (format_to_new.py) to also get short_url

  Crinibus

  committed
  ec23bc6

  Copy full SHA for ec23bc6

• Add database functions - 'delete_all', 'get_product_by_product_code', 'get_products_by_product_codes', 'add_product' and 'add_datapoint'

  Show description for 543e3a4
  Crinibus

  committed
  543e3a4

  Copy full SHA for 543e3a4

• Update add_product.py to use database

  Crinibus

  committed
  b428789

  Copy full SHA for b428789

• Add database functions - 'get_all_products', 'get_all_datapoints', 'get_products_by_categories' and 'get_products_by_names'

  Crinibus

  committed
  dcf61db

  Copy full SHA for dcf61db

• Update delete_data.py to use database

  Crinibus

  committed
  be5ddb8

  Copy full SHA for be5ddb8

• Add functions 'get_master_products' and 'get_products_from_master_products' to format_to_new.py

  Show description for aa2f09e
  Crinibus

  committed
  aa2f09e

  Copy full SHA for aa2f09e

• Move two database functions to the top of functions.py

  Crinibus

  committed
  7840bf0

  Copy full SHA for 7840bf0

• Reorder import from '.functions' in database/__init__.py

  Crinibus

  committed
  bf7179e

  Copy full SHA for bf7179e

• Delete database functions 'add_product' and 'add_datapoint' and replace with generic 'add' and 'add_all'

  Crinibus

  committed
  675716d

  Copy full SHA for 675716d

• Fix database functions 'get_products_by_product_codes', 'get_products_by_categories' and 'get_products_by_names'

  Show description for 788cf97
  Crinibus

  committed
  788cf97

  Copy full SHA for 788cf97

• Add logging to start of function 'delete'

  Crinibus

  committed
  4e94f02

  Copy full SHA for 4e94f02

• Add database functions 'get_datapoints_by_categories', 'get_datapoints_by_names' and 'get_datapoints_by_product_codes'

  Crinibus

  committed
  9843b89

  Copy full SHA for 9843b89

• Update reset_data.py to use database

  Crinibus

  committed
  51d10a6

  Copy full SHA for 51d10a6

• Update README to reflect changes of what reset does - deletes only datapoints

  Crinibus

  committed
  598882c

  Copy full SHA for 598882c

Commits on Oct 2, 2023

• Add database path to Filemanager and use it to create and connect to the database

  Crinibus

  committed
  c5138c1

  Copy full SHA for c5138c1

• Update ProductInfo - change field 'is_up_to_date' to a calculated property

  Crinibus

  committed
  a2235b0

  Copy full SHA for a2235b0

• Add database function 'get_all_products_with_datapoints'

  Crinibus

  committed
  a635a27

  Copy full SHA for a635a27

• Add database function 'get_product_infos_from_products' and update 'get_all_products_with_datapoints'

  Crinibus

  committed
  364a2cb

  Copy full SHA for 364a2cb

• Update visualize.py to use database

  Crinibus

  committed
  a808ee1

  Copy full SHA for a808ee1

• Add database function 'get_products_by_names_fuzzy'

  Crinibus

  committed
  47a6bbe

  Copy full SHA for 47a6bbe

• Add database function 'get_all_unique_categories'

  Crinibus

  committed
  1d26869

  Copy full SHA for 1d26869

• Move database function 'get_all_unique_categories' further up in the file

  Crinibus

  committed
  544882e

  Copy full SHA for 544882e

• Update search_data.py to use database

  Crinibus

  committed
  c2f1808

  Copy full SHA for c2f1808

• Delete import of List from typing - instead use built-in list as type

  Crinibus

  committed
  3706910

  Copy full SHA for 3706910

Commits on Oct 3, 2023

• Update function 'clean_datapoints' to use database functions instead of a db session

  Crinibus

  committed
  aaa0c5d

  Copy full SHA for aaa0c5d

• Add database functions 'get_all_unique_domains', 'get_products_by_domains' and 'get_all_products_grouped_by_domains'

  Crinibus

  committed
  cf739de

  Copy full SHA for cf739de

• Add static method 'scraper_to_db_product' to class Format

  Show description for ec216ac
  Crinibus

  committed
  ec216ac

  Copy full SHA for ec216ac

• Add static method 'db_product_to_scraper' to class Format

  Crinibus

  committed
  2fd3fa6

  Copy full SHA for 2fd3fa6

• Use parameter 'isActive' in method 'Format.scraper_to_db_product'

  Crinibus

  committed
  f4a976f

  Copy full SHA for f4a976f

• Move static method 'get_user_product_name' from class Format to class Config

  Crinibus

  committed
  77cb2c8

  Copy full SHA for 77cb2c8

• Delete import of Config in Format.py

  Crinibus

  committed
  13b79a1

  Copy full SHA for 13b79a1

• Add static method 'db_products_to_scrapers' to class Format

  Crinibus

  committed
  42010e4

  Copy full SHA for 42010e4

• Add database function 'get_all_active_products'

  Crinibus

  committed
  ba9e007

  Copy full SHA for ba9e007

Commits on Oct 10, 2023

• Update database functions 'get_products_by_domains' and 'get_all_products_grouped_by_domains' - add parameter to only select active products

  Crinibus

  committed
  9676eb1

  Copy full SHA for 9676eb1

• Update scraper/__init__.py - add import of class Format and module database

  Crinibus

  committed
  4564cb2

  Copy full SHA for 4564cb2

• Update database function 'get_all_products' - add parameter to only select active products

  Crinibus

  committed
  ec0eb09

  Copy full SHA for ec0eb09

• Update database function 'get_all_products_with_datapoints' - add parameter to only select active products

  Crinibus

  committed
  c8b6ec4

  Copy full SHA for c8b6ec4

Commits on Oct 11, 2023

• Delete database function 'get_all_active_products'

  Show description for f07d007
  Crinibus

  committed
  f07d007

  Copy full SHA for f07d007

• Add database functions 'group_products_by_domains' and 'group_products_by_names'

  Show description for c4b9272
  Crinibus

  committed
  c4b9272

  Copy full SHA for c4b9272

• Update print_products.py to use database

  Crinibus

  committed
  61c8642

  Copy full SHA for 61c8642

• Fix 'print_latest_datapoints_for_products' - group by name

  Crinibus

  committed
  62fde49

  Copy full SHA for 62fde49

• Add parameter 'categories' to function 'print_latest_datapoints'

  Crinibus

  committed
  ff128b4

  Copy full SHA for ff128b4

• Fix function 'get_master_products'

  Show description for 81ce2c8
  Crinibus

  committed
  81ce2c8

  Copy full SHA for 81ce2c8

• Rename functions 'add_new_product' and 'add_new_datapoint' to '..._to_db'

  Crinibus

  committed
  fc21444

  Copy full SHA for fc21444

• Add column 'created_at' to Product and DataPoint database tables

  Crinibus

  committed
  a92310d

  Copy full SHA for a92310d

• Add condition to function 'add_new_datapoint_with_scraper'

  Show description for f9fd311
  Crinibus

  committed
  f9fd311

  Copy full SHA for f9fd311

• Update main.py to use database

  Crinibus

  committed
  be855a4

  Copy full SHA for be855a4

• Update scrape.py - delete logic to save to json

  Show description for d22d4cb
  Crinibus

  committed
  d22d4cb

  Copy full SHA for d22d4cb

Commits on Oct 14, 2023

• Delete misc.py

  Show description for 786f6a9
  Crinibus

  committed
  786f6a9

  Copy full SHA for 786f6a9

• Remove pandas from requirements.txt

  Crinibus

  committed
  eff2a46

  Copy full SHA for eff2a46

• Add section to README about how data is stored after V3.0.0

  Crinibus

  committed
  2166538

  Copy full SHA for 2166538

• Rename database column "isActive" to "is_active"

  Crinibus

  committed
  abb125b

  Copy full SHA for abb125b

• Rename function 'active_existing_product' to 'set_existing_product_is_active' and add parameter 'is_active'

  Crinibus

  committed
  3267877

  Copy full SHA for 3267877

• Update function 'print_all_products' - don't print product_code instead of id

  Crinibus

  committed
  0663fb7

  Copy full SHA for 0663fb7

• Update function 'print_all_products' - add marker to indicate which products are activated

  Crinibus

  committed
  da60462

  Copy full SHA for da60462

• Update section "View all products" in README with new product activated check markl

  Crinibus

  committed
  a28102a

  Copy full SHA for a28102a

• Add function 'update_products_is_active_with_product_codes'

  Crinibus

  committed
  26ef5f4

  Copy full SHA for 26ef5f4

• Add arguments --activate and --deactivate

  Crinibus

  committed
  f997d39

  Copy full SHA for f997d39

• Fix missing rename of Product 'isActive' to 'is_active'

  Crinibus

  committed
  254c9dd

  Copy full SHA for 254c9dd

• Fix function 'search_product_name'

  Show description for 9f83b78
  Crinibus

  committed
  9f83b78

  Copy full SHA for 9f83b78

• Rename function 'search_categories' to 'search_category'

  Crinibus

  committed
  9a8f5ee

  Copy full SHA for 9a8f5ee

• Update search_data.py - search for all search terms at the start and show results for product names in one line and categories in another

  Crinibus

  committed
  c43704c

  Copy full SHA for c43704c

• Delete function 'search_category' and update function 'search_categories'

  Crinibus

  committed
  8994e1d

  Copy full SHA for 8994e1d

• Update database function 'get_product_infos_from_products' - handle if no datapoints was found

  Crinibus

  committed
  1052416

  Copy full SHA for 1052416

Commits on Oct 15, 2023

• Update function 'get_products_by_names_fuzzy' - rename variable 'name' to 'fuzzy_name'

  Crinibus

  committed
  4ef60aa

  Copy full SHA for 4ef60aa

• Update parameter type hint in function 'add_all'

  Crinibus

  committed
  b3dac76

  Copy full SHA for b3dac76

• Add logging to delete functions in delete_data.py

  Crinibus

  committed
  216ef11

  Copy full SHA for 216ef11

Commits on Oct 27, 2023

• Rename and update method Config.get_user_product_names

  Show description for 149e02f
  Crinibus

  committed
  149e02f

  Copy full SHA for 149e02f

• Fix function 'scrape_with_threads'

  Show description for d6c1f35
  Crinibus

  committed
  d6c1f35

  Copy full SHA for d6c1f35

Commits on Nov 9, 2023

• Delete function 'get_product_data'

  Show description for f5ce7c7
  Crinibus

  committed
  f5ce7c7

  Copy full SHA for f5ce7c7

• Use build-in 'dict' type hint instead of 'from typing import Dict''

  Crinibus

  committed
  f0ff51c

  Copy full SHA for f0ff51c

Commits on Nov 14, 2023

• Fix tests/test_website_handlers.py - update import of class 'Info' and add function 'read_json'

  Show description for fbdfbe5
  Crinibus

  committed
  fbdfbe5

  Copy full SHA for fbdfbe5

• Delete comment

  Crinibus

  committed
  e9d5662

  Copy full SHA for e9d5662

• Set attribute 'product_info' in class 'Scraper' to None in __init__ method

  Crinibus

  committed
  1ec172a

  Copy full SHA for 1ec172a

• Update method 'scrape_info' in class 'Scraper' to return product info

  Show description for d38b767
  Crinibus

  committed
  d38b767

  Copy full SHA for d38b767

• Update tests\test_add_product.py to reflect changes of database update

  Crinibus

  committed
  95313a7

  Copy full SHA for 95313a7

• Add configurations for ruff in pyproject.toml

  Crinibus

  committed
  df950bd

  Copy full SHA for df950bd

Commits on Nov 17, 2023

• Update test_objects.json

  Show description for 30c0045
  Crinibus

  committed
  30c0045

  Copy full SHA for 30c0045

• Update EbayHandler

  Show description for ed75b3c
  Crinibus

  committed
  ed75b3c

  Copy full SHA for ed75b3c

• Add if condition to function 'print_latest_datapoint'

  Show description for add6ca4
  Crinibus

  committed
  add6ca4

  Copy full SHA for add6ca4

• Update README.md

  Show description for 02255a3
  Crinibus

  authored
  02255a3

  Copy full SHA for 02255a3

• Merge branch 'master' into add-database

  Crinibus

  authored
  7ef14ec

  Copy full SHA for 7ef14ec

• Update main.py - create database and tables at the start of the script

  Crinibus

  committed
  fccaf5b

  Copy full SHA for fccaf5b

• Update README - reword the a sentence in section "UPDATE TO HOW DATA IS STORED IN V3.0.0"

  Crinibus

  committed
  ecae5a3

  Copy full SHA for ecae5a3

• Update README - Add a note to section "UPDATE TO HOW DATA IS STORED IN V3.0.0"

  Show description for 2c8a945
  Crinibus

  committed
  2c8a945

  Copy full SHA for 2c8a945