Code

Optimize scraping with threads#192

Code

Merged

Crinibus merged 14 commits intomasterCrinibus/scraper:masterfrom update-scrape-with-threads-group-products-df-by-domainsCrinibus/scraper:update-scrape-with-threads-group-products-df-by-domainsCopy head branch name to clipboardJan 30, 2023

+123-68

Commits

Commits on Nov 8, 2022

• Move functions from misc.py to new file print_products.py

  Crinibus

  committed
  0f315da

• Add functions to add column and group pandas dataframe and function to group products df by domains

  Crinibus

  committed
  fe77070

• Add function to get grouped products by domain

  Show description for 837b6c8
  Crinibus

  committed
  837b6c8

• Add function to start threads sequentially

  Crinibus

  committed
  f1f4e7c

• Update scraper\__init__.py

  Show description for 2764397
  Crinibus

  committed
  2764397

• Update function scrape_with_threads

  Show description for a27fed7
  Crinibus

  committed
  a27fed7

Commits on Nov 9, 2022

• Update comment

  Crinibus

  committed
  feaed1a

• Delete empty line

  Crinibus

  committed
  02973ec

Commits on Nov 13, 2022

• Add missing return type hints

  Crinibus

  committed
  13d74a6

Commits on Nov 22, 2022

• Merge branch 'master' into update-scrape-with-threads-group-products-df-by-domains

  Crinibus

  authored
  258d63f

Commits on Dec 27, 2022

• Format with black

  Crinibus

  committed
  7905694

• Update function get_products_df_grouped_by_domains - move getting product dataFrame to outside of function

  Crinibus

  committed
  20ec448

Commits on Dec 28, 2022

• Update function start_threads_sequentially - add thread.join()

  Crinibus

  committed
  0572942

Commits on Jan 19, 2023

• Delete import of List, Tuple and Dict from typing

  Crinibus

  committed
  58e12dd