BookSearcher

A pyLucene-based crawl and search module for searching books from goodreads.com

Module	Files	Description
Crawling	crawler.py, scraper.py, SetQueue.py	These files contain the implementation of a multi-threaded crawler. Specifically, scraper.py is the implementation for scraping webpages from Goodreads website. crawler.py implements the crawler threads, thread-safe url queues and related stuff. A thread-safe queue with a built-in set for url deduplication is implemented in SetQueue.py
Indexing	booktype.py, indexer.py, PageRank.py	Indexing using two libraries has been implemented. You can choose between Elasticsearch and Lucene for indexing. For document boosting, PageRank scoring is also implemented.
Searching	searcher.py	It is a web.py-based minimal web server for searching indexes and displaying results in a web page.

Libraries used

Web.py: web.py is a web framework for Python that is as simple as it is powerful.
Beautifulsoup4: For web scraping

Scipy/Numpy was also used for calculating PageRank scores.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.vscode		.vscode
NewJsons2		NewJsons2
NewJsons3		NewJsons3
NewJsons4		NewJsons4
crawler_src		crawler_src
static		static
tempUrl2		tempUrl2
tempUrl4		tempUrl4
tempUrl4_1		tempUrl4_1
templates		templates
.gitignore		.gitignore
JSON Playground and PageRank.ipynb		JSON Playground and PageRank.ipynb
JSONs.rar		JSONs.rar
LICENSE		LICENSE
NewJsons2.zip		NewJsons2.zip
NewJsons3.zip		NewJsons3.zip
NewJsons4-old.zip		NewJsons4-old.zip
NewJsons4.zip		NewJsons4.zip
NewJsons6.zip		NewJsons6.zip
PageRank.py		PageRank.py
README.md		README.md
Scraping GoodReads.ipynb		Scraping GoodReads.ipynb
SetQueue.py		SetQueue.py
booktype.py		booktype.py
crawl_en.zip		crawl_en.zip
crawler.py		crawler.py
crawler2.py		crawler2.py
crawler3.py		crawler3.py
crawler4.py		crawler4.py
crawler_src.zip		crawler_src.zip
elasticsearch-test.py		elasticsearch-test.py
generateSeeds2.py		generateSeeds2.py
generateSeeds3.py		generateSeeds3.py
generateSeeds4.py		generateSeeds4.py
homepage.png		homepage.png
indexer.py		indexer.py
log2.txt		log2.txt
log4.txt		log4.txt
removeIlegal.py		removeIlegal.py
scraper-demo.py		scraper-demo.py
scraper.py		scraper.py
scraper2.py		scraper2.py
scraper3.py		scraper3.py
scraper4.py		scraper4.py
searcher.py		searcher.py
setqueue-demo.py		setqueue-demo.py
temp.txt		temp.txt
test.txt		test.txt
url.txt		url.txt
url2.txt		url2.txt
url2.zip		url2.zip
url3.txt		url3.txt
url4.txt		url4.txt
url_problem.txt		url_problem.txt