Skip to content

panyuan520/spider

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

if windows: run the run.bat
else: spider.py -u "http://www.sina.com.cn" -d 2 -f "spider.log" -l 2 --test="" -t 10 --dbfile "spider.db" --key="HTML5"

keys:
-u crawl url -d deep -f logpath
-l loglevel (1<= deep <=5)
--test doctest (parse_url, insert_db, run) --dbfile database path
--crawl key

if the key is null will use default.

Releases

No releases published

Packages

No packages published