The goal is to scrape contents from a website using Scrapy and to save the scraped data in a csv file. This project contains python script to scrape bike's specifications from carandbike website.
- You can find the python script in scrape_bikes/spiders/bike.py
Installation : Anaconda
Scrapy documentation : Scrapy
- Installing scrapy using command prompt/terminal :
pip install scrapy - Installing scrapy using anaconda's cmd.exe prompt :
conda install -c conda-forge scrapy pylint autopep8 -y
Installation : VSCode.
Documentation : link
- Download the whole project and save it in a folder.
- Open cmd.exe Prompt in anaconda.
- To run the spider, use this command :
scrapy crawl bike - To save the data in csv form :
scrapy crawl bike -o bike_dataset.csv - To save the data in json format :
scrapy crawl bike -o bike_dataset.json