README.md

Police Data Trust Scrapers

Table of Contents

• Setup
• Run scrapers
  • 50-a
  • Citizens Police Data Project

Setup

1. Due to the way Scrapy structures projects, we need to add the path to this repo to python path.

   export PYTHONPATH=/path/to/police-data-trust-scrapers/

2. Create a virtual environment with Python 3.13.0

3. Install requirements

   pip install -r requirements_dev.txt

Run scrapers

Note: You can add arguments to scrapers by adding -a {argument_name}={argument_value} to the end of the bash command.

50-a

1. Go to the fifty_a folder

   cd scrapers/fifty_a

2. Run the office spider

   scrapy crawl officer -O officers.jsonl

3. Run the command spider

   scrapy crawl command -O commands.jsonl

Citizens Police Data Project

This is not a wb scraper. It rather pulls data from their API endpoint.

From the repo root, run the following. It will pull the data and create json files in the data/citizens_police_data_project folder.

python scrapers/citizens_police_data_project.py