forked from rit-hc-website/data-ingest
-
Notifications
You must be signed in to change notification settings - Fork 1
Development environment setup
Adrian Edwards edited this page Feb 24, 2024
·
15 revisions
To get started helping out, you need to set up the event-feed-ingest
cli. From there, you'll be able to start writing scraper pipeline stages. To date, we've built pipeline stages in bash
and python
, but aren't dogmatic about it.
-
Clone the repository:
git clone https://github.com/CampusPulse/data-ingest
-
Navigate to your git checkout
cd event-feed-ingest
-
Run the setup script:
bash ./setup.sh
-
Test your installation!
poetry run event-feed-ingest --help
-
Head on over to running
event-feed-ingest
.
If you'd prefer to run commands by hand or are running Linux (or WSL), check out the advanced setup instructions
- Find an issue you'd like to help out with
- Set up a development environment
- Read up on how our pipeline works
- Run the pipeline locally
Some pre-built tools to help fetch or parse common data types