Consolidate efforts

[fgregg]

Hi News Apps Team!

I've been working on a very similar project to this, https://github.com/datamade/school-report-cards

There are a few differences.

1. The data transformation pipeline is expressed in a Makefile and is more oriented to the command line
2. We have two output targets: csv files with year-over-year consistent column names and a normalized database.

It seems like the biggest difference here is between your python oriented process and our Makefile oriented process. Any chance you guys might come over to our way of doing it, so we can combine efforts?

Best,

Forest

[ghing]

I think I'd definitely be interested in consolidating efforts and don't have strong feelings about using a Python taskrunner vs make (though it's been a looooooooong time since I've written a Makefile).

I should chat with our data reporter and education reporter to see what our long-term needs are, but I can't imagine there's any reason we'd want to replicate the same data cleaning work.

For programatic access, if we need it, it would be easy enough to write a wrapper around the csv/postgres database.