What is the (simplest) way to maintain datasets in github.com/datasets?

[rufuspollock]

Quite a few of the core datasets are not getting updated. This raises the questions how we maintain the core datasets e.g.

• What patterns and tooling for creating and running data pipelines
• orchestration / running of that e.g. github actions or something else
• how we log errors
• how we (socially) know what datasets need updating and when (i.e. their frequency)

We could try this out re work of getting core datasets up to date #376

[sabas]

I try to maintain my main dataset (UN/LOCODE) when I remember, perhaps it could be useful to have a way of monitoring when a new release is needed by scraping the page from which we grab the dataset?
On datahub I see my dataset is stuck to 3 years ago, why?

[rufuspollock]

@sabas your updates are much appreciated.

perhaps it could be useful to have a way of monitoring when a new release is needed by scraping the page from which we grab the dataset?

Interesting suggestion. I wonder what kind of infrastructure would be required? Also how would you deal with false positives e.g. page changing but not data?

Another option would be to just to have some a pipeline that is run regularly plus some data validation - then if pipeline errors or validation fails we know something has happened.

On datahub I see my dataset is stuck to 3 years ago, why?

This relates to some bitrot in the loading infrastructure and other items in backend of current platform for running DataHub.io. We are currently working on updating that - see https://datahub.io/blog/we-have-some-important-updates-re-datahub

[sabas]

A life ago I used https://morph.io/ to make a scraper for a dataset I used when I wrote my bachelor thesis on Open Data, nowadays perhaps there are other solutions (Github Actions?). For the second question it's up to the person writing the scraper to decide how to detect when a dataset is updated.
In my case I could look at https://unece.org/trade/uncefact/unlocode and check for the Last Update date or the release number.

[davidgasquez]

Interesting discussion! From my experience, I think these ideas might help:

• Wrap the scripts in Docker images. This will ensure increase compatibility and make it easier for the next person to pick things up.
• Create a common entrypoint for all repositories. Personally, I rely on very simple Makefiles. They are a great way to document and consolidate different projects. Each project has a make that "runs" it, make data that gets the data, and perhaps a make deploy that deploys it. Language and tool independent, aswell as available almost everywhere.
• In terms of orchestration it becomes much easier if all datasets have a Dockerfile. You can run things in GH Actions but swap it easily if that doesn't work in the future. If all repositories follow the Dockerfile + Makefile pattern, a templated GH Action might be great since it could be deployed with one click. Might be overkill though!
• Now, "How we (socially) know what datasets need updating and when (i.e. their frequency) might be much trickier. I think a great starting point might be having a few badges in the repository with things like script status, latest time the script was run, ...

[rufuspollock]

@davidgasquez great suggestions. wdyt of creating a template repo in github.com/datasets e.g. dataset-template with a standard basic setup e.g. makefile, docker image, datapackage.json (?), README etc.

[davidgasquez]

Might be a nice approach for new packages! I can give it a go if that sounds good.

[rufuspollock]

Let's go for it.