Flat Data Editor for crowd-sourcing data through user-friendly PR interface

[aspiers]

I've only just discovered this great project, so forgive me if this question is somewhat uninformed ...

It seems that the Flat Data paradigm is currently based on Simon Willison's git-scraping model, with the assumption that the authoritative source of data is external to git, and will be fetched / updated via an HTTP or SQL request. And of course this is a great model for many use cases. But I have use cases in mind (involving public data) which would require a slightly different approach to maintaining the data:

• The data would still be kept in flat files within git, thereby harnessing all the same tools and benefits of Flat Data.
• However, rather than updating by fetching from an external source, the data would be maintained by crowd-sourcing changes from users which could then be reviewed and merged by the maintainers. Since the GitHub PR UI is too technical for most users, especially when changes to CSV or JSON files are presented in a diff-like format, this would require a simple web UI which allows editing of cells within the data, and then submitting those edits as a change request which would be a PR behind the scenes.

One example use case is crowd-sourcing data about public transport networks which is not already openly accessible. The data set could be bootstrapped by project founders, and then opened up for the world to contribute to, a bit like OpenStreetMap but of course with a drastically reduced and simplified data set.

This alternative approach to data sourcing within git repos could be called "git sourcing", serving as a complementary technique to "git scraping". Indeed I could imagine scenarios where they could be combined; for example git scraping continually fetches / updates datasets from public APIs, and then git sourcing allows the general public to apply manual corrections / modifications on top of those via git rebase or merge. (The question of how to resolve merge conflicts in this scenario is left as an exercise for the reader 😉 )

So my questions are:

• Does anything like this (non-technical UI allowing users to submit PRs modifying CSV/JSON etc.) exist already?
• If not, can anyone suggest the quickest/easiest approach to building it on top of existing code (Flat Data or otherwise)? For example, might it make sense to consider extending the Flat Viewer so that it becomes read-write rather than just read-only?

Thanks!

[Wattenberger]

This is a wonderful suggestion, thanks for writing it out! We unfortunately don't know of anything existing that would support this workflow, but spent some time adding "edit" functionality to the Flat ui React component:

https://github.com/githubocto/flat-ui/releases/tag/v0.11.0

We'd love to hear if anyone has other suggestions here, or wants to build an interface that creates a PR using the flat-ui component!

[aspiers]

Thanks, really glad you see value in it! And the edit.mov demo looks awesome - am super impressed you implemented that so quickly! Do I understand right that the edits don't currently persist, and you are asking for people interested in building a non-technical UI to wrap around the traditional GitHub Flow PR creation UX, so that these edits could be committed as a batch into a PR? That would be so cool, I'm sure it would spawn all kinds of interesting use cases. In sense it's a bit like part of what Airtable offers, except with simpler data models.

[Wattenberger]

That's spot on! The flat-ui component could be used similarly to an input, where the wrapper keeps track of the original and edited state, then creates a branch & PR with those changes. I totally agree - it's something that would make a way broader group of people comfortable with editing data on GitHub!

[aspiers]

I have discovered https://github.com/dolthub/dolt and DoltHub which also look like very promising technologies in this space, and perhaps could be used as alternative techniques for the "git sourcing" approach I propose above.