Support more open datasets

[utdemir]

This is an umbrella issue to gather useful datasets which can be used freely.

Things to consider:

• We should be able to access them freely & quickly. It helps if the data is provided in a partitioned fashion (a set of files).
• It should be sufficiently large. There's no point using distributed-dataset if the data is smaller than ~10 gigabytes. It's even more impressive if the data is hundreds of gigabytes/terabytes (eg. gharchive).

Useful Collections:

• Registry of Open Data on AWS
• Google Cloud Public Datasets
• Kaggle Datasets (these tend to be quite small)

[utdemir]

Amazon Customer Reviews Dataset

• 30-50GB, tsv & .parquet.snappy, HTTP & S3

[utdemir]

CommonCrawl

• petabytes (partitioned by date), mix of uncommon formats (WARC, WAT, WET...) , HTTP & S3

[utdemir]

Global Database of Events, Languages and Tone

[utdemir]

It might be an interesting experiment to implement a dataset of the Bitcoin transactions, if we have a way to process them in a partitioned way.

[aycanirican]

https://www.ligo.caltech.edu/page/ligo-data

[utdemir]

https://www.data.gov/ has a lot of open datasets. They tend to be small on size, but there probably are exceptions.

[utdemir]

Tons of taxi trips, the partitioning seem ideal for distributed-dataset. Will have to investigate how performant the website is.

https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page

[arobertn]

Not mentioned (as far as I could see) in google's list:

http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html

Googling for "google ngram data" will turn up lots of scripts various people have developed for munging this data.