Extend 2 billion row benchmarks e.g. memory usage, sorting, joining, by-reference

[arunsrinivasan]

We've currently gone to 2E9 rows (the 32bit index limit) with 9 columns (100GB). See benchmarks page on wiki.

Ideally it would be great to compare all available tools that are either specifically developed for large in-memory data manipulation or are capable of handling data at these sizes much better than base. Of course base-R should also be included, typically as control.

Aspect of benchmarking should be to highlight not just run time (speed), but also memory usage. The sorting/ordering by reference, sub-assignment by reference etc.. features, for example, at this data size should display quite clearly on speed and memory gains attainable.

[mattdowle]

For memory usage, perhaps: https://github.com/gsauthof/cgmemtime

[arunsrinivasan]

Figured cgmemtime out. Quite straightforward. Just need some time to run through grouping/joins/reshaping benchmarks.

[jangorecki]

I would like to close this one as it is already epic, and will be epic for a long time, due to broad scope defined here.
Work on benchmarks has been shifted to a dedicated h2o project https://github.com/h2oai/db-benchmark and related issues are:

• memory: measure memory usage h2oai/db-benchmark#9
• join: advanced questions for join tests h2oai/db-benchmark#18
• sort: sort task h2oai/db-benchmark#61
• by reference ops: "by reference" operations task h2oai/db-benchmark#62