Large amounts of data

[rufuspollock]

Pandas and R both support very large datasets seamlessly. This will usually require some special work and avoid loading all of data into memories. Most libraries I have seen in JS do not do that - they load all data into memory (and fairly inefficiently).

For example, the classic http://learnjsdata.com/ uses lodash and does everything in memory.

This would be especially important for the browser where even loading fairly small datasets into memory (e.g. 6Mb CSV) may cause performance issues (esp on e.g. a phone).

Solutions

A solution requires "backing" your dataframe API implementation on a backend stores that can leave most data on disk or in some special storage mechanism.

• Browser: For example, for the browser using IndexedDB to hold the data.
• Node: ?? - you can just use disk (but you would then need to wrap yourself)