Speed comparison with DataPusher

[davidread]

Summary

Express Loader loads the data in 11.4 times the speed compared with DataPusher

Test conditions:

• Load of Boston 311 dataset (1033882 rows, 475MB)
• Run locally on a MacBook Pro (i7, 2013 model)

stats with ckanext-xloader

12s - retrieve the file (over HTTP) from local FileStore
23s - convert to UTF8
21s - copy CSV file into PostgreSQL table (one COPY command)
160s - create search index

Total: 206 seconds

At this point the full data is made available to the user.

Afterwards the column indexes are generated which simply speed up common queries - this takes a further 1262s. However we exclude this from the load time, as it is merely an optimization.

stats with datapusher

12s - retrieve the file (over HTTP) from local FileStore
2338s - convert to UTF8 and then to JSON, setup postgres indexes to be generated during load, load JSON into table (4000 INSERT statements).

Total: 2350s