cif_loop_add_packet bottlenecks for input with large loops (~30000 rows) on ARM64

[DylanLukes]

In an application I am working on, cif_loop_add_packet has become a bottleneck on loading large CIF files. Real world examples include large protein complices such as RNA polymerase II.

The root of this problem seems to be that cif_api inserts packets into SQLite one at a time with cif_loop_add_packet. Profiling suggests that the issue is the accumulation of SQL overhead. To give some concrete numbers: for a 2.3MB CIF file containing a single entry, with a loop containing ~34,000 packets, on an Apple A9X ARM64 SOC, parsing takes up ~44s, of which 89% is spent in sqlite3_step.

I suspect this may be related to how the build configuration of iOS' system sqlite3, in particular journaling and caching configuration.

[DylanLukes]

I built an additional copy of libcif.a, disabling synchronous writes and keeping the journal in memory by adding DEBUG_WRAP(sqlite3_exec(temp->db, "pragma synchronous = off; pragma journal_mode = memory;", NULL, NULL, NULL)); to cif_create here:

cif_api/src/cif.c

Line 230 in 1ddf445

/* Any other DB setup / configuration needed in the future should go here */

In this case, for the same file, ~20s is spent in cif_parse (of which 73% is spent in sqlite3_step).

For comparison, a copy of libcif.a with SQLITE_MEMORY_ONLY enabled takes only ~13s (of which 85% is spent in sqlite3_step)