SQLite implementation of query support for DB based index

[habdelra]

This PR provides a SQLite implementation of queries for our DB based index. A handy test index setup helper is also included in this PR. This PR only focuses on the type filter and eq filter. And specifically no containsMany field support yet.

[github-actions]

Test Results

558 tests  +8   554 ✔️ +8   7m 56s ⏱️ -10s
    1 suites ±0       4 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 5feaf04. ± Comparison against base commit 2f163c1.

This pull request removes 10 and adds 18 tests. Note that renamed tests count towards both.

Chrome 123.0 ‑ Unit | index-db: can create a new generation of index entries
Chrome 123.0 ‑ Unit | index-db: can get "production" index entry
Chrome 123.0 ‑ Unit | index-db: can get work in progress index entry
Chrome 123.0 ‑ Unit | index-db: can perform invalidations for an index entry
Chrome 123.0 ‑ Unit | index-db: can prevent concurrent batch invalidations from colliding
Chrome 123.0 ‑ Unit | index-db: can prevent concurrent batch invalidations from colliding when making new generation
Chrome 123.0 ‑ Unit | index-db: can remove an index entry
Chrome 123.0 ‑ Unit | index-db: can update an index entry
Chrome 123.0 ‑ Unit | index-db: does not create invalidation record for non-JSON invalidation
Chrome 123.0 ‑ Unit | index-db: returns undefined when getting a deleted entry

Chrome 123.0 ‑ Unit | indexer: can create a new generation of index entries
Chrome 123.0 ‑ Unit | indexer: can get "production" index entry
Chrome 123.0 ‑ Unit | indexer: can get work in progress index entry
Chrome 123.0 ‑ Unit | indexer: can perform invalidations for an index entry
Chrome 123.0 ‑ Unit | indexer: can prevent concurrent batch invalidations from colliding
Chrome 123.0 ‑ Unit | indexer: can prevent concurrent batch invalidations from colliding when making new generation
Chrome 123.0 ‑ Unit | indexer: can remove an index entry
Chrome 123.0 ‑ Unit | indexer: can update an index entry
Chrome 123.0 ‑ Unit | indexer: does not create invalidation record for non-JSON invalidation
Chrome 123.0 ‑ Unit | indexer: returns undefined when getting a deleted entry
…

♻️ This comment has been updated with latest results.

[lukemelia]

Looking good

[jurgenwerk]

Appreciate all the comments, looks well thought out.

[jurgenwerk]

What do you mean by cursors in the classic sense? I am familiar with implementing cursor based pagination where the last value of the column from the previous query is selected as the cursor for the next set. This sounds like it would be achievable with SQLite also. Just wondering what kind of pagination you had in mind here

[habdelra]

SQLite does have a cursor available, but it’s only available from the synchronous API.

Normally you get it like this:

import sqlite3;
con = sqlite3.connect("tutorial.db");
con.cursor()cur = con.cursor();
result = cur.execute("SELECT name FROM sqlite_master");
result.fetchone()

however from the async interface you no longer have a direct handle on the connection. Rather the connection is only available in the worker thread since it’s not possible for the connection to be serialized over the postMessage boundary.

The worker thread only has a couple API calls:
close(), config-get(), execute(), open(). https://sqlite.org/wasm/doc/trunk/api-worker1.md
the idea being only POJO’s are available to cross the postMessage boundary between a worker thread and the main thread

[habdelra]

So instead we’ll have to manually control the result set as I described in this comment. We’ll need to use the realm version so that we can prevent the index mutations made between pages from effecting the results (so the realm version will be part of the query), as well as only fetching one page at a time using the ROW_NUMBER() and/or LIMIT sql commands by having the caller keep track of what page number of results they are requesting and including that in our Query interface.

I made a pagination linear ticket for this work.

[jurgenwerk]

Ah! I see now what you mean by the cursor and the boundary.