Have it work on Node using LevelDB for persistence?

[brianjenkins94]

IndexedDB under the hood is just LevelDB.

Could lovefield-ts be made to persist to LevelDB on Node?

[brianjenkins94]

Here's as far as I got:

import * as lf from "./lovefield-ts/dist/lf";

const DeferredLevelDOWN = require("deferred-leveldown");
const encode = require("encoding-down");
const leveldown = require("leveldown");

const options = {
	"valueEncoding": "utf8",
	"keyEncoding": "utf8"
};

const callback = function() { };

const indexedDb = require("level")("db"); //new DeferredLevelDOWN(encode(leveldown("db", options), options), options, callback);

// @ts-expect-error
globalThis["window"] = {};

// @ts-expect-error
globalThis.window["indexedDB"] = indexedDb;

(async function() {
	const schemaBuilder = lf.schema.create("todo", 1);

	schemaBuilder
		.createTable("Item")
		.addColumn("id", lf.Type.INTEGER)
		.addColumn("description", lf.Type.STRING)
		.addColumn("deadline", lf.Type.DATE_TIME)
		.addColumn("done", lf.Type.BOOLEAN)
		.addPrimaryKey(["id"])
		.addIndex("idxDeadline", ["deadline"], false, lf.Order.DESC);

	// In node.js, the options are defaulted to memory-DB only. No persistence.
	const db = await schemaBuilder.connect({
		"storeType": 0
	});

	const data = [{ "id": 1, "description": "get some coffee", "deadline": 1587947145675, "done": false }, { "id": 2, "description": "prepare demo", "deadline": 1587948145675, "done": false }, { "id": 3, "description": "completed task", "deadline": 1587847145675, "done": true }, { "id": 4, "description": "get enough sleep", "deadline": 1587847045675, "done": false }];

	let item = db.getSchema().table("Item");

	const rows = data.map((d) => item.createRow({
		"id": d.id,
		"description": d.description,
		"deadline": new Date(d.deadline),
		"done": d.done
	}));

	await db.insertOrReplace().into(item).values(rows).exec();

	item = db.getSchema().table("Item");

	const res = await db
		.select()
		.from(item)
		// @ts-expect-error
		.where(item["done"].eq(false))
		// @ts-expect-error
		.orderBy(item["deadline"])
		.exec();

	// @ts-expect-error
	for (const row of res) {
		console.log("Finish [", row.description, "] before", row.deadline.toLocaleString());
	}
})();

But I run into issues here:

lovefield-ts/lib/backstore/indexed_db.ts

Lines 82 to 88 in b344a93

	let request: IDBOpenDBRequest;
	try {
	request = indexedDB.open(this.schema.name(), this.schema.version());
	} catch (e) {
	reject(e);
	return;
	}

Since my LevelDB instance masquerading as IndexedDB has a completely different function signature for open() (not to mention it's already in the status of "open", which could cause additional issues).

[brianjenkins94]

Maybe this is easier than I thought.

I may be able to just duplicate https://github.com/arthurhsu/lovefield-ts/blob/master/lib/backstore/indexed_db.ts and replace IndexedDB with the equivalent LevelDB calls.

[brianjenkins94]

LevelDB not having transactions without cshum/level-transactions seems like it's going to be a problem.

[arthurhsu]

The design of BackStore is an abstraction layer to provide a backing storage for query engine. The requirement is the backing store needs to support ACID style transaction. IndexedDB, LocalStorage both claims to support that. JS memory by nature is ACID due to its single-thread policy, so MemoryStore is possible.

For other stores, AFAIK none of them can meet the ACID requirements. Even if some of them claim to, I'd like to see a thorough test suite to prove it before adopting to Lovefield. For example, sql.js claims to do a lot, but the tests I can find on GitHub is not even a fraction of existing sqlite unit tests. So good luck using it.

(Side note: Lovefield-ts has a super extensive test suite with code coverage 90%+)

If they do, then you can add three classes to enable it. See memory/memory_tx/memory_table for the minimum example. I'll accept the pull request if proper tests are added and passing (see tests/backstore/memory_back_store_test.ts and tests/backstore/memory_table_test.ts for example).

[brianjenkins94]

Any suggestions on other tests I can run to help identify gaps in functionality?

[arthurhsu]

See if you can pass Lovefield's mocha test suite. If so that would be promising.
That means, you should be writing something like memory_back_store_test.ts and run using gulp test.