MWS: Add support for node-sqlite-wasm alongside better-sqlite3 #7996
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Seems to be slower, but might make cloud deployments easier by not having any binary dependencies
We will make this configurable
Gives a 20% reduction in startup time on my machine
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
20 tasks
Astonishingly, Azure does this
I am not intending to make this a long term feature. We will choose one engine and stick with it until we choose to change to another.
https://github.com/tndrle/node-sqlite3-wasm doesn't have transaction support so I've tried to implement it using SQL statements directly. @hoelzro do you think this is right? Should we be rolling back the transaction in the finally clause? It would be nice to have tests in this area... I looked at better-sqlite3's implementation - https://github.com/WiseLibs/better-sqlite3/blob/master/lib/methods/transaction.js
|
I've now added support for both better-sqlite3 and node-sqlite3-wasm at the same time. Switching between them requires a change to the default in Even with transactions, node-sqlite3-wasm is about 10 times slower than better-sqlite3, which is not encouraging. I had hoped that @hoelzro's transaction support would help but it appears not. Anyhow, I think this is ready to merge. We can continue to experiment with the choice of engine, and explore improvements to node-sqlite3-wam |
Jermolene
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduction
This PR targets the the Multi Wiki Server PR. It swaps the SQLite implementation from better-sqlite3 to node-sqlite3-wasm.
The difference is that better-sqlite3 consists of JavaScript bindings around a copy of the SQLite3 engine that has been compiled to native code while node-sqlite3-wasm is an adapation of the official SQLite3 Wasm distribution that has been modified to work on Node.js.
At the moment, this change introduces a very significant 100x performance degradation. The measured time for loading the initial bags and recipes is:
The performance degradation needs to be analysed and addressed before we can consider merging this PR. One positive observation is that the performance degradation goes away when using a memory database, so there's a good chance that file I/O can be optimised further.
Note that this PR currently includes switching to a memory based database to make testing easier.
Why switch to SQLite3 Wasm?
It fits with our long term plans of using the same SQLite3 engine both in the browser and under Node.js. The advantage is that it allows TiddlyWiki to continue to be self contained, without requiring a complex installation and build step.
More immediately, I have run into many problems getting better-sqlite3 running correctly on an Azure Web Service which are all due to binary compatibility issues. It's an area in which I have no expertise and I find myself constantly confused, particularly given the long turnaround times for making changes in cloud environments like Azure. If I am having difficulties with this then I think there's a good chance that these problems will impact ordinary users of TiddlyWiki as well. (The project where I have been exploring running TiddlyWiki on Azure can be seen at https://github.com/Jermolene/tiddlywiki-azure-ci).
In contrast, the Wasm implementation just works on Azure without any fuss and bother, and offers the promise of a single codebase for the browser and the server.
Questions
Appreciation of node-sqlite3-wasm
I'd like to mention @tndrle to congratulate them on a nicely conceived and very neatly executed project. There was very little work involved in switching from better-sqlite3 to node-sqlite3-wasm (the commit is here), and it's been a pleasure to work with it.
I hope that the SQLite team themselves might eventually address the Node.js gap themselves, but in the meantime @tndrle's work is much appreciated.