lib,src,test,doc: add node:sqlite module

[cjihrig]

#53264 has been open for over a month with no objections, so I am opening this PR with an initial node:sqlite module. There is other functionality that could potentially be exposed in the future, but I believe this is enough for an experimental MVP.

Fixes: #53264

Summary: Node.js now includes a built-in sqlite module (require('node:sqlite')) that becomes available when using the --experimental-sqlite flag

The following example shows the basic usage of the node:sqlite module to open
an in-memory database, write data to the database, and then read the data back.

import { DatabaseSync } from 'node:sqlite';
const database = new DatabaseSync(':memory:');

// Execute SQL statements from strings.
database.exec(`
  CREATE TABLE data(
    key INTEGER PRIMARY KEY,
    value TEXT
  ) STRICT
`);
// Create a prepared statement to insert data into the database.
const insert = database.prepare('INSERT INTO data (key, value) VALUES (?, ?)');
// Execute the prepared statement with bound values.
insert.run(1, 'hello');
insert.run(2, 'world');
// Create a prepared statement to read data from the database.
const query = database.prepare('SELECT * FROM data ORDER BY key');
// Execute the prepared statement and log the result set.
console.log(query.all());
// Prints: [ { key: 1, value: 'hello' }, { key: 2, value: 'world' } ]

[nodejs-github-bot]

Review requested:

• @nodejs/gyp
• @nodejs/loaders
• @nodejs/startup

[targos]

I can't review the C++ code, but API and tests LGTM for a MVP.

[benjamingr]

Yay, happy to see this come to fruition. Let's land as experimental and iterate 🎉

[benjamingr]

It would be nifty to also support Symbol.dispose here but that can be in a follow up PR.

[benjamingr]

This is a lot more feature complete and ready than I was expecting, good job Colin!

[RedYetiDev]

This is adding a new feature, so shouldn't be semver-minor? (and with this kind of change, notable-change as well?)

[ShogunPanda]

Love it! Amazing job @cjihrig

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/60156/

[Qard]

Should we be concerned at all about the performance impact of people using these sync APIs within requests and blocking the event loop? With sync being the only option here I feel like the risk is increased.

If the concern is simplicity, I feel like only async would be better than only sync. Especially given top-level await. What reason is there to favour sync over async here?

I won't block on that concern, and in fact I think the work done so far is great. I would be happy landing what's here now. I just wanted to share my concern with going for sync first rather than async first.

[benjamingr]

Should we be concerned at all about the performance impact of people using these sync APIs within requests and blocking the event loop? With sync being the only option here I feel like the risk is increased.

SQLite runs in process and is often CPU bound, we do have async APIs for some CPU intensive stuff (like in crypto) but I suspect for most people this API is the better one performance wise (that said, in some cases an async API is preferable and we should expose that as well)

[targos]

none because a specific backport PR was not needed

[karlhorky]

If anyone wants to try out node:sqlite interactively in a sandbox, I've set up a CodeSandbox with @hemanth's node-nightly here:

CodeSandbox demo: https://codesandbox.io/p/devbox/node-sqlite-with-node-js-nightly-vygw6p?file=%2Findex.js

[karimfromjordan]

I just tested this and it's very slow and appears to ship with SQLite 3.46. Is this expected in the Nightly build?

[bholmesdev]

@karimfromjordan I think that's a limitation of codesandbox. I was unable to run the sandbox on my machine, and I was also unable to get node-nightly to succeed. Luckily, I found these instructions on how to install the latest nightly release using nvm. This worked for me:

NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/nightly/ nvm i node

Running against an index.mjs file with the snippet from those PR notes ran very fast for me. Your machine may vary of course.

[karimfromjordan]

@bholmesdev I tested it locally. I ran 1000 inserts, with WAL enabled and it completes in about 1 - 1.5 secs which is really slow. better-sqlite3 finishes the same statements in 50 ms.

[RedYetiDev]

This PR is about the addition of SQLite into Node.js. If you have concerns about its usage, please visit the help repo.

If you have concerns specific to the sandbox environment, please reach out to @karlhorky privately.

[cjihrig]

@karimfromjordan thanks for sharing. I can confirm a slowdown, but I'm not seeing anything near 20x - more like 4x. Please note this is not optimized at all yet. When I start working on this again, I'll look into improving the perf if no one beats me to it.

[mpnkhan]

This is exciting. I always loved the PHP+MySql combo. And this one is one more reason for me to love nodeJS more.

[cjihrig]

From a quick test, compiling SQLite in Node with the same settings used by better-sqlite3 yielded roughly the same performance. There may be other things to optimize as well, but it's good to know we aren't doing anything too awful 😄.

[thomasdao]

I'm really grateful that Sqlite is natively supported in NodeJS and want to thank @cjihrig and everyone involved for working on this feature. Thanks for choosing Sqlite - it's fast and much more reliable than most alternatives. This feature would be super useful for Electron users - currently most Sqlite bindings don't work well for Electron apps, and having native support for Sqlite would make it easier to interact with the database.

[louwers]

This is awesome! Is support for official SQLite extensions something that is being considered? In particular I am interested in the Session Extension.

[cjihrig]

@louwers it hasn't been discussed AFAIK, but there is a PR for it, so it's not out of the question: #53900.

EDIT: Oh, that one is built in already, we would just need to support enabling it.

[louwers]

@cjihrig Actually the Session Extension is not a runtime loadable extension. It needs to be compiled into SQLite.

To be able to use it with Node.js you would need to wrap its C API: https://www.sqlite.org/session/intro.html

[TheOneTheOnlyJJ]

To be able to use it with Node.js you would need to wrap its C API: https://www.sqlite.org/session/intro.html

+1 for this.

Supporting all of the sqlite3 amalgamation extensions should be a strong goal for Node.js (through wrapping the C APIs of each).
While at first glance they may not look relevant, these extensions are very powerful & vital for the projects that make use of them.

Maybe a new issue concerning this topic specifically should be opened?

[louwers]

@cjihrig @TheOneTheOnlyJJ I opened a PR so we can have some discussion about the API (among other things): #54181

[webda2l]

@cjihrig Thanks for your work!

Regarding this case, I am a bit skeptical because better-sqlite3 allows for more named parameters to be passed than those actually used in the SQL. This can be useful when you build your SQL with conditions in a literal string, for example.