LMDB on Electron renderer

[thomasdao]

Hi,

I've tried to integrate this library into our Electron app and want to use LMDB on the renderer process. However when I import the library import { open } from 'lmdb', I found the error below:

TypeError: The URL must be of scheme file

I guess the error is from the native.js file:

let dirName = dirname(fileURLToPath(import.meta.url)).replace(/dist$/, '');

My project uses Webpack to bundle JS files, and have set lmdb as externals in the webpack.config.js file, so that Webpack treats the module as external module and doesn't try to bundle it.

I wonder is there any other way to detect the dirName or work around this issue? I've tried to log import.meta.url and found it starts with scheme file://: file:///Users/username/project/home/src/renderer/main.js, so I don't really know why the error about The URL must be of scheme file.

Thank you.

[thomasdao]

Update: I got it working!

It turns out that this library works fine with Webpack and doesn't really require any special treatment! So I removed lmdb from Webpack externals settings.

The only thing I need to modify is that SharedArrayBuffer is not available by default in Electron, so I need to set the required header when create a new BrowserWindow:

// Enable SharedArrayBuffer
// https://stackoverflow.com/a/74623813/622510
browserWindow.webContents.session.webRequest.onHeadersReceived((details, callback) => {
  details.responseHeaders['Cross-Origin-Opener-Policy'] = ['same-origin'];

  // Needs to set `Cross-Origin-Embedder-Policy ` to `credentialless` so that images from other sources
  // can be loaded.
  // https://github.com/ffmpegwasm/react-app/issues/3#issuecomment-1016234599
  details.responseHeaders['Cross-Origin-Embedder-Policy'] = ['credentialless'];
  callback({ responseHeaders: details.responseHeaders });
})

And now I can access the database from the renderer process. Thanks a lot for the library!

I do have another question: is it possible to access the same database from multiple renderer process? Or should I only access the DB from one process, and use ipcRenderer to return data to other process? Thank you.

[thomasdao]

Update: unfortunately, changing response header for all requests causes many security issues, so I have to patch write.js file and replace all SharedArrayBuffer with ArrayBuffer, and LMDB works normally.

However, is it safe to do so, if I only use LMDB in one main thread and does not access the DB elsewhere? Thank you!

[kriszyp]

I do have another question: is it possible to access the same database from multiple renderer process?

Yes, LMDB is designed to support multi-process access to the same database, and handles locking between processes.

replace all SharedArrayBuffer with ArrayBuffer, and LMDB works normally. However, is it safe to do so

Yes, that should be safe to do. lmdb-js doesn't actually share the buffers between different JS worker threads, there is only shared access with the native C threads, and I don't believe SharedArrayBuffer and ArrayBuffer differ at all with respect to that, they just have different interfaces/functionality that they support (which varies across platform and version in rather unpredictable ways), and the important functionality is access to the Atomics API which, in most recent versions of V8, work just fine with ArrayBuffers (and if they don't they would immediately error out).

When you say that SharedArrayBuffer is not available, does that mean the global variable doesn't exist? I could certainly add some code to check for the presence of SharedArrayBuffer and fallback to ArrayBuffer if it is not there.

[thomasdao]

Yes, the global variable SharedArrayBuffer doesn't exist if I don't enable the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy. The error is below. It would be great if you could add the check so the manual patch can be removed. Thank you :)

ReferenceError: SharedArrayBuffer is not defined

[thomasdao]

I can confirm version 3.0.0 works fine, thank's for fixing :)