WebAssembly build of LevelDB

[puzrin]

It would be nice to have webassembly build of leveldown. There are many cases when platform independent build is more preferable than top performance.

Example: wrapped Google's woff2 https://github.com/fontello/wawoff2. ~ 2x slower than native, but no need to recompile, download and so on. Convenient.

[vweevers]

FWIW you might be better off with one of the many compatible implementations, following the same interface as leveldown. As for compiling: leveldown has prebuilt binaries for linux, windows, osx and electron. If your platform isn't supported, please do tell us.

Of course a WebAssembly build would be fun, don't let me stop you!

[mafintosh]

how would a webassembly build deal with all the file io happening in leveldb?

[vweevers]

Emscripten can simulate a file system using IndexedDB (source)

🤣

[vweevers]

Revisiting this thread and reading my comment above, I now realize it may seem I was dismissing the idea, apologies.

Given that:

1. Chrome is gonna be removing snapshots from IndexedDB in favor of blocking readwrite-transactions (which negatively affects the performance of level-js, concurrent streams in particular)
2. This is how other engines already work (Chrome is the only engine that snapshots)
3. Emscripten does have a file system based on File and Blob (only in workers though)

I think it's worth our time to (at the very least) explore a WebAssembly build of LevelDB.

[vweevers]

Emscripten does have a file system based on File and Blob

Scratch that:

This file system provides read-only access to File and Blob objects.

So in browsers, IndexedDB is still the only option. Ugh.

[puzrin]

When i created this issue, i forgot about fs IO. Intial idea was to have portable lib with almost zero cost. Compiling to WASM does not seems as easy as expected. Probably this can be closed until far future...

[vweevers]

@puzrin In case you opened the issue with Node.js in mind, Emscripten has NODEFS, which:

lets a program in node map directories (via a mount operation) on the host filesystem to directories in Emscripten’s virtual filesystem. It uses node’s synchronous FS API to immediately persist any data written to the Emscripten file system to your local disk.

It being synchronous is problematic.

As for browsers, I think we need to push browser makers a little bit. IndexedDB is going in the wrong direction IMO and no alternative is in the works AFAIK (except for kv-storage which is a Web API built on top of IndexedDB, inheriting its flaws).

[puzrin]

I don't know if port to browser needed at all. Even if needed, IMHO performance is not required there and any FS backend could be ok. My intent was to get fast enougth universal alternative for node.js only (slowdown 2x may be acceptable price).

But as i said, idea was to get all this sweets with almost zero effort (as was done with woff2). Now i understand, things are not so easy for this package. Seems not all of my ideas are clever :)

[vweevers]

I don't know if port to browser needed at all. Even if needed, IMHO performance is not required there and any FS backend could be ok.

The problem with IDB is not the performance of an individual read or write. Besides feature bloat and hand-holding, the basic semantics of transaction lifetime are broken. This affects overall performance of an application, esp. one with long-running concurrent reads and writes. It's strange that backpressure is not a first-class citizen (in a runtime environment that has a UI thread!), or even recognized by spec and engine authors as a real need. I wouldn't mind if IDB stripped 90% of its features, in favor of a focused core with solid primitives. Lower level is better.

My intent was to get fast enougth universal alternative for node.js only (slowdown 2x may be acceptable price). [..] Now i understand, things are not so easy for this package.

Roger. I think that settles the node-related discussion, for now at least. Thanks for opening this issue!

The browser-related discussion is out of scope for leveldown, so I'll move this elsewhere.

[tracker1]

I know it's a little old... was literally thinking that this would be really cool combined with node, in order to support "any" platform from a single build... but that the storage primitives would need to be loaded somehow in a compatible/secure way. I don't think this is close to standardized yet though.

[vweevers]

IMO we've come pretty far already with level@5, which ships prebuilt binaries for linux, mac, windows, arm, android, and it works in browsers. It means most of the time you don't need to build.

I know it's a little old

I'd say it's young ;)

[tracker1]

@vweevers understood... I'm glad that this group has taken the time to ensure prebuilt binaries (saves a lot of time in build/release cycles)... as to old, I mean this issue being nearly a year old. ;-)

[only-cliches]

Recently attempted this and got pretty far.

The biggest issue I ran into is LevelDB can't get a proper file lock because Node's fs module doesn't provide the capability. Even with -s FORCE_FILESYSTEM=1 -s NODERAWFS=1 emcc flags LevelDB reports an IO error when attempting to lock the database files.

At the moment it seems impossible to make a working webassembly build of LevelDB that actually saves to the disk.

I think it might work if there's some way to spoof the lock, but I'm not knowledgable enough at the moment to make this work. It might also work if you tell emscripten to use a memory filesystem.

The only use case I can see the webassembly build for is opening LevelDB stores inside the browser. You could generate the LevelDB database on the server, then load the files into webassembly with ajax and get the LevelDB api in the browser. Honestly I think there are better libraries for this use case, so the webassembly build seems pointless until better file system support is provided in emscripten.

Edit: FYI the error LevelDB spit out is identical to the one in this issue.

One other thing, I was trying to build exactly what's been suggested here a few times, a webassembly powered key-value store that runs in NodeJS and doesn't require any native compilation. The result is here. I tried several C/C++ powered stores (lmdb, unqlite, leveldb, sqlite4 LSM) and ran into various issues with all of them. In the end the only database I found that would compile to webassembly and actually save to disk was sqlite. It's probable that redis would also work but I wanted something more robust for data security.

[jacoscaz]

Not an expert but this might have become possible with the introduction of the Origin Private File System, also referenced in Level/browser-level#7 .

EDIT: good pointers: emscripten-core/emscripten#19408
EDIT: more helpful pointers: WordPress/wordpress-playground#1030 (comment)