Move RocksDB to git submodule

[vweevers]

Forked from #82. Closes #140.

I've yet to review it myself. I only rebased it, and removed the hardcoded sha1 for build_version.cc, which I know we will forget to update at some point - so I prefer not having it.

[vweevers]

Remaining tasks:

• Review
  • Can probably remove 'rocksdb/util/' from include_dirs now
• Merge Removed additional build step, freezed build_version, updated rocksdb.gyp #142
• Check .gitignore and .npmignore (we made some changes there, make sure paths still match)
• Upgrade to at least facebook/rocksdb@3b4d1b7 to avoid a regression of Add sys/sysmacros.h for build on modern glibc #86
• Test compiling inside npm package (without git metadata being available)

[vweevers]

@lu4 How's it going with the RocksDB upgrade? Do you wanna take it from here?

[lu4]

Here is the MR: #143

[lu4]

For greater verbosity I've commented out the files excluded from build, plus I moved the port files and added build_version.cc in the end of rocksdb.gyp, also the tests currently failing, looking into what's wrong with them

[lu4]

@vweevers ok, as per https://github.com/facebook/rocksdb/search?q=ods&unscoped_q=ods it turns out that I've included some strange ODS piece of code that has some relation to RocksDB stats. It probably shouldn't be included into production build. Investigating for potential reasons...

[MeirionHughes]

Its a shame this stalled - I was just looking at rockdb and it turns out they enabled 1 primary write + secondary read process support, in a newer version.

Was there something drastic that killed this PR?

[vweevers]

1 primary write + secondary read process support

Sweet!

Was there something drastic that killed this PR?

We just need someone to pick up the work. We can start by reviewing this PR, merging it, then continuing with the tasks listed here: #141 (comment)

[vweevers]

I updated this to RocksDB 6.17.3, fixed the few remaining issues, and squashed the commits. Thank you @filoozom @lu4!

[MeirionHughes]

tested local clone with yarn link and yarn link rocksdb and it hasn't broken my work's app. my electron rebuild fails - but I suspect that is because I'm in a mono repo and it cannot find a relative path to node-gyp while using the link.

[vweevers]

FWIW, rebuilding for Electron should not be needed because the N-API builds are runtime agnostic (unless you have an old electron version.

[MeirionHughes]

This will need a major version bump and a warning - its fine with an old database, but that auto-updated database won't be compatible with an older version of rocksdb. maybe due to https://rocksdb.org/blog/2019/03/08/format-version-4.html ?

Corruption: Unknown Footer version. Maybe this file was created with newer version of RocksDB?
Error [ReadError]: Corruption: Unknown Footer version. Maybe this file was created with newer version of RocksDB?

[vweevers]

Yea, I picked this PR up because I was planning to release a major anyway for Level/community#98.

Will add a note to our UPGRADING.md with a link to their HISTORY.md. I think the breaking version is 6.9.0:

Updated default format_version in BlockBasedTableOptions from 2 to 4. SST files generated with the new default can be read by RocksDB versions 5.16 and newer, and use more efficient encoding of keys in index blocks.

[vweevers]

5.0.0