FoundationDB physical backend #4900

jblache · 2018-07-11T00:48:31Z

This PR implements a FoundationDB physical storage backend for Vault, along with documentation, tests, and build system integration for the FoundationDB Go bindings.

The backend implements the transactional and HA interfaces; locks rely on correct time synchronization of the Vault nodes to operate properly.

Data model

The FoundationDB key is built by decorating the item path handed over to the backend by Vault core, to allow for an efficient list operation implementation using a single range read. The backend stores the data using 3 subspaces under a top-level directory:

a data subspace containing the actual data;
a shadow copy of the data, consisting of only the keys with nil values, used for the list operation;
a subspace dedicated to locks for the HA backend implementation.

The shadow copy is expected to eventually go away when a range read operation for keys only will be available. Bookkeeping for sub-levels employs FoundationDB’s atomic operations to reduce transaction conflicts; this is not true of the delete path at present, pending the availability of a decrease-delete-if-0 type of operation.

Testing

Caveat: there isn’t a readily available FoundationDB Docker image at this time. This is being worked on (apple/foundationdb#355), but until that lands, you will need to run a FoundationDB server locally to run the tests (https://apple.github.io/foundationdb/getting-started-linux.html).

$ export FOUNDATIONDB_CLUSTER_FILE=/etc/foundationdb/fdb.cluster
$ make test TEST=./physical/foundationdb

The test code includes the required setup code for using a Docker image, so a minor touch up should be all that’s needed once a Docker image is available for FoundationDB.

FoundationDB does not have a readily available Docker image yet; the test harness will need a touch-up when one lands.

jefferai · 2018-07-11T01:20:11Z

Please remove the changes to the Makefile. Cgo must be forced to zero and bootstrap should not bring in outside libraries.

jblache · 2018-07-11T18:46:10Z

Hey Jeff,

On CGO: FoundationDB language bindings are built on top of the C client library, so I'm afraid CGO is a requirement. Please let me know if I'm mistaken in this regard, and I'll be happy to make the necessary changes.

On the bootstrap changes, this could certainly be done differently. I am not sure the FoundationDB Go bindings can be treated similarly to other Go dependencies though, and any advice here would be appreciated. Due to the code that needs to be generated/built and the CGO nature of the bindings, I don't know that a different approach would necessarily result in a better situation overall; the fdb-go-install script is the supported way to get the job done at the moment. Suggestions on how to make this better are welcome!

jefferai · 2018-07-11T19:07:06Z

@jblache We don't currently offer dynamic builds of Vault. I don't know how much interest there is in FoundationDB as a storage backend but we could probably have a make target like dynamic-dev or dynamic-bin. Or -- and I'm not sure if this is actually doable -- it'd be cool if you could do something like make dynamic dev where the dynamic target modifies something in the running context such that the next target is then built with cgo enabled. That'd be way better if it's possible.

As for bootstrap, the only things that should be in bootstrap are items necessary for actually building Vault as a whole. If you wanted a separate Makefile in physical/foundationdb to allow for installation of tools that's fine, but we should not require all users to install FoundationDB bindings just to build Vault.

FDB_ENABLED being non-empty gates the foundationdb build tag and enabling CGO; it also gates installing the FDB Go bindings during make bootstrap.

jblache · 2018-07-12T00:14:03Z

@jefferai I believe this batch of changes should address your concerns. I've made the backend conditional; the Makefile will enable CGO for all build targets if the backend is enabled, as it is required. By default, the build is exactly as it is today. Attempts to use the backend will fail with a descriptive error message in a build that doesn't have it enabled.

I've removed the Go bindings installation from the bootstrap target and documented that step instead.

jefferai · 2018-07-12T14:54:04Z

.gitignore

@@ -87,3 +87,7 @@ ui/vault-ui-integration-server.pid
 # for building static assets
 node_modules
 package-lock.json
+
+# FoundationDB
+vendor/fdb-go-install.sh


These don't appear to be actual files.

jefferai · 2018-07-12T14:54:28Z

scripts/fdb-go-install.sh

@@ -0,0 +1,330 @@
+#!/bin/bash -eu


This should be in physical/foundationdb

jefferai · 2018-07-12T14:54:56Z

website/source/docs/configuration/storage/foundationdb.html.md

+- **High Availability** – the FoundationDB storage backend supports high
+  availability. The HA implementation relies on the clocks of the Vault
+  nodes inside the cluster being properly sychronized; clock skews are
+  susceptible to cause issues.


What kinds of issues, specifically?

jefferai · 2018-07-12T14:55:42Z

website/source/docs/configuration/storage/foundationdb.html.md

+storage "foundationdb" {
+  api_version  = 510
+  cluster_file = "/path/to/fdb.cluster"
+  path         = "vault-top-level-directory"


Please add an ha_enabled flag and gate HA support on it. See e.g. dynamo for guidance.

Update comment for FoundationDB Go bindings.

jblache · 2018-07-12T18:50:50Z

@jefferai All done!

jefferai · 2018-07-13T13:50:17Z

.gitignore

@@ -87,3 +87,6 @@ ui/vault-ui-integration-server.pid
 # for building static assets
 node_modules
 package-lock.json
+
+# FoundationDB Go bindings
+vendor/foundation


What is this here for? I assumed it was because go bindings were being put into vendor, but the script below doesn't mention this directory at all so I don't see any indication it's being added to the vendor directory (and I'd argue it's the wrong place for it since the vendor directory is automatically managed by govendor).

Drats, I could have caught that last time around. This is a leftover from my early attempts, and it can go.

jefferai · 2018-07-13T13:51:33Z

physical/foundationdb/fdb-go-install.sh

+#
+
+DESTDIR="${DESTDIR:-}"
+FDBVER="${FDBVER:-5.1.0}"


Should this be something like a latest tag? I just wonder whether this will end up just being out of date. Future PRs could bump it, so if it's a good idea to use "known good versions" that's a good argument, but just trying to understand the situation.

Yes, this is a known-good version kind of situation. The script is taken from the 5.1.x branch, so it defaults to that. FDBVER can be set when calling the script to get another version, and it should work just fine.

Having the script in the repo made obvious sense when it was called from bootstrap, but since that isn't the case anymore, I guess we can remove it and update the documentation to match. Let me know which way you are leaning.

I'm happy for the script to be in the repo, just want to understand these questions for the inevitable user queries :-)

This was a leftover from early work.

jefferai · 2018-07-16T14:18:01Z

Just a note: one of the other Vault team members looked at the code and said LGTM so I didn't examine the code too carefully, just have been looking at overall organization, which I'm happy with now. Going to merge, and I'm hoping you'll be pingable in case people find issues :-)

jblache · 2018-07-16T18:06:06Z

Yay! Thanks for merging :)

jefferai · 2018-07-16T20:52:16Z

Thanks for submitting :-D

1saifj · 2024-08-18T16:25:37Z

thanks @jblache for the support. thank you @jefferai for the follow up... good work.

jblache added 7 commits July 10, 2018 16:54

Integrate fdb-go-install.sh in make bootstrap

196353c

Add FoundationDB bindings to gitignore

f35f909

Enable CGO

00fb463

FoundationDB backend

b637a4d

Wire up FoundationDB backend

e21cc25

Implement backend tests for FoundationDB

dc39b68

FoundationDB does not have a readily available Docker image yet; the test harness will need a touch-up when one lands.

Document FoundationDB backend configuration

3afa7f5

jblache added 4 commits July 11, 2018 17:06

Enable FoundationDB backend by passing FDB_ENABLED=1 at build time

a83288f

FDB_ENABLED being non-empty gates the foundationdb build tag and enabling CGO; it also gates installing the FDB Go bindings during make bootstrap.

Stub out the FoundationDB backend if not enabled

e817118

Do not build FoundationDB backend test if the backend is not enabled

c1ae088

Document extra steps required to build the FoundationDB backend

dfd8c63

jefferai reviewed Jul 12, 2018

View reviewed changes

scripts/fdb-go-install.sh Outdated

@@ -0,0 +1,330 @@

#!/bin/bash -eu

Copy link

Member

jefferai Jul 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in physical/foundationdb

jefferai reviewed Jul 12, 2018

View reviewed changes

jblache added 4 commits July 12, 2018 10:52

Remove vendor/fdb-go-install.sh from .gitignore

5243a0f

Update comment for FoundationDB Go bindings.

Move fdb-go-install.sh to backend directory

87260f9

Update backend documentation

528c396

Add ha_enabled option

cd1926b

jefferai reviewed Jul 13, 2018

View reviewed changes

Remove gitignore entry for FDB bindings

cb7df7f

This was a leftover from early work.

jefferai merged commit e59d193 into hashicorp:master Jul 16, 2018

jefferai added this to the 0.10.4 milestone Jul 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FoundationDB physical backend #4900

FoundationDB physical backend #4900

jblache commented Jul 11, 2018

jefferai commented Jul 11, 2018

jblache commented Jul 11, 2018

jefferai commented Jul 11, 2018

jblache commented Jul 12, 2018

jefferai Jul 12, 2018

jefferai Jul 12, 2018

jefferai Jul 12, 2018

jefferai Jul 12, 2018

jblache commented Jul 12, 2018

jefferai Jul 13, 2018

jblache Jul 13, 2018

jefferai Jul 13, 2018

jblache Jul 13, 2018

jefferai Jul 16, 2018

jefferai commented Jul 16, 2018

jblache commented Jul 16, 2018

jefferai commented Jul 16, 2018

1saifj commented Aug 18, 2024

FoundationDB physical backend #4900

FoundationDB physical backend #4900

Conversation

jblache commented Jul 11, 2018

Data model

Testing

jefferai commented Jul 11, 2018

jblache commented Jul 11, 2018

jefferai commented Jul 11, 2018

jblache commented Jul 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jblache commented Jul 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jefferai commented Jul 16, 2018

jblache commented Jul 16, 2018

jefferai commented Jul 16, 2018

1saifj commented Aug 18, 2024