Cache ZedTokens for resources using NATS #209
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using full consistency when doing permissions checks is slow. In general, this is addressed by using ZedTokens to indicate minimum bounds on freshness when looking up cached data. Something has to keep track of those tokens, either on the client side or server-side. This commit introduces worker caching of ZedTokens for resources on updates to relationships and updates the query engine to use those tokens when performing permissions checks. When a worker updates a relationship, it persists the ZedToken for all resources directly affected by that update to a NATS KV bucket. NATS KV writes are immediately consistent, so the new ZedToken for that resource is available to all consumers, including permissions-api API frontends. When the query engine performs a permissions check, it checks to see if a ZedToken is available for the resource. If a ZedToken was found, that ZedToken is used along with the at_least_as_fresh SpiceDB API consistency strategy. If not, or if there was an error accessing NATS, the query engine falls back to the minimize_latency API consistency strategy. If the NATS KV bucket is configured with a TTL at least as high as the quantization interval for SpiceDB, this ensures that by the time the ZedToken is evicted from the cache, all SpiceDB frontends will be updated with data at least as fresh as the last relationship update for a resource. Clients that wish to force an update for a resource (e.g., making role changes immediately available to tenant users) can thus issue a relationship update to permissions-api and get the latest data for that resource. This commit assumes that the KV bucket used already exists; permissions-api will not attempt to create it. This is because the intention is that the KV bucket has a TTL set to something close to the SpiceDB quantization interval, which permissions-api is not necessarily aware of. Signed-off-by: John Schaeffer <jschaeffer@equinix.com>
This commit adds the necessary configs to the Helm chart to support populating a ZedToken cache for permissions-api. Signed-off-by: John Schaeffer <jschaeffer@equinix.com>
fishnix
previously approved these changes
Jan 16, 2024
| } | ||
|
|
||
| // If an error happened when creating or updating the token, record it. | ||
| if err != nil { |
There was a problem hiding this comment.
ill be interested to see how often we hit this condition - i expect not a lot since its for an individual relationship
There was a problem hiding this comment.
Probably not often, but I can think of a few cases:
- NATS is unavailable
- Something created a bucket entry before this function did
- Something updated a bucket entry
| // updateRelationshipZedTokens updates the NATS KV bucket for ZedTokens, setting the given ZedToken | ||
| // as the latest point in time snapshot for every resource in the given list of relationships. | ||
| func (e *engine) updateRelationshipZedTokens(ctx context.Context, rels []types.Relationship, zedToken string) { | ||
| resourceIDMap := map[string]struct{}{} |
There was a problem hiding this comment.
why this over just a slice here since you're just looping over the keys anyways?
There was a problem hiding this comment.
Relationships don't necessarily need to refer to unique resources (i.e., multiple relationships could be persisted that affect the same tenant). This just ensures that we aren't sending redundant ZedToken updates.
This commit adds NATS creds to the server deployment in the Helm chart. Signed-off-by: John Schaeffer <jschaeffer@equinix.com>
mikemrm
previously approved these changes
Jan 16, 2024
This commit adds tests for determineConsistency. Signed-off-by: John Schaeffer <jschaeffer@equinix.com>
jnschaeffer
force-pushed
the
use-zedtokens
branch
from
a65d212
to
30e1adc
Compare
mikemrm
approved these changes
Jan 16, 2024
fishnix
approved these changes
Jan 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using full consistency when doing permissions checks is slow. In general, this is addressed by using ZedTokens to indicate minimum bounds on freshness when looking up cached data. Something has to keep track of those tokens, either on the client side or server-side.
This PR introduces worker caching of ZedTokens for resources on updates to relationships and updates the query engine to use those tokens when performing permissions checks. When a worker updates a relationship, it persists the ZedToken for all resources directly affected by that update to a NATS KV bucket. NATS KV writes are immediately consistent, so the new ZedToken for that resource is available to all consumers, including permissions-api API frontends. When the query engine performs a permissions check, it checks to see if a ZedToken is available for the resource.
If a ZedToken was found, that ZedToken is used along with the
at_least_as_freshSpiceDB API consistency strategy. If not, or if there was an error accessing NATS, the query engine falls back to theminimize_latencyAPI consistency strategy. If the NATS KV bucket is configured with a TTL at least as high as the quantization interval for SpiceDB, this ensures that by the time the ZedToken is evicted from the cache, all SpiceDB frontends will be updated with data at least as fresh as the last relationship update for a resource. Clients that wish to force an update for a resource (e.g., making role changes immediately available to tenant users) can thus issue a relationship update to permissions-api and get the latest data for that resource.This PR assumes that the KV bucket used already exists; permissions-api will not attempt to create it. This is because the intention is that the KV bucket has a TTL set to something close to the SpiceDB quantization interval, which permissions-api is not necessarily aware of.