multitenant: introduce kv portion of the tenant capabilities framework #94643
Assignees
Labels
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-multitenant
Issues owned by the multi-tenant virtual team
Comments
arulajmani
added
C-enhancement
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 4, 2023
This commit adds the skeletal structure for a `TenantCapabilities` proto, which is intended to encapsulate capabilities for a specific tenant. Capabilities are intended to be stored in the `system.tenants` table, in its Info column. To that end, we modify `TenantInfo` to contain capabilities. However, actually populating this field through SQL is left to a future commit. Future commits will also add the infrastructure required to check a tenant's requests against its capabilities for "privileged" operations. For now, I've only accounted for the `CanAdminSplit` capability -- this will likely expand to a fuller set as we introduce other capabilities in the system. References cockroachdb#94643 Epic: CRDB-18503 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 4, 2023
This patch introduces a decoder, which takes raw KVs from the system.tenants table and decodes them into (TenantID, TenantCapability) pairs. This is currently unusued. In a future commit, we'll establish a rangefeed over the `system.tenants` table to maintain a cached view of the global capabilities state. This decoder will be used to parse the rangefeed output. Looking towards such a watcher component, the decoder is placed in the `tenantcapabilitieswatcher` package. References cockroachdb#94643 Epic: CRDB-18503 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 5, 2023
This commit adds the skeletal structure for a `TenantCapabilities` proto, which is intended to encapsulate capabilities for a specific tenant. Capabilities are intended to be stored in the `system.tenants` table, in its Info column. To that end, we modify `TenantInfo` to contain capabilities. However, actually populating this field through SQL is left to a future commit. Future commits will also add the infrastructure required to check a tenant's requests against its capabilities for "privileged" operations. For now, I've only accounted for the `CanAdminSplit` capability -- this will likely expand to a fuller set as we introduce other capabilities in the system. References cockroachdb#94643 Epic: CRDB-18503 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 5, 2023
94644: tenantcapabilities: introduce TenantCapabilities proto r=ecwall a=arulajmani This commit adds the skeletal structure for a `TenantCapabilities` proto, which is intended to encapsulate capabilities for a specific tenant. Capabilities are intended to be stored in the `system.tenants` table, in its Info column. To that end, we modify `TenantInfo` to contain capabilities. However, actually populating this field through SQL is left to a future commit. Future commits will also add the infrastructure required to check a tenant's requests against its capabilities for "privileged" operations. For now, I've only accounted for the `CanAdminSplit` capability -- this will likely expand to a fuller set as we introduce other capabilities in the system. References #94643 Epic: CRDB-18503 Release note: None 94754: split: init replica lb splitter with global rand r=erikgrinaker a=kvoli In #93838 we started initializing a new seeded rand for use in the load based splitter's reservoir sampling algorithm. Previously, the splitter was initialized using the global rand. `rand.Source` heap allocates on init approximately 4kb. When initialized per-replica, this is problematic as nodes scale to large replica counts. This patch replaces initializing a new random source per-replica with using the global rand instead. This removes the heap allocation. This is shown below running `kv/splits/nodes=3/quiesce=true` (not at identical replica counts in the test, the proportions are the important part). before:  after:  resolves: #94752 resolves: #94737 Release note: None Co-authored-by: Arul Ajmani <arulajmani@gmail.com> Co-authored-by: Austen McClernon <austen@cockroachlabs.com>
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 6, 2023
This patch introduces a decoder, which takes raw KVs from the system.tenants table and decodes them into (TenantID, TenantCapability) pairs. This is currently unusued. In a future commit, we'll establish a rangefeed over the `system.tenants` table to maintain a cached view of the global capabilities state. This decoder will be used to parse the rangefeed output. Looking towards such a watcher component, the decoder is placed in the `tenantcapabilitieswatcher` package. References cockroachdb#94643 Epic: CRDB-18503 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 7, 2023
94634: storage: configuration for writing sstables with value blocks r=jbowens,nicktrav a=sumeerbhola Cluster version V23_1EnablePebbleFormatSSTableValueBlocks gates writing of value blocks. Additionally, storage.value_blocks.enabled needs to be set to true (default is false). The lock table key space is excluded from value blocks. The tombstone bit is extracted as a ShortAttribute, to avoid fetching the value to determine whether it is a tombstone. Informs cockroachdb/pebble#1170 Epic: CRDB-20378 Release note(ops change): The cluster setting storage.value_blocks.enabled, when set to true, writes the values of older versions of the same key to separate value blocks in the same sstable. For workloads that create many versions, this can improve the performance of reads by increasing locality. It can also help scan performance with single versions due to an optimization that avoids a key comparison (useful if the key are not very short). The default value of this setting is currently false. 94722: tenantcapabilities: introduce a decoder for tenant capabilities r=knz a=arulajmani This patch introduces a decoder, which takes raw KVs from the system.tenants table and decodes them into (TenantID, TenantCapability) pairs. This is currently unusued. In a future commit, we'll establish a rangefeed over the `system.tenants` table to maintain a cached view of the global capabilities state. This decoder will be used to parse the rangefeed output. Looking towards such a watcher component, the decoder is placed in the `tenantcapabilitieswatcher` package. References #94643 Epic: [CRDB-18503](https://cockroachlabs.atlassian.net/browse/CRDB-18503) Release note: None Co-authored-by: sumeerbhola <sumeer@cockroachlabs.com> Co-authored-by: Arul Ajmani <arulajmani@gmail.com>
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 11, 2023
This patch introduces two new interfaces -- a CapabilitiesWatcher and Authorizer. They're not hooked up yet, but they'll work together to provide (in-memory) capability checks for tenant requests. The CapabiltiesWatcher establishes a rangefeed over `system.tenants` to incrementally (and transparently) maintain an in-memory view of the global tenant capabilities state. This in-memory state is stored in the Authorizer, to which incremental updates are directly applied. Operations that require capability checks (currently batch requests, as they may contain AdminSplit requests) are authorized by consulting the tenant capabilities state for the requesting tenant. Epic: CRDB-18503 References: cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 19, 2023
This patch introduces three new interfaces -- a Watcher, a Reader, and Authorizer. They're not hooked up yet, but once they are, they'll work together to provide (in-memory) capability checks for incoming tenant requests. The Watcher establishes a rangefeed over `system.tenants` to incrementally (and transparently) maintain an in-memory view of the global tenant capability state. Publicly, it exposes a `Reader` interface. The `Reader` provides access to the global tenant capability state. The `Watcher` and `Authorizer` communicate with each other using the `Reader` interface. The `Authorizer` consulsts the global tenant capability state to perform authorization checks for incoming requests issued by tenants. Part of the motivation to structure the code as such is to expand the set of inputs the `Authorizer` uses to authorize requests. One could imagine other dependencies being injected into the `Authorizer` in the future. Epic: CRDB-18503 References: cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Jan 19, 2023
This patch introduces three new interfaces -- a Watcher, a Reader, and Authorizer. They're not hooked up yet, but once they are, they'll work together to provide (in-memory) capability checks for incoming tenant requests. The Watcher establishes a rangefeed over `system.tenants` to incrementally (and transparently) maintain an in-memory view of the global tenant capability state. Publicly, it exposes a `Reader` interface. The `Reader` provides access to the global tenant capability state. The `Watcher` and `Authorizer` communicate with each other using the `Reader` interface. The `Authorizer` consults the global tenant capability state to perform authorization checks for incoming requests issued by tenants. Part of the motivation to structure the code as such is to expand the set of inputs the `Authorizer` uses to authorize requests. One could imagine other dependencies being injected into the `Authorizer` in the future. In this PR, we conservatively estimate a tenant capabilities proto to be 50 bytes in size. Given the `Watcher` keeps the entire capabilities state in memory, this means that for 10,0000 tenants, we would have a very sane memory footprint of 500KB. Epic: CRDB-18503 References: cockroachdb#94643 Release note: None
craig bot
pushed a commit
that referenced
this issue
Jan 19, 2023
93952: sql: measure CPU time spent during SQL execution r=DrewKimball a=DrewKimball This commit adds tracking for CPU time spent during SQL execution. The CPU time is tracked at the operator granularity when statistics collection is enabled, similar to execution time. For now, the CPU time is only surfaced in the output of `EXPLAIN ANALYZE` variants. A future PR will add support for logging this value in the statement statistics. Informs: #87213 Release note (sql change): CPU time spent during SQL execution is now visible in the output of queries run with `EXPLAIN ANALYZE`. This measure does not include CPU time spent while serving KV requests. This can be useful for diagnosing performance issues and optimizing SQL queries. 95040: tenantcapabilities: introduce a Watcher over system.tenants r=knz a=arulajmani This patch introduces three new interfaces -- a Watcher, a Reader, and Authorizer. They're not hooked up yet, but once they are, they'll work together to provide (in-memory) capability checks for incoming tenant requests. The Watcher establishes a rangefeed over `system.tenants` to incrementally (and transparently) maintain an in-memory view of the global tenant capability state. Publicly, it exposes a `Reader` interface. The `Reader` provides access to the global tenant capability state. The `Watcher` and `Authorizer` communicate with each other using the `Reader` interface. The `Authorizer` consulsts the global tenant capability state to perform authorization checks for incoming requests issued by tenants. Part of the motivation to structure the code as such is to expand the set of inputs the `Authorizer` uses to authorize requests. One could imagine other dependencies being injected into the `Authorizer` in the future. Epic: CRDB-18503 References: #94643 Release note: None 95361: sql/schemachanger: add compatibility with 22.2 rules r=fqazi a=fqazi This PR will make the following changes: 1. Refactor the existing rules package so that there is a common package, and multiple dep/op rules registries to allow us to support rules from older releases. This also includes splitting out helper functions to make this process easier 2. Import the 22.2 rules and have them adopt the same refactoring, so that a new rules/deps registry exists with them 3. Select the dep/op rules based on the active version of CRDB Co-authored-by: Drew Kimball <drewk@cockroachlabs.com> Co-authored-by: Arul Ajmani <arulajmani@gmail.com> Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 2, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 7, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 8, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. With the Authorizer wired up, we can now start using it to perform capability checks for incoming tenant requests. Currently, this is limited to batch requests. Note that the Authorizer is only responsible for performing capability checks -- other authorization checks, such as bounds checks, continue to happen outside of Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 8, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. With the Authorizer wired up, we can now start using it to perform capability checks for incoming tenant requests. Currently, this is limited to batch requests. Note that the Authorizer is only responsible for performing capability checks -- other authorization checks, such as bounds checks, continue to happen outside of Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
arulajmani
added a commit
to arulajmani/cockroach
that referenced
this issue
Feb 9, 2023
This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. With the Authorizer wired up, we can now start using it to perform capability checks for incoming tenant requests. Currently, this is limited to batch requests. Note that the Authorizer is only responsible for performing capability checks -- other authorization checks, such as bounds checks, continue to happen outside of Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs cockroachdb#94643 Release note: None
craig bot
pushed a commit
that referenced
this issue
Feb 9, 2023
96390: server: hook up tenant capabilities subsystem on startup r=arulajmani a=arulajmani This patch wires up the tenant capabilities subsystem during server startup. This includes both starting the subsystem and adding a handle to the Authorizer so that it can be used by GRPC interceptors to perform authorization checks. While working through this patch, I realized we can't instantiate an Authorizer with a handle to the tenant capability state (tenantcapabilities.Reader). This is because the afformentioned GRPC setup happens early on during the Server startup process, and at that point we do not have access to the dependencies required to setup the capabilities Watcher (which is what provides the Reader interface to the Authorizer). To break this dependency cycle, we end up with an approach to lazily bind the Reader to the Authorizer. This patch also adds a datadriven framework to test tenant capabilities end to end. The nice thing about it is it hides the asynchronous nature of capability checks from test writers. The hope is that we'll be able to extend this as we add more capabilities. Informs #94643 Release note: None 96484: cloud,ccl,*: break `sql` -> `cloud/externalconn/utils` dependency r=ZhouXing19 a=ZhouXing19 This commit is to break `cloud/externalconn/utils/`'s dependency on the `sql` package, as `sql` is just used to get the `sql.ExecCfg` for the validation function for external connection uri. But in fact, only a few fields of `sql.ExecCfg` are used; hence it makes more sense just to use a lighter `externalconn.ExternalConnEnv`. Release note: None Epic: None 96583: tree: make FmtPgwireText more accurate r=rafiss a=otan FmtPgwireText today currently does not actually match the pgwire text format for every type. This is now resolved. spinoff from #94408 Release note: None Epic: None 96820: cluster-ui: remove `.only` from test r=THardy98 a=THardy98 Epic: none Removes a `.only` from test file on `cluster-ui`. Release note: None Co-authored-by: Arul Ajmani <arulajmani@gmail.com> Co-authored-by: Jane Xing <zhouxing@uchicago.edu> Co-authored-by: Oliver Tan <otan@cockroachlabs.com> Co-authored-by: Thomas Hardy <thardy@cockroachlabs.com>
|
I'm going to close this out now that the KV portion is implemented and things are wired up end to end. There's a separate issue to track allowing secondary tenant's to introspect their capability state. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
This issue tracks the KV work required to enable tenant capabilities, as described in #85954.
Required
TenantCapabilitiesproto which describes the capabilities a specific tenant has tenantcapabilities: introduce TenantCapabilities proto #94644system.tenantstable. tenantcapabilities: introduce a decoder for tenant capabilities #94722Watcher, which establishes a range feed oversystem.tenantsand maintains an in-memory view of all tenant capabilities in the system. tenantcapabilities: introduce a Watcher over system.tenants #95040Authorizerinterface, which accepts tenant capabilities state as input, and makes authorizes incoming requests based on the tenant capabilities state. tenantcapabilities: introduce a Watcher over system.tenants #95040Watcherto server startup and use theAuthorizerin the RPC auth layer. server: hook up tenant capabilities subsystem on startup #96390Follow-on:
See https://cockroachlabs.atlassian.net/browse/CRDB-18503 for the JIRA epic describing this work.
Jira issue: CRDB-23060
Epic CRDB-18503
The text was updated successfully, but these errors were encountered: