Multi-Release-Epic: Sharding

[sttts]

Demo Objective

• kcp admin should be able to add shards for added capacity, each with its own etcd server.
• shards should keep working (to be defined what this means exactly) when other shards die or are unavailable.
• Add shard proxying and add (1-shard) sharded CI job with front-proxy #1203 the front-proxy knows how to forward workspace requests to shards
• add (hard-coded) replication of certain object between shards:
  • defining ClusterWorkspace X in the parent workspace parent(X) on the shard the workspace X lives on.
  • APIExport that are bound by at least one non-local binding

Action Items

• Add shard proxying and add (1-shard) sharded CI job with front-proxy #1203 forward requests to the right shard from the front-proxy
• Add shard proxying and add (1-shard) sharded CI job with front-proxy #1203 sharded-test-server binary
• Add shard proxying and add (1-shard) sharded CI job with front-proxy #1203 setup single-shard + front-proxy CI job
• @p0lyn0mial verify where we are with watch cache and performant label selector

========= WIP, but won't merge by v0.6 =========

• @p0lyn0mial hardcode second informer with non-local data.
• @p0lyn0mial prototype and experiment with informer merging, and how to enable existing controller to work with second informer
• get 2-shard e2e-sharded into place in CI, with limited test coverage
• get ClusterWorkspace, APIExports, APIBindings, auth to work
  • @p0lyn0mial APIBindings on shards can bind APIExport on other shard
  • @p0lyn0mial ClusterWorkspaces can be scheduled to second shard

========= up to here for v0.7 ==========

add identities configmap in system:shard @p0lyn0mial

• adds ApiExportIdentityProviderController #1725: implement a controller that maintains a local cm with export identities
• the identity provider gets data from a local cm too #1727 update the identity provider to read data from a local config map
• wires the APIIdentityCacheController #1781 wire the APIIdentityCacheController to the kcp server

implement replication controller / cache server

• @p0lyn0mial create initial cmd/cache-server and pkg/cache/server/{config.go,server.go} and pkg/cache/server/options plumbing (cache-server scaffolding #1790)
  • follow options/config/server pattern tightly
  • options should be compatible and easily composable with those in pkg/server/options
• @p0lyn0mial instantiate apiextensions-apiserver and wire CRD lister similar to today's pkg/server/apiextensions.go with at least APIExports and APIResourceSchemas
  • use empty schemas, as dumb as possible: APIExport/APIResourceSchema CRDs can be static as in

    kcp/pkg/virtual/apiexport/schemas/apis.go

    Line 36 in 8d2ed6a

    if err := configcrds.Unmarshal(fmt.Sprintf("apis.kcp.dev_%s.yaml", resource), &crd); err != nil {

  • no admission, no validation, no conversion, no pruning, no SSA (if possible), for now no authorization
  • PRs: cache-server: starts empty apiextentions-server #1811, cache-server: expose apiresourceschemas and apiexports #1815, UPSTREAM: <carry>: collapse with extensions-apiserver udpates kubernetes#93

========= up to here for v0.8 ==========

implement replication controller / cache server (continued)

• @p0lyn0mial wire custom etcd prefix for "cached CRDs" returned by the CRD lister: /registry/group/cache:resource:identity:shard/… (cache-server: adds WithShardScope HTTP filter #1841)
• @p0lyn0mial assign shard names to objects (UPSTREAM: <carry>: assign shard names to objects kubernetes#100, cache-server: adds ShardNameFromObjectRoundTripper #1926, 🌱 cache: makes all internal clients shard and cluster aware #1928)
• @p0lyn0mial serve /services/cache/shards/*/clusters/* (🌱 cache: expose the cache-server under "/services/cache" path #1961)
• @p0lyn0mial serve /services/cache/shards/{shard-name}/clusters/{cluster-name} (🌱 cache: expose the cache-server under "/services/cache" path #1961)
• @p0lyn0mial instantiate cache-server inside of kcp binary. (🌱 cache: small refactor to make wiring into kcp easier #1949, 🌱 cache: moves common HTTP handlers to a shared pkg #1947, 🌱 Wire the cache server into the kcp server #1954)
• @p0lyn0mial wire --cache-url into kcp, defaulting to localhost. (🌱 kcp: add flags related to the cache server #1970)
• @p0lyn0mial create shard controller pushing (labeled to be published) apiexports, apiresourceschemas, ClusterWorkspaces to cache
• create cache controller creating "cached CRDs" or create "cached CRD" on the fly in the CRD lister for all cached APIExports
• serve internal APIs through schema-less CRDs, wired statically
• wire secondary informers to cache instead of the other shard
• enable the watch-cache

authorization

• shards should only be able to write their shard partition of the cache

subscription model

• implement subscription model @sttts

type CachingInformer interface {
    Informer

    KeepWatching(clusterName logicalcluster.Name, labelSelector, ttl time.Time)
}

data removal

unused resources could be automatically removed

shard management, a few loose ideas:

what will decide when a new shard needs to be created ?
which cluster it gets deployed to ?
making other shards aware of the new shard ?

(in parallel, best effort) add some other controller(s), e.g. ClusterWorkspaces

[ ] wire informer both ways @p0lyn0mial

• this includes a controller to cope with not-yet-synced informers

[ ] start the second informer (aka. TemporaryRootShardKcpSharedInformerFactory) in a dedicated post-start-hook @p0lyn0mial

Testing

• get all e2e green with two shards

Backlog:

• ThingReplicationClaim
• prototype CRDB service
• field selectors
• authz of replication
• controller programming model, related to Define Proper Controller Author Behavior re: Connection URLs #1339
• Define Proper Controller Author Behavior re: Connection URLs #1339
• Build A MVP Persistent Caching Proxy #342
• Prototype Workspace Life Cycle on Shards #354
• Clients of the Caching Server Need to be Cross-workspace list/watch resilient #348

Stories

• CRDB implements ThingReplicationClaim
• etcd implements ThingReplicationClaim

Context

• ThingReplicationClaim https://docs.google.com/document/d/10-HCQPwLSXfs3-6I-_dKNXdenTeJ3rrx8ZTbX00X-Nc/edit#

[stevekuznetsov]

Superseded by #1999