diff --git a/geps/gep-713.md b/geps/gep-713.md index 520c18130b..cd1bf9676a 100644 --- a/geps/gep-713.md +++ b/geps/gep-713.md @@ -1,4 +1,4 @@ -# GEP-713: Metaresources and Policy Attachment_metaresource_ +# GEP-713: Metaresources and Policy Attachment * Issue: [#713](https://github.com/kubernetes-sigs/gateway-api/issues/713) * Status: Experimental @@ -248,7 +248,25 @@ in this document. Examples to further illustrate these rules are given below. -## API +## Naming Policy objects + +The preceding rules discuss how Policy objects should _behave_, but this section +describes how Policy objects should be _named_. + +Policy objects should be clearly named so as to indicate that they are Policy +metaresources. + +The simplest way to do that is to ensure that the type's name contains the `Policy` +string. + +Implementations _should_ use `Policy` as the last part of the names of object types +that use this pattern. + +If an implementation does not, then they _must_ clearly document what objects +are Policy metaresources in their documentation. Again, this is _not recommended_ +without a _very_ good reason. + +## Policy Attachment examples and behavior This approach is building on concepts from all of the alternatives discussed below. This is very similar to the (now removed) BackendPolicy resource in the API, @@ -560,52 +578,570 @@ ties: For a better user experience, a validating webhook can be implemented to prevent these kinds of conflicts all together. -### Kubectl Plugin +## Status and the Discoverability Problem + +So far, this document has talked about what Policy Attachment is, different types +of attachment, and how those attachments work. + +Probably the biggest impediment to this GEP moving forward is the discoverability +problem; that is, it’s critical that an object owner be able to know what policy +is affecting their object, and ideally its contents. + +To understand this a bit better, let’s consider this parable, with thanks to Flynn: + +### The Parable + +It's a sunny Wednesday afternoon, and the lead microservices developer for +Evil Genius Cupcakes is windsurfing. Work has been eating Ana alive for the +past two and a half weeks, but after successfully deploying version 3.6.0 of +the `baker` service this morning, she's escaped early to try to unwind a bit. + +Her shoulders are just starting to unknot when her phone pings with a text +from Charlie, down in the NOC. Waterproof phones are a blessing, but also a +curse. + +**Charlie**: _Hey Ana. Things are still running, more or less, but latencies +on everything in the `baker` namespace are crazy high after your last rollout, +and `baker` itself has a weirdly high load. Sorry to interrupt you on the lake +but can you take a look? Thanks!!_ + +Ana stares at the phone for a long moment, heart sinking, then sighs and +turns back to shore. + +What she finds when dries off and grabs her laptop is strange. `baker` does +seem to be taking much more load than its clients are sending, and its clients +report much higher latencies than they’d expect. She doublechecks the +Deployment, the Service, and all the HTTPRoutes around `baker`; everything +looks good. `baker`’s logs show her mostly failed requests... with a lot of +duplicates? Ana checks her HTTPRoute again, though she's pretty sure you +can't configure retries there, and finds nothing. But it definitely looks like +clients are retrying when they shouldn’t be. + +She pings Charlie. + +**Ana**: _Hey Charlie. Something weird is up, looks like requests to `baker` +are failing but getting retried??_ + +A minute later they answer. + +**Charlie**: 🤷 _Did you configure retries?_ + +**Ana**: _Dude. I don’t even know how to._ 😂 + +**Charlie**: _You just attach a RetryPolicy to your HTTPRoute._ + +**Ana**: _Nope. Definitely didn’t do that._ + +She types `kubectl get retrypolicy -n baker` and gets a permission error. + +**Ana**: _Huh, I actually don’t have permissions for RetryPolicy._ 🤔 + +**Charlie**: 🤷 _Feels like you should but OK, guess that can’t be it._ + +Minutes pass while both look at logs. + +**Charlie**: _I’m an idiot. There’s a RetryPolicy for the whole namespace – +sorry, too many policies in the dashboard and I missed it. Deleting that since +you don’t want retries._ + +**Ana**: _Are you sure that’s a good–_ + +Ana’s phone shrills while she’s typing, and she drops it. When she picks it +up again she sees a stack of alerts. She goes pale as she quickly flips +through them: there’s one for every single service in the `baker` namespace. + +**Ana**: _PUT IT BACK!!_ + +**Charlie**: _Just did. Be glad you couldn't hear all the alarms here._ 😕 + +**Ana**: _What the hell just happened??_ + +**Charlie**: _At a guess, all the workloads in the `baker` namespace actually +fail a lot, but they seem OK because there are retries across the whole +namespace?_ 🤔 + +Ana's blood runs cold. + +**Charlie**: _Yeah. Looking a little closer, I think your `baker` rollout this +morning would have failed without those retries._ 😕 + +There is a pause while Ana's mind races through increasingly unpleasant +possibilities. + +**Ana**: _I don't even know where to start here. How long did that +RetryPolicy go in? Is it the only thing like it?_ + +**Charlie**: _Didn’t look closely before deleting it, but I think it said a few +months ago. And there are lots of different kinds of policy and lots of +individual policies, hang on a minute..._ + +**Charlie**: _Looks like about 47 for your chunk of the world, a couple hundred +system-wide._ + +**Ana**: 😱 _Can you tell me what they’re doing for each of our services? I +can’t even_ look _at these things._ 😕 + +**Charlie**: _That's gonna take awhile. Our tooling to show us which policies +bind to a given workload doesn't go the other direction._ + +**Ana**: _...wait. You have to_ build tools _to know if retries are turned on??_ + +Pause. + +**Charlie**: _Policy attachment is more complex than we’d like, yeah._ 😐 +_Look, how ‘bout roll back your `baker` change for now? We can get together in +the morning and start sorting this out._ + +Ana shakes her head and rolls back her edits to the `baker` Deployment, then +sits looking out over the lake as the deployment progresses. + +**Ana**: _Done. Are things happier now?_ + +**Charlie**: _Looks like, thanks. Reckon you can get back to your sailboard._ 🙂 + +Ana sighs. + +**Ana**: _Wish I could. Wind’s died down, though, and it'll be dark soon. +Just gonna head home._ + +**Charlie**: _Ouch. Sorry to hear that._ 😐 + +One more look out at the lake. + +**Ana**: _Thanks for the help. Wish we’d found better answers._ 😢 + +### The Problem, restated +What this parable makes clear is that, in the absence of information about what +Policy is affecting an object, it’s very easy to make poor decisions. + +It’s critical that this proposal solve the problem of showing up to three things, +listed in increasing order of desirability: + +- _That_ some Policy is affecting a particular object +- _Which_ Policy is (or Policies are) affecting a particular object +- _What_ settings in the Policy are affecting the object. + +In the parable, if Ana and Charlie had known that there were Policies affecting +the relevant object, then they could have gone looking for the relevant Policies +and things would have played out differently. If they knew which Policies, they +would need to look less hard, and if they knew what the settings being applied +were, then the parable would have been able to be very short indeed. + +(There’s also another use case to consider, in that Charlie should have been able +to see that the Policy on the namespace was in use in many places before deleting +it.) + +To put this another way, Policy Attachment is effectively adding a fourth Persona, +the Policy Admin, to Gateway API’s persona list, and without a solution to the +discoverability problem, their actions are largely invisible to the Application +Developer. Not only that, but their concerns cut across the previously established +levels. + +![Gateway API diagram with Policy Admin](images/713-the-diagram-with-policy-admin.png) + + +From the Policy Admin’s point of view, they need to know across their whole remit +(which conceivably could be the whole cluster): + +- _What_ Policy has been created +- _Where_ it’s applied +- _What_ the resultant policy is saying + +Which again, come down to discoverability, and can probably be addressed in similar +ways at an API level to the Application Developer's concerns. + +An important note here is that a key piece of information for Policy Admins and +Cluster Operators is “How many things does this Policy affect?”. In the parable, +this would have enabled Charlie to know that deleting the Namespace Policy would +affect many other people than just Ana. + +### Problems we need to solve + +Before we can get into solutions, we need to discuss the problems that solutions +may need to solve, so that we have some criteria for evaluating those solutions. + +#### User discoverability + +Let's go through the various users of Gateway API and what they need to know about +Policy Attachment. + +In all of these cases, we should aim to keep the troubleshooting distance low; +that is, that there should be a minimum of hops required between objects from the +one owned by the user to the one responsible for a setting. + +Another way to think of the troubleshooting distance in this context is "How many +`kubectl` commands would the user need to do to understand that a Policy is relevant, +which Policy is relevant, and what configuration the full set of Policy is setting?" + +##### Application Developer Discoverability + +How does Ana, or any Application Devloper who owns one or more Route objects know +that their object is affected by Policy, which Policy is affecting it, and what +the content of the Policy is? + +The best outcome is that Ana needs to look only at a specific route to know what +Policy settings are being applied to that Route, and where they come from. +However, some of the other problems below make it very difficult to achieve this. + +##### Policy Admin Discoverability + +How does the Policy Admin know what Policy is applied where, and what the content +of that Policy is? +How do they validate that Policy is being used in ways acceptable to their organization? +For any given Policy object, how do they know how many places it's being used? + +##### Cluster Admin Discoverability + +The Cluster Admin has similar concerns to the Policy Admin, but with a focus on +being able to determine what's relevant when something is broken. + +How does the Cluster Admin know what Policy is applied where, and what the content +of that Policy is? + +For any given Policy object, how do they know how many places it's being used? + +#### Evaluating and Displaying Resultant Policy + +For any given Policy type, whether Direct Attached or Inherited, implementations +will need to be able to _calculate_ the resultant set of Policy to be able to +apply that Policy to the correct parts of their data plane configuration. +However, _displaying_ that resultant set of Policy in a way that is straightforward +for the various personas to consume is much harder. + +The easiest possible option for Application Developers would be for the +implementation to make the full resultant set of Policy available in the status +of objects that the Policy affects. However, this runs into a few problems: + +- The status needs to be namespaced by the implementation +- The status could get large if there are a lot of Policy objects affecting an + object +- Building a common data representation pattern that can fit into a single common + schema is not straightforward. +- Updating one Policy object could cause many affected objects to need to be + updated themselves. This sort of fan-out problem can be very bad for apiserver + load, particularly if Policy chnges rapidly, there are a lot of objects, or both. + +##### Status needs to be namespaced by implementation + +Because an object can be affected by multiple implementations at once, any status +we add must be namespaced by the implementation. + +In Route Parent status, we've used the parentRef plus the controller name for this. + +For Policy, we can do something similar and namespace by the reference to the +implementation's controller name. + +We can't easily namespace by the originating Policy because the source could be +more than one Policy object. + +##### Creating common data representation patterns + +The problem here is that we need to have a _common_ pattern for including the +details of an _arbitrarily defined_ object, that needs to be included in the base +API. + +So we can't use structured data, because we have no way of knowing what the +structure will be beforehand. + +This suggests that we need to use unstructured data for representing the main +body of an arbitrary Policy object. + +Practically, this will need to be a string representation of the YAML form of the +body of the Policy object (absent the metadata part of every Kubernetes object). + +Policy Attachment does not mandate anything about the design of the object's top +level except that it must be a Kubernetes object, so the only thing we can rely +on is the presence of the Kubernetes metadata elements: `apiVersion`, `kind`, +and `metadata`. + +A string representation of the rest of the file is the best we can do here. + +##### Fanout status update problems + +The fanout problem is that, when an update takes place in a single object (a +Policy, or an object with a Policy attached), an implementation may need to +update _many_ objects if it needs to place details of what Policy applies, or +what the resultant set of policy is on _every_ object. + +Historically, this is a risky strategy and needs to be carefully applied, as +it's an excellent way to create apiserver load problems, which can produce a large +range of bad effects for cluster stability. + +This does not mean that we can't do anything at all that affects multiple objects, +but that we need to carefully consider what information is stored in status so +that _every_ Policy update does not require a status update. + +#### Solution summary + +Because Policy Attachment is a pattern for APIs, not an API, and needs to address +all the problems above, the strategy this GEP proposes is to define a range of +options for increasing the discoverabilty of Policy resources, and provide +guidelines for when they should be used. + +It's likely that at some stage, the Gateway API CRDs will include some Policy +resources, and these will be designed with all these discoverabiity solutions +in mind. + + +### Solution cookbook + +This section contains some required patterns for Policy objects and some +suggestions. Each will be marked as MUST, SHOULD, or MAY, using the standard +meanings of those terms. + +Additionally, the status of each solution is noted at the beginning of the section. + +#### Standard label on CRD objects + +Status: Required + +Each CRD that defines a Policy object MUST include a label that specifies that +it is a Policy object, and that label MUST specify the _type_ of Policy attachment +in use. + +The label is `gateway.networking.k8s.io/policy: inherited|direct`. + +This solution is intended to allow both users and tooling to identify which CRDs +in the cluster should be treated as Policy objects, and so is intended to help +with discoverability generally. It will also be used by the forthcoming `kubectl` +plugin. + +##### Design considerations + +This is already part of the API pattern, but is being lifted to more prominience +here. + +#### Standard status Condition + +Support: Provisional + +This solution is IN PROGRESS and so is not binding yet. + +This solution requires definition in a GEP of its own to become binding. + +**The description included here is intended to illustrate the sort of solution +that an eventual GEP will need to provide, _not to be a binding design.** + +Implementations that use Policy objects MUST put a Condition into `status.Conditions` +of any objects affected by a Policy. + +That Condition must have a `type` ending in `PolicyAffected` (like +`gateway.networking.k8s.io/PolicyAffected`), +and have the optional `observedGeneration` field kept up to date when the `spec` +of the Policy-attached object changes. + +Implementations _should_ use their own unique domain prefix for this Condition +`type` - it is recommended that implementations use the same domain as in the +`controllerName` field on GatewayClass (or some other implementation-unique +domain for implementations that do not use GatewayClass).) + +For objects that do _not_ have a `status.Conditions` field available (`Secret` +is a good example), that object MUST instead have an annotation of +`gateway.networking.k8s.io/PolicyAffected: true` (or with an +implementation-specifc domain prefix) added instead. + + +##### Design Considerations +The intent here is to add at least a breadcrumb that leads object owners to have +some way to know that their object is being affected by another object, while +minimizing the number of updates necessary. + +Minimizing the object updates is done by only having an update be necesary when +the affected object starts or stops being affected by a Policy, rather than if +the Policy itself has been updated. + +There is already a similar Condition to be placed on _Policy_ objects, rather +than on the _targeted_ objects, so this solution is also being included in the +Condiions section below. + +#### GatewayClass status Extension Types listing + +Support: Provisional + +This solution is IN PROGRESS, and so is not binding yet. + +Each impelementation MUST list all relevant CRDs in its GatewayClass status (like +Policy, and other extension types, like paramsRef targets, filters, and so on). + +This is going to be tracked in its own GEP, https://github.com/kubernetes-sigs/gateway-api/discussions/2118 +is the initial discussion. This document will be updated with the details once +that GEP is opened. + +##### Design Considerations + +This solution: + +- is low cost in terms of apiserver updates (because it's only on the GatewayClass, + and only on implementation startup) +- provides a standard place for all users to look for relevant objects +- ties in to the Conformance Profiles design and other efforts about GatewayClass + status + +#### Standard status stanza + +Support: Provisional + +This solution is IN PROGRESS and so is not binding yet. + +This solution requires definition in a GEP of its own to become binding. + +**The description included here is intended to illustrate the sort of solution +that an eventual GEP will need to provide, _not to be a binding design. THIS IS +AN EXPERIMENTAL SOLUTION DO NOT USE THIS YET.** + +An implementation SHOULD include the name, namespace, apiGroup and Kind of Policies +affecting an object in the new `effectivePolicy` status stanza on Gateway API +objects. + +This stanza looks like this: +```yaml +kind: Gateway +... +status: + effectivePolicy: + - name: some-policy + namespace: some-namespace + apiGroup: implementation.io + kind: AwesomePolicy + ... +``` + +##### Design Considerations + +This solution is designed to limit the number of status updates required by an +implementation to when a Policy starts or stops being relevant for an object, +rather than if that Policy's settings are updated. + +It helps a lot with discoverability, but comes at the cost of a reasonably high +fanout cost. Implemenations using this solution should ensure that status updates +are deduplicated and only sent to the apiserver when absolutely necessary. + +Ideally, these status updates SHOULD be in a separate, lower-priority queue than +other status updates or similar solution. + +#### PolicyBinding resource + +Support: Provisional + +This solution is IN PROGRESS and so is not binding yet. + +This solution requires definition in a GEP of its own to become binding. + +**The description included here is intended to illustrate the sort of solution +that the eventual GEP will need to provide, _not to be a binding design. THIS IS +AN EXPERIMENTAL SOLUTION DO NOT USE THIS YET.** + +Implemenations SHOULD create an instance of a new `gateway.networking.k8s.io/EffectivePolicy` +object when one or more Policy objects become relevant to the target object. + +The `EffectivePolicy` object MUST be in the same namespace as the object targeted +by the Policy, and must have the _same name_ as the object targeted like the Policy. +This is intended to mirror the Services/Endpoints naming convention, to allow for +ease of discovery. + +The `EffectivePolicy` object MUST set the following information: + +- The name, namespace, apiGroup and Kind of Policy objects affecting the targeted + object. +- The full resultant set of Policy affecting the targeted object. + +The above details MUST be namespaced using the `controllerName` of the implementation +(could also be by GatewayClass), similar to Route status being namespaced by +`parentRef`. + +An example `EffectivePolicy` object is included here - this may be superseded by +a later GEP and should be updated or removed in that case. Note that it does +_not_ contain a `spec` and a `status` stanza - by definition this object _only_ +contains `status` information. + +```yaml +kind: EffectivePolicy +apiVersion: gateway.networkking.k8s.io/v1alpha2 +metadata: + name: targeted-object + namespace: targeted-object-namespace +policies: +- controllerName: implementation.io/ControllerName + objects: + - name: some-policy + namespace: some-namespace + apiGroup: implementation.io + kind: AwesomePolicy + resultantPolicy: + awesomePolicy: + configitem1: + defaults: + foo: 1 + overrides: + bar: important-setting + +``` + +Note here that the `resultantPolicy` setting is defined using the same mechanisms +as an `unstructured.Unstructured` object in the Kubernetes Go libraries - it's +effectively a `map[string]struct{}` that is stored as a `map[string]string` - +which allows an arbitrary object to be specified there. + +Users or tools reading the config underneath `resultantPolicy` SHOULD display +it in its encoded form, and not try to deserialize it in any way. + +The rendered YAML MUST be usable as the `spec` for the type given. + +##### Design considerations + +This will provide _full_ visibility to end users of the _actual settings_ being +applied to their object, which is a big discoverability win. + +However, it relies on the establishment and communication of a convention ("An +EffectivePolicy is right next to your affected object"), that may not be desirable. + +Thus its status as EXPERIMENTAL DO NOT USE YET. + +#### Validating Admission Controller to inform users about relevant Policy + +Implementations MAY supply a Validating Admission Webhook that will return a +WARNING message when an applied object is affected by some Policy, which may be +an inherited or indirect one. + +The warning message MAY include the name, namespace, apiGroup and Kind of relevant +Policy objects. + +##### Design Considerations + +Pro: + +- This gives object owners a very clear signal that something some Policy is + going to affect their object, at apply time, which helps a lot with discoverability. + +Cons: + +- Implementations would have to have a webhook, which is another thing to run. +- The webhook will need to have the same data model that the implementation uses, + and keep track of which GatewayClasses, Gateways, Routes, and Policies are + relevant. Experience suggests this will not be a trivial engineering exercise,and will add a lot of implementation complexity. + +#### `kubectl` plugin or command-line tool To help improve UX and standardization, a kubectl plugin will be developed that will be capable of describing the computed sum of policy that applies to a given resource, including policies applied to parent resources. Each Policy CRD that wants to be supported by this plugin will need to follow -the API structure defined above and add a `gateway.networking.k8s.io/policy: -true` label to the CRD. - -### Status - -In the current iteration of this GEP, metaresources and Policy objects don't -have any standard way to record what they're attaching to, or applying settings -to in the case of Policy Attachment. There are some recommended Condition types -defined below, but further work on the status design is required to ensure that -some problems are resolved: - -* When multiple controllers are implementing the same Route and recognize a - policy, it must be possible to determine which controller was - responsible for adding that policy reference to status. Adding Conditions to - status on the Policy instead can be helpful here, but we're still lacking a way - for the Route or Gateway owner to find _all_ the Policies that are influencing - their object. -* For this to be somewhat scalable, we must limit the number of status updates - that can result from a metaresource update. -* Since we only control some of the resources a policy might be attached to, - adding policies to status would only be possible on the policy objects themselves - or on Gateway API resources, not Services or other kinds of backends. - -Previous experience in the Kubernetes API has made it clear that having a single -object that can cause status updates to occur across many other objects can have -a big performance impact, so the status design must be very carefully done to -avoid these kind of fanout problems. - -However, the whole purpose of having a standardized Policy API structure and -patterns is intended to make this problem solvable both for human users and with -tooling. - -This is currently a _very_ open question. A discussion is ongoing at -[#1531](https://github.com/kubernetes-sigs/gateway-api/discussions/1531), and this -GEP will be updated with any outcomes. +the API structure defined above and add a `gateway.networking.k8s.io/policy: true` +label to the CRD. ### Conditions -Controllers using the Gateway API policy attachment model SHOULD populate the -following condition and reasons on policy resources to provide a consistent -experience across implementations. + +Implementations using Policy objects MUST include a `spec` and `status` stanza, and the `status` stanza MUST contain a `conditions` stanza, using the standard Condition format. + +Policy authors should consider namespacing the `conditions` stanza with a +`controllerName`, as in Route status, if more than one implementation will be +reconciling the Policy type. + +#### On `Policy` objects + +Controllers using the Gateway API policy attachment model MUST populate the +`Accepted` condition and reasons as defined below on policy resources to provide +a consistent experience across implementations. ```go // PolicyConditionType is a type of condition for a policy. @@ -649,6 +1185,33 @@ const ( ) ``` +#### On targeted resources + +(copied from [Standard Status Condition][#standard-status-condition]) + +This solution requires definition in a GEP of its own to become binding. + +**The description included here is intended to illustrate the sort of solution +that an eventual GEP will need to provide, _not to be a binding design.** + +Implementations that use Policy objects MUST put a Condition into `status.Conditions` +of any objects affected by a Policy. + +That Condition must have a `type` ending in `PolicyAffected` (like +`gateway.networking.k8s.io/PolicyAffected`), +and have the optional `observedGeneration` field kept up to date when the `spec` +of the Policy-attached object changes. + +Implementations _should_ use their own unique domain prefix for this Condition +`type` - it is recommended that implementations use the same domain as in the +`controllerName` field on GatewayClass (or some other implementation-unique +domain for implementations that do not use GatewayClass).) + +For objects that do _not_ have a `status.Conditions` field available (`Secret` +is a good example), that object MUST instead have an annotation of +`gateway.networking.k8s.io/PolicyAffected: true` (or with an +implementation-specifc domain prefix) added instead. + ### Interaction with Custom Filters and other extension points There are multiple methods of custom extension in the Gateway API. Policy attachment and custom Route filters are two of these. Policy attachment is diff --git a/geps/images/713-the-diagram-with-policy-admin.png b/geps/images/713-the-diagram-with-policy-admin.png new file mode 100644 index 0000000000..181d1d15d7 Binary files /dev/null and b/geps/images/713-the-diagram-with-policy-admin.png differ