Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for Validation Rules feature to Custom Resource Definitions task docs #30494

Closed
wants to merge 4 commits into from
Closed

Add documentation for Validation Rules feature to Custom Resource Definitions task docs #30494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
57b04ca
Add stub for validatiton rule documentation
jpbetz Nov 15, 2021
9d638bf
Apply feedback
jpbetz Nov 18, 2021
b3d010a
Add feature gate `CustomResourceValidationExpressions` into /command-…
cici37 Nov 18, 2021
aadf90a
Adding details on CEL validation for CRD.
cici37 Nov 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions content/en/docs/reference/command-line-tools-reference/feature-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ different Kubernetes components.
| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | 1.21 |
| `ControllerManagerLeaderMigration` | `true` | Beta | 1.22 | |
| `CustomCPUCFSQuotaPeriod` | `false` | Alpha | 1.12 | |
| `CustomResourceValidationExpressions` | `false` | Alpha | 1.23 | |
| `DaemonSetUpdateSurge` | `false` | Alpha | 1.21 | 1.21 |
| `DaemonSetUpdateSurge` | `true` | Beta | 1.22 | |
| `DefaultPodTopologySpread` | `false` | Alpha | 1.19 | 1.19 |
Expand Down Expand Up @@ -669,6 +670,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
version 1 of the same controller is selected.
- `CustomCPUCFSQuotaPeriod`: Enable nodes to change `cpuCFSQuotaPeriod` in
[kubelet config](/docs/tasks/administer-cluster/kubelet-config-file/).
- `CustomResourceValidationExpressions`: Enable expression language validation in CRD which will validate customer resource based on validation rules written in `x-kubernetes-validations` extension.
- `CustomPodDNS`: Enable customizing the DNS settings for a Pod using its `dnsConfig` property.
Check [Pod's DNS Config](/docs/concepts/services-networking/dns-pod-service/#pods-dns-config)
for more details.
Expand Down
302 changes: 300 additions & 2 deletions ...en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,8 +556,9 @@ deleted by Kubernetes.
### Validation

Custom resources are validated via
[OpenAPI v3 schemas](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject)
and you can add additional validation using
[OpenAPI v3 schemas](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject),
by x-kubernetes-validations when the [Validation Rules feature](#validation-rules) is enabled, and you
can add additional validation using
[admission webhooks](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook).

Additionally, the following restrictions are applied to the schema:
Expand All @@ -577,6 +578,11 @@ Additionally, the following restrictions are applied to the schema:
- The field `additionalProperties` cannot be set to `false`.
- The field `additionalProperties` is mutually exclusive with `properties`.

The `x-kubernetes-validations` extension can be used to validate custom resources using [Common
Expression Language (CEL)](https://github.com/google/cel-spec) expressions when the [Validation
rules](#validation-rules) feature is enabled and the CustomResourceDefinition schema is a
[structural schema](#specifying-a-structural-schema).

The `default` field can be set when the [Defaulting feature](#defaulting) is enabled,
which is the case with `apiextensions.k8s.io/v1` CustomResourceDefinitions.
Defaulting is in GA since 1.17 (beta since 1.16 with the `CustomResourceDefaulting`
Expand Down Expand Up @@ -693,6 +699,298 @@ kubectl apply -f my-crontab.yaml
crontab "my-new-cron-object" created
```

## Validation rules

{{< feature-state state="alpha" for_k8s_version="v1.23" >}}

Validation rules are in alpha since 1.23 and validate custom resources when the
`CustomResourceValidationExpressions` [feature
gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled and the schema is a
[structural schema](#specifying-a-structural-schema).

Validation rules use the [Common Expression Language (CEL)](https://github.com/google/cel-spec)
to validate custom resource values. Validation rules are included in
CustomResourceDefinition schemas using the `x-kubernetes-validations` extension.

The Rule is scoped to the location of the `x-kubernetes-validations` extension in the schema.
And `self` variable in the CEL expression is bound to the scoped value.

For example:

```yaml
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "self.minReplicas <= self.replicas"
message: "replicas should be greater than or equal to minReplicas."
- rule: "self.replicas <= self.maxReplicas"
message: "replicas should be smaller than or equal to maxReplicas."
properties:
...
minReplicas:
type: integer
replicas:
type: integer
maxReplicas:
type: integer
```

will reject a request to create this custom resource:

```yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
name: my-new-cron-object
spec:
minReplicas: 0
replicas: 20
maxReplicas: 10
```

with the response:

```
The CronTab "my-new-cron-object" is invalid:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: replicas should be smaller than or equal to maxReplicas.
```

`x-kubernetes-validations` could have multiple rules.

The `rule` under `x-kubernetes-validations` represents the expression which will be evaluated by CEL.

The `message` represents the message displayed when validation fails. If message is unset, the above response would be:
```
The CronTab "my-new-cron-object" is invalid:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: failed rule: self.replicas <= self.maxReplicas
```

Validation rules are compiled when CRDs are created/updated.
jpbetz marked this conversation as resolved.
Show resolved Hide resolved
The request will fail if contains compilation errors no mater feature gate is enabled or not.
Compilation process includes type checking as well.

The compilation failure:
- `no_matching_overload`: this function has no overload for the types of the arguments.

e.g. Rule like `self == true` against a field of integer type will get error:
```
Invalid value: apiextensions.ValidationRule{Rule:"self == true", Message:""}: compilation failed: ERROR: \<input>:1:6: found no matching overload for '_==_' applied to '(int, bool)'
```

- `no_such_field`: does not contain the desired field.

e.g. Rule like `self.nonExistingField > 0` against a non-existing field will return the error:
```
Invalid value: apiextensions.ValidationRule{Rule:"self.nonExistingField > 0", Message:""}: compilation failed: ERROR: \<input>:1:5: undefined field 'nonExistingField'
```

- `invalid argument`: invalid argument to macros.

e.g. Rule like `has(self)` will return error:
```
Invalid value: apiextensions.ValidationRule{Rule:"has(self)", Message:""}: compilation failed: ERROR: <input>:1:4: invalid argument to has() macro
```


Validation Rules Examples:

| Rule | Purpose |
| ---------------- | ------------ |
| `self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas` | Validate that the three fields defining replicas are ordered appropriately |
| `'Available' in self.stateCounts` | Validate that an entry with the 'Available' key exists in a map |
| `(size(self.list1) == 0) != (size(self.list2) == 0)` | Validate that one of two lists is non-empty, but not both |
| <code>!('MY_KEY' in self.map1) &#124;&#124; self['MY_KEY'].matches('^[a-zA-Z]*$')</code> | Validate the value of a map for a specific key, if it is in the map |
| `self.envars.filter(e, e.name = 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$')` | Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV' |
| `has(self.expired) && self.created + self.ttl < self.expired` | Validate that 'expired' date is after a 'create' date plus a 'ttl' duration |
| `self.health.startsWith('ok')` | Validate a 'health' string field has the prefix 'ok' |
| `self.widgets.exists(w, w.key == 'x' && w.foo < 10)` | Validate that the 'foo' property of a listMap item with a key 'x' is less than 10 |
| `type(self) == string ? self == '100%' : self == 1000` | Validate an int-or-string field for both the the int and string cases |
| `self.metadata.name == 'singleton'` | Validate that an object's name matches a specific value (making it a singleton) |
| `self.set1.all(e, !(e in self.set2))` | Validate that two listSets are disjoint |
| `size(self.names) == size(self.details) && self.names.all(n, n in self.details)` | Validate the 'details' map is keyed by the items in the 'names' listSet |

Xref: [Supported evaluation on CEL](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#evaluation)


- If the Rule is scoped to the root of a resource, it may make field selection into any fields
declared in the OpenAPIv3 schema of the CRD as well as `apiVersion`, `kind`, `metadata.name` and
`metadata.generateName`. This includes selection of fields in both the `spec` and `status` in the
same expression:
```yaml
...
openAPIV3Schema:
type: object
x-kubernetes-validation-rules:
- rule: "elf.status.availableReplicas >= self.spec.minReplicas"
properties:
spec:
type: object
properties:
minReplicas:
type: integer
...
status:
type: object
properties:
availableReplicas:
type: integer
```

- If the Rule is scoped to an object with properties, the accessible properties of the object are field selectable
via `self.field` and field presence can be checked via `has(self.field)`. Null valued fields are treated as
absent fields in CEL expressions.

```yaml
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "has(self.foo)"
properties:
...
foo:
type: integer
```

- If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map
are accessible via `self[mapKey]`, map containment can be checked via `mapKey in self` and all entries of the map
are accessible via CEL macros and functions such as `self.all(...)`.
```yaml
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-validation-rules:
- rule: "self['xyz'].foo > 0"
additionalProperties:
...
type: object
properties:
foo:
type: integer
```

- If the Rule is scoped to an array, the elements of the array are accessible via `self[i]` and also by macros and
functions.
```yaml
...
openAPIV3Schema:
type: object
properties:
...
foo:
type: array
x-kubernetes-validation-rules:
- rule: "self[0].startsWith('kube')"
items:
type: string
```

- If the Rule is scoped to a scalar, `self` is bound to the scalar value.
```yaml
...
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
...
foo:
type: integer
x-kubernetes-validation-rules:
- rule: "self > 0"
```
Examples:

|type of the field rule scoped to | Rule example |
| -----------------------| -----------------------|
| root object | `self.status.actual <= self.spec.maxDesired`|
| map of objects | `self.components['Widget'].priority < 10`|
| list of integers | `self.values.all(value, value >= 0 && value < 100)`|
| string | `self.startsWith('kube')`|


The `apiVersion`, `kind`, `metadata.name` and `metadata.generateName` are always accessible from the root of the
object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.

Unknown data preserved in custom resources via `x-kubernetes-preserve-unknown-fields` is not accessible in CEL
expressions. This includes:
- Unknown field values that are preserved by object schemas with x-kubernetes-preserve-unknown-fields.
- Object properties where the property schema is of an "unknown type". An "unknown type" is recursively defined as:
- A schema with no type and x-kubernetes-preserve-unknown-fields set to true
- An array where the items schema is of an "unknown type"
- An object where the additionalProperties schema is of an "unknown type"


Only property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible.
Accessible property names are escaped according to the following rules when accessed in the expression:

| escape sequence | property name equivalent |
| ----------------------- | -----------------------|
| `__underscores__` | `__` |
| `__dot__` | `.` |
|`__dash__` | `-` |
| `__slash__` | `/` |
| `__{keyword}__` | [CEL RESERVED keyword](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#syntax) |

Note: CEL RESERVED keyword needs to match the exact property name to be escaped (e.g. int in the word sprint would not be escaped).

Examples on escaping:

|property name | rule with escaped property name |
| ----------------| ----------------------- |
| namespace | "self.\_\_namespace__ > 0" |
| x-prop | "self.x__dash__prop > 0" |
| redact__d | "self.redact__underscores__d > 0" |
| string | "self.startsWith('kube')" |


Equality on arrays with `x-kubernetes-list-type` of `set` or `map` ignores element order, i.e. [1, 2] == [2, 1].
Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:
- `set`: `X + Y` performs a union where the array positions of all elements in `X` are preserved and
non-intersecting elements in `Y` are appended, retaining their partial order.
- `map`: `X + Y` performs a merge where the array positions of all keys in `X` are preserved but the values
are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with
non-intersecting keys are appended, retaining their partial order.


Here is the declarations type mapping between OpenAPIv3 and CEL type:

| OpenAPIv3 type | CEL type |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| 'object' with Properties | object / "message type" |
| 'object' with AdditionalProperties | map |
| 'object' with x-kubernetes-embedded-type | object / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema |
| 'object' with x-kubernetes-preserve-unknown-fields | object / "message type", unknown fields are NOT accessible in CEL expression |
| x-kubernetes-int-or-string | dynamic object that is either an int or a string, `type(value)` can be used to check the type |
| 'array | list |
| 'array' with x-kubernetes-list-type=map | list with map based Equality & unique key guarantees |
| 'array' with x-kubernetes-list-type=set | list with set based Equality & unique entry guarantees |
| 'boolean' | boolean |
| 'number' (all formats) | double |
| 'integer' (all formats) | int (64) |
| 'null' | null_type |
| 'string' | string |
| 'string' with format=byte (base64 encoded) | bytes |
| 'string' with format=date | timestamp (google.protobuf.Timestamp) |
| 'string' with format=datetime | timestamp (google.protobuf.Timestamp) |
| 'string' with format=duration | duration (google.protobuf.Duration) |

xref: [CEL types](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#values), [OpenAPI
types](https://swagger.io/specification/#data-types), [Kubernetes Structural Schemas](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema).



### Defaulting

{{< note >}}
Expand Down