From e308bcc355b533cb1b75f736d541d4aff2a5e743 Mon Sep 17 00:00:00 2001 From: "Dr. Stefan Schimanski" Date: Tue, 3 Sep 2019 08:20:39 +0200 Subject: [PATCH] CRDs: promote to v1 --- .../api-extension/custom-resources.md | 4 +- .../custom-resource-definition-versioning.md | 513 +++++++++++++++++- .../custom-resource-definitions.md | 461 +++++++++++++--- 3 files changed, 881 insertions(+), 97 deletions(-) diff --git a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md index 33143c04a5b6a..891c4d3cda716 100644 --- a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md +++ b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md @@ -174,7 +174,7 @@ Aggregated APIs offer more advanced API features and customization of other feat | Feature | Description | CRDs | Aggregated API | | ------- | ----------- | ---- | -------------- | | Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Yes. Most validation can be specified in the CRD using [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation). Any other validations supported by addition of a [Validating Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook-alpha-in-1-8-beta-in-1-9). | Yes, arbitrary validation checks | -| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) `default` keyword (alpha in 1.15), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9) | Yes | +| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) `default` keyword (beta in 1.16), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9) | Yes | | Multi-versioning | Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions. | [Yes](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning) | Yes | | Custom Storage | If you need storage with a different performance mode (for example, time-series database instead of key-value store) or isolation for security (for example, encryption secrets or different | No | Yes | | Custom Business Logic | Perform arbitrary checks or actions when creating, reading, updating or deleting an object | Yes, using [Webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks). | Yes | @@ -183,7 +183,7 @@ Aggregated APIs offer more advanced API features and customization of other feat | Other Subresources | Add operations other than CRUD, such as "logs" or "exec". | No | Yes | | strategic-merge-patch | The new endpoints support PATCH with `Content-Type: application/strategic-merge-patch+json`. Useful for updating objects that may be modified both locally, and by the server. For more information, see ["Update API Objects in Place Using kubectl patch"](/docs/tasks/run-application/update-api-object-kubectl-patch/) | No | Yes | | Protocol Buffers | The new resource supports clients that want to use Protocol Buffers | No | Yes | -| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | Yes, based on the [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) schema (beta in 1.15) | Yes | +| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | Yes, based on the [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) schema (GA in 1.16) | Yes | ### Common Features diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md index 12c6f5697290f..45cabeefd4319 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md @@ -1,7 +1,6 @@ --- -title: Versions of CustomResourceDefinitions +title: Versions in CustomResourceDefinitions reviewers: -- mbohlool - sttts - liggitt content_template: templates/task @@ -19,7 +18,7 @@ level of your CustomResourceDefinitions or advance your API to a new version wit {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} -* Make sure your Kubernetes cluster has a master version of 1.11.0 or higher. +* Make sure your Kubernetes cluster has a master version of 1.16.0 or higher for `apiextensions.k8s.io/v1`, or 1.11.0 or higher for `apiextensions.k8s.io/v1beta1`. * Read about [custom resources](/docs/concepts/api-extension/custom-resources/). @@ -29,7 +28,7 @@ level of your CustomResourceDefinitions or advance your API to a new version wit ## Overview -{{< feature-state state="beta" for_kubernetes_version="1.15" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} The CustomResourceDefinition API supports a `versions` field that you can use to support multiple versions of custom resources that you have developed. Versions @@ -38,7 +37,7 @@ Webhook conversions should follow the [Kubernetes API conventions](https://githu Specifically, See the [API change documentation](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md) for a set of useful gotchas and suggestions. {{< note >}} -Earlier iterations included a `version` field instead of `versions`. The +In `apiextensions.k8s.io/v1beta1`, there was a `version` field instead of `versions`. The `version` field is deprecated and optional, but if it is not empty, it must match the first item in the `versions` field. {{< /note >}} @@ -49,7 +48,67 @@ This example shows a CustomResourceDefinition with two versions. For the first example, the assumption is all versions share the same schema with no conversion between them. The comments in the YAML provide more context. +{{< tabs name="CustomResourceDefinition_versioning_example_1" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + # name must match the spec fields below, and be in the form: . + name: crontabs.example.com +spec: + # group name to use for REST API: /apis// + group: example.com + # list of versions supported by this CustomResourceDefinition + versions: + - name: v1beta1 + # Each version can be enabled/disabled by Served flag. + served: true + # One and only one version must be marked as the storage version. + storage: true + # A schema is required + schema: + openAPIV3Schema: + type: object + properties: + host: + type: string + port: + type: string + - name: v1 + served: true + storage: false + schema: + openAPIV3Schema: + type: object + properties: + host: + type: string + port: + type: string + # The conversion section is introduced in Kubernetes 1.13+ with a default value of + # None conversion (strategy sub-field set to None). + conversion: + # None conversion assumes the same schema for all versions and only sets the apiVersion + # field of custom resources to the proper value + strategy: None + # either Namespaced or Cluster + scope: Namespaced + names: + # plural name to be used in the URL: /apis/// + plural: crontabs + # singular name to be used as an alias on the CLI and for display + singular: crontab + # kind is normally the CamelCased singular type. Your resource manifests use this. + kind: CronTab + # shortNames allow shorter string to match your resource on the CLI + shortNames: + - ct +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -68,6 +127,14 @@ spec: - name: v1 served: true storage: false + validation: + openAPIV3Schema: + type: object + properties: + host: + type: string + port: + type: string # The conversion section is introduced in Kubernetes 1.13+ with a default value of # None conversion (strategy sub-field set to None). conversion: @@ -87,6 +154,8 @@ spec: shortNames: - ct ``` +{{% /tab %}} +{{< /tabs >}} You can save the CustomResourceDefinition in a YAML file, then use `kubectl apply` to create it. @@ -149,7 +218,7 @@ the version. ## Webhook conversion -{{< feature-state state="beta" for_kubernetes_version="1.15" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} {{< note >}} Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13. The @@ -169,13 +238,13 @@ To cover all of these cases and to optimize conversion by the API server, the co ### Write a conversion webhook server Please refer to the implementation of the [custom resource conversion webhook -server](https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/crd-conversion-webhook/main.go) +server](https://github.com/kubernetes/kubernetes/tree/v1.15.0/test/images/crd-conversion-webhook/main.go) that is validated in a Kubernetes e2e test. The webhook handles the `ConversionReview` requests sent by the API servers, and sends back conversion results wrapped in `ConversionResponse`. Note that the request contains a list of custom resources that need to be converted independently without changing the order of objects. -The example server is organized in a way to be reused for other conversions. Most of the common code are located in the [framework file](https://github.com/kubernetes/kubernetes/tree/v1.14.0/test/images/crd-conversion-webhook/converter/framework.go) that leaves only [one function](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/crd-conversion-webhook/converter/example_converter.go#L29-L80) to be implemented for different conversions. +The example server is organized in a way to be reused for other conversions. Most of the common code are located in the [framework file](https://github.com/kubernetes/kubernetes/tree/v1.15.0/test/images/crd-conversion-webhook/converter/framework.go) that leaves only [one function](https://github.com/kubernetes/kubernetes/blob/v1.15.0/test/images/crd-conversion-webhook/converter/example_converter.go#L29-L80) to be implemented for different conversions. {{< note >}} The example conversion webhook server leaves the `ClientAuth` field @@ -208,7 +277,75 @@ if a different port is used for the service. The `None` conversion example can be extended to use the conversion webhook by modifying `conversion` section of the `spec`: +{{< tabs name="CustomResourceDefinition_versioning_example_2" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + # name must match the spec fields below, and be in the form: . + name: crontabs.example.com +spec: + # group name to use for REST API: /apis// + group: example.com + # list of versions supported by this CustomResourceDefinition + versions: + - name: v1beta1 + # Each version can be enabled/disabled by Served flag. + served: true + # One and only one version must be marked as the storage version. + storage: true + # Each version can define it's own schema when there is no top-level + # schema is defined. + schema: + openAPIV3Schema: + type: object + properties: + hostPort: + type: string + - name: v1 + served: true + storage: false + schema: + openAPIV3Schema: + type: object + properties: + host: + type: string + port: + type: string + conversion: + # a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources. + strategy: Webhook + # webhook is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server. + webhook: + # conversionReviewVersions indicates what ConversionReview versions are understood/preferred by the webhook. + # The first version in the list understood by the API server is sent to the webhook. + # The webhook must respond with a ConversionReview object in the same version it received. + conversionReviewVersions: ["v1","v1beta1"] + clientConfig: + service: + namespace: default + name: example-conversion-webhook-server + path: /crdconvert + caBundle: "Ci0tLS0tQk......tLS0K" + # either Namespaced or Cluster + scope: Namespaced + names: + # plural name to be used in the URL: /apis/// + plural: crontabs + # singular name to be used as an alias on the CLI and for display + singular: crontab + # kind is normally the CamelCased singular type. Your resource manifests use this. + kind: CronTab + # shortNames allow shorter string to match your resource on the CLI + shortNames: + - ct +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -217,6 +354,8 @@ metadata: spec: # group name to use for REST API: /apis// group: example.com + # prunes object fields that are not specified in OpenAPI schemas below. + preserveUnknownFields: false # list of versions supported by this CustomResourceDefinition versions: - name: v1beta1 @@ -228,6 +367,7 @@ spec: # schema is defined. schema: openAPIV3Schema: + type: object properties: hostPort: type: string @@ -236,6 +376,7 @@ spec: storage: false schema: openAPIV3Schema: + type: object properties: host: type: string @@ -244,14 +385,13 @@ spec: conversion: # a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources. strategy: Webhook - # webhookClientConfig is required when strategy is `Webhook` and it configure the webhook endpoint to be - # called by API server. + # webhookClientConfig is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server. webhookClientConfig: service: namespace: default name: example-conversion-webhook-server path: /crdconvert - caBundle: + caBundle: "Ci0tLS0tQk......tLS0K" # either Namespaced or Cluster scope: Namespaced names: @@ -265,6 +405,8 @@ spec: shortNames: - ct ``` +{{% /tab %}} +{{< /tabs >}} You can save the CustomResourceDefinition in a YAML file, then use `kubectl apply` to apply it. @@ -309,7 +451,25 @@ Fragments ("#...") and query parameters ("?...") are also not allowed. Here is an example of a conversion webhook configured to call a URL (and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle): +{{< tabs name="CustomResourceDefinition_versioning_example_3" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + webhook: + clientConfig: + url: "https://my-webhook.example.com:9443/my-webhook-path" +... +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition ... @@ -321,6 +481,8 @@ spec: url: "https://my-webhook.example.com:9443/my-webhook-path" ... ``` +{{% /tab %}} +{{< /tabs >}} ### Service Reference @@ -333,7 +495,30 @@ Here is an example of a webhook that is configured to call a service on port "12 at the subpath "/my-path", and to verify the TLS connection against the ServerName `my-service-name.my-service-namespace.svc` using a custom CA bundle. +{{< tabs name="CustomResourceDefinition_versioning_example_4" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +apiVersion: apiextensions.k8s.io/v1b +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + webhook: + clientConfig: + service: + namespace: my-service-namespace + name: my-service-name + path: /my-path + port: 1234 + caBundle: "Ci0tLS0tQk......tLS0K" +... +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} ```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition ... @@ -350,6 +535,312 @@ spec: caBundle: "Ci0tLS0tQk......tLS0K" ... ``` +{{% /tab %}} +{{< /tabs >}} + +## Webhook request and response + +### Request + +Webhooks are sent a POST request, with `Content-Type: application/json`, +with a `ConversionReview` API object in the `apiextensions.k8s.io` API group +serialized to JSON as the body. + +Webhooks can specify what versions of `ConversionReview` objects they accept +with the `conversionReviewVersions` field in their CustomResourceDefinition: + +{{< tabs name="conversionReviewVersions" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + webhook: + conversionReviewVersions: ["v1", "v1beta1"] + ... +``` + +`conversionReviewVersions` is a required field when creating +`apiextensions.k8s.io/v1` custom resource definitions. +Webhooks are required to support at least one `ConversionReview` +version understood by the current and previous API server. +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + conversionReviewVersions: ["v1", "v1beta1"] + ... +``` + +If no `conversionReviewVersions` are specified, the default when creating +`apiextensions.k8s.io/v1beta1` custom resource definitions is `v1beta1`. +{{% /tab %}} +{{< /tabs >}} + +API servers send the first `ConversionReview` version in the `conversionReviewVersions` list they support. +If none of the versions in the list are supported by the API server, the custom resource definition will not be allowed to be created. +If an API server encounters a conversion webhook configuration that was previously created and does not support any of the `ConversionReview` +versions the API server knows how to send, attempts to call to the webhook will fail. + +This example shows the data contained in an `ConversionReview` object +for a request to convert `CronTab` objects to `example.com/v1`: + + +{{< tabs name="ConversionReview_request" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +{ + "apiVersion": "apiextensions.k8s.io/v1", + "kind": "ConversionReview", + "request": { + # Random uid uniquely identifying this conversion call + "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", + + # The API group and version the objects should be converted to + "desiredAPIVersion": "example.com/v1", + + # The list of objects to convert. + # May contain one or more objects, in one or more versions. + "objects": [ + { + "kind": "CronTab", + "apiVersion": "example.com/v1beta1", + "metadata": { + "creationTimestamp": "2019-09-04T14:03:02Z", + "name": "local-crontab", + "namespace": "default", + "resourceVersion": "143", + "uid": "3415a7fc-162b-4300-b5da-fd6083580d66" + }, + "hostPort": "localhost:1234" + }, + { + "kind": "CronTab", + "apiVersion": "example.com/v1beta1", + "metadata": { + "creationTimestamp": "2019-09-03T13:02:01Z", + "name": "remote-crontab", + "resourceVersion": "12893", + "uid": "359a83ec-b575-460d-b553-d859cedde8a0" + }, + "hostPort": "example.com:2345" + } + ] + } +} +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +{ + # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 + "apiVersion": "apiextensions.k8s.io/v1beta1", + "kind": "ConversionReview", + "request": { + # Random uid uniquely identifying this conversion call + "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", + + # The API group and version the objects should be converted to + "desiredAPIVersion": "example.com/v1", + + # The list of objects to convert. + # May contain one or more objects, in one or more versions. + "objects": [ + { + "kind": "CronTab", + "apiVersion": "example.com/v1beta1", + "metadata": { + "creationTimestamp": "2019-09-04T14:03:02Z", + "name": "local-crontab", + "namespace": "default", + "resourceVersion": "143", + "uid": "3415a7fc-162b-4300-b5da-fd6083580d66" + }, + "hostPort": "localhost:1234" + }, + { + "kind": "CronTab", + "apiVersion": "example.com/v1beta1", + "metadata": { + "creationTimestamp": "2019-09-03T13:02:01Z", + "name": "remote-crontab", + "resourceVersion": "12893", + "uid": "359a83ec-b575-460d-b553-d859cedde8a0" + }, + "hostPort": "example.com:2345" + } + ] + } +} +``` +{{% /tab %}} +{{< /tabs >}} + +### Response + +Webhooks respond with a 200 HTTP status code, `Content-Type: application/json`, +and a body containing a `ConversionReview` object (in the same version they were sent), +with the `response` stanza populated, serialized to JSON. + +If conversion succeeds, a webhook should return a `response` stanza containing the following fields: +* `uid`, copied from the `request.uid` sent to the webhook +* `result`, set to `{"status":"Success"}` +* `convertedObjects`, containing all of the objects from `request.objects`, converted to `request.desiredVersion` + +Example of a minimal successful response from a webhook: + +{{< tabs name="ConversionReview_response_success" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +{ + "apiVersion": "apiextensions.k8s.io/v1", + "kind": "ConversionReview", + "response": { + # must match + "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", + "result": { + "status": "Success" + }, + # Objects must match the order of request.objects, and have apiVersion set to . + # kind, metadata.uid, metadata.name, and metadata.namespace fields must not be changed by the webhook. + # metadata.labels and metadata.annotations fields may be changed by the webhook. + # All other changes to metadata fields by the webhook are ignored. + "convertedObjects": [ + { + "kind": "CronTab", + "apiVersion": "example.com/v1", + "metadata": { + "creationTimestamp": "2019-09-04T14:03:02Z", + "name": "local-crontab", + "namespace": "default", + "resourceVersion": "143", + "uid": "3415a7fc-162b-4300-b5da-fd6083580d66" + }, + "host": "localhost", + "port": "1234" + }, + { + "kind": "CronTab", + "apiVersion": "example.com/v1", + "metadata": { + "creationTimestamp": "2019-09-03T13:02:01Z", + "name": "remote-crontab", + "resourceVersion": "12893", + "uid": "359a83ec-b575-460d-b553-d859cedde8a0" + }, + "host": "example.com", + "port": "2345" + } + ] + } +} +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +{ + # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 + "apiVersion": "apiextensions.k8s.io/v1beta1", + "kind": "ConversionReview", + "response": { + # must match + "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", + "result": { + "status": "Failed" + }, + # Objects must match the order of request.objects, and have apiVersion set to . + # kind, metadata.uid, metadata.name, and metadata.namespace fields must not be changed by the webhook. + # metadata.labels and metadata.annotations fields may be changed by the webhook. + # All other changes to metadata fields by the webhook are ignored. + "convertedObjects": [ + { + "kind": "CronTab", + "apiVersion": "example.com/v1", + "metadata": { + "creationTimestamp": "2019-09-04T14:03:02Z", + "name": "local-crontab", + "namespace": "default", + "resourceVersion": "143", + "uid": "3415a7fc-162b-4300-b5da-fd6083580d66" + }, + "host": "localhost", + "port": "1234" + }, + { + "kind": "CronTab", + "apiVersion": "example.com/v1", + "metadata": { + "creationTimestamp": "2019-09-03T13:02:01Z", + "name": "remote-crontab", + "resourceVersion": "12893", + "uid": "359a83ec-b575-460d-b553-d859cedde8a0" + }, + "host": "example.com", + "port": "2345" + } + ] + } +} +``` +{{% /tab %}} +{{< /tabs >}} + +If conversion fails, a webhook should return a `response` stanza containing the following fields: +* `uid`, copied from the `request.uid` sent to the webhook +* `result`, set to `{"status":"Failed"}` + +{{< warning >}} +Failing conversion can disrupt read and write access to the custom resources, +including the ability to update or delete the resources. Conversion failures +should be avoided whenever possible, and should not be used to enforce validation + constraints (use validation schemas or webhook admission instead). +{{< /warning >}} + +Example of a response from a webhook indicating a conversion request failed, with an optional message: +{{< tabs name="ConversionReview_response_failure" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +{ + "apiVersion": "apiextensions.k8s.io/v1", + "kind": "ConversionReview", + "response": { + "uid": "", + "result": { + "status": "Failed", + "message": "hostPort could not be parsed into a separate host and port" + } + } +} +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +{ + # Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 + "apiVersion": "apiextensions.k8s.io/v1beta1", + "kind": "ConversionReview", + "response": { + "uid": "", + "result": { + "status": "Failed", + "message": "hostPort could not be parsed into a separate host and port" + } + } +} +``` +{{% /tab %}} +{{< /tabs >}} ## Writing, reading, and updating versioned CustomResourceDefinition objects diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md index 690b76053d468..2ebf3ad21de1f 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md @@ -2,7 +2,9 @@ title: Extend the Kubernetes API with CustomResourceDefinitions reviewers: - deads2k -- enisoc +- jpbetz +- liggitt +- roycaihw - sttts content_template: templates/task weight: 20 @@ -19,7 +21,7 @@ into the Kubernetes API by creating a {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} -* Make sure your Kubernetes cluster has a master version of 1.7.0 or higher. +* Make sure your Kubernetes cluster has a master version of 1.16.0 or higher to use `apiextensions.k8s.io/v1`, or 1.7.0 or higher for `apiextensions.k8s.io/v1beta1`. * Read about [custom resources](/docs/concepts/api-extension/custom-resources/). @@ -37,7 +39,54 @@ are available to all namespaces. For example, if you save the following CustomResourceDefinition to `resourcedefinition.yaml`: +{{< tabs name="CustomResourceDefinition_example_1" >}} +{{% tab name="admissionregistration.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + # name must match the spec fields below, and be in the form: . + name: crontabs.stable.example.com +spec: + # group name to use for REST API: /apis// + group: stable.example.com + # list of versions supported by this CustomResourceDefinition + versions: + - name: v1 + # Each version can be enabled/disabled by Served flag. + served: true + # One and only one version must be marked as the storage version. + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + # either Namespaced or Cluster + scope: Namespaced + names: + # plural name to be used in the URL: /apis/// + plural: crontabs + # singular name to be used as an alias on the CLI and for display + singular: crontab + # kind is normally the CamelCased singular type. Your resource manifests use this. + kind: CronTab + # shortNames allow shorter string to match your resource on the CLI + shortNames: + - ct +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -80,6 +129,8 @@ spec: replicas: type: integer ``` +{{% /tab %}} +{{< /tabs >}} And create it: @@ -192,11 +243,11 @@ If you later recreate the same CustomResourceDefinition, it will start out empty ## Specifying a structural schema -{{< feature-state state="beta" for_kubernetes_version="1.15" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} CustomResources traditionally store arbitrary JSON (next to `apiVersion`, `kind` and `metadata`, which is validated by the API server implicitly). With [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) a schema can be specified, which is validated during creation and updates, compare below for details and limits of such a schema. -With `apiextensions.k8s.io/v1` the definition of a structural schema will be mandatory for CustomResourceDefinitions, while in `v1beta1` this is still optional. +With `apiextensions.k8s.io/v1` the definition of a structural schema is mandatory for CustomResourceDefinitions, while in `v1beta1` this is still optional. A structural schema is an [OpenAPI v3.0 validation schema](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) which: @@ -305,24 +356,34 @@ anyOf: Violations of the structural schema rules are reported in the `NonStructural` condition in the CustomResourceDefinition. -Not being structural disables the following features: +Structural schemas are a requirement for `apiextensions.k8s.io/v1`, and disables the following features for `apiextensions.k8s.io/v1beta1`: * [Validation Schema Publishing](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#publish-validation-schema-in-openapi-v2) * [Webhook Conversion](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/#webhook-conversion) -* [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) * [Pruning](#preserving-unknown-fields) -and possibly more features in the future. - ### Pruning versus preserving unknown fields -{{< feature-state state="beta" for_kubernetes_version="1.15" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} CustomResourceDefinitions traditionally store any (possibly validated) JSON as is in etcd. This means that unspecified fields (if there is a [OpenAPI v3.0 validation schema](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) at all) are persisted. This is in contrast to native Kubernetes resources like e.g. a pod where unknown fields are dropped before being persisted to etcd. We call this "pruning" of unknown fields. - -If a [structural OpenAPI v3 validation schema](#specifying-a-structural-schema) is defined (either in the global `spec.validation.openAPIV3Schema` or for each version) in a CustomResourceDefinition, pruning can be enabled by setting `spec.preserveUnknownFields` to `false`. Then unspecified fields on creation and on update are dropped. -Compare the CustomResourceDefinition `crontabs.stable.example.com` above. It has pruning enabled. Hence, if you save the following YAML to `my-crontab.yaml`: +{{< tabs name="CustomResourceDefinition_pruning" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} + +For CustomResourceDefinitions created in `apiextensions.k8s.io/v1`, [structural OpenAPI v3 validation schemas](#specifying-a-structural-schema) are required and pruning is enabled and cannot be disabled (note that CRDs converted from `apiextensions.k8s.io/v1beta1` to `apiextensions.k8s.io/v1` might lack structural schemas, and `spec.preserveUnknownFields` might be `true`). + +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} + +For CustomResourceDefinitions created in `apiextensions.k8s.io/v1beta1`, if a [structural OpenAPI v3 validation schema](#specifying-a-structural-schema) is defined (either in the global `spec.validation.openAPIV3Schema` in `apiextensions.k8s.io/v1beta1` or for each version) in a CustomResourceDefinition, pruning can be enabled by setting `spec.preserveUnknownFields` to `false`. + +{{% /tab %}} +{{% /tabs %}} + +If pruning is enabled, unspecified fields in CustomResources on creation and on update are dropped. + +Compare the CustomResourceDefinition `crontabs.stable.example.com` above. It has pruning enabled (both in `apiextensions.k8s.io/v1` and `apiextensions.k8s.io/v1beta1). Hence, if you save the following YAML to `my-crontab.yaml`: ```yaml apiVersion: "stable.example.com/v1" @@ -362,11 +423,9 @@ The field `someRandomField` has been pruned. Note that the `kubectl create` call uses `--validate=false` to skip client-side validation. Because the [OpenAPI validation schemas are also published](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#publish-validation-schema-in-openapi-v2) to kubectl, it will also check for unknown fields and reject those objects long before they are sent to the API server. -In `apiextensions.k8s.io/v1beta1`, pruning is disabled by default, i.e. `spec.preserveUnknownFields` defaults to `true`. In `apiextensions.k8s.io/v1` no new CustomResourceDefinitions with `spec.preserveUnknownFields: true` will be allowed to be created. - ### Controlling pruning -With `spec.preserveUnknownField: false` in the CustomResourceDefinition, pruning is enabled for all custom resources of that type and in all versions. It is possible though to opt-out of that for JSON sub-trees via `x-kubernetes-preserve-unknown-fields: true` in the [structural OpenAPI v3 validation schema](#specifying-a-structural-schema): +If pruning is enabled (enforced in `apiextensions.k8s.io/v1`, or as opt-in via `spec.preserveUnknownField: false` in `apiextensions.k8s.io/v1beta1`) in the CustomResourceDefinition, all unspecified fields in custom resources of that type and in all versions are pruned. It is possible though to opt-out of that for JSON sub-trees via `x-kubernetes-preserve-unknown-fields: true` in the [structural OpenAPI v3 validation schema](#specifying-a-structural-schema): ```yaml type: object @@ -545,10 +604,11 @@ meaning all finalizers have been executed. ### Validation -{{< feature-state state="beta" for_kubernetes_version="1.9" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} Validation of custom objects is possible via -[OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject) or [validatingadmissionwebhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook). +[OpenAPI v3 schemas](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject) or [validatingadmissionwebhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook). In `apiextensions.k8s.io/v1` schemas are required, in `apiextensions.k8s.io/v1beta1` they are optional. + Additionally, the following restrictions are applied to the schema: - These fields cannot be set: @@ -568,15 +628,10 @@ Additionally, the following restrictions are applied to the schema: These fields can only be set with specific features enabled: -- `default`: the `CustomResourceDefaulting` feature gate must be enabled, compare [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting). +- `default`: can be set for `apiextensions.k8s.io/v1` CustomResourceDefinitions. Defaulting is in beta since 1.16 and requires the `CustomResourceDefaulting` feature gate to be enabled (which is the case automatically for many clusters for beta features). Compare [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting). Note: compare with [structural schemas](#specifying-a-structural-schema) for further restriction required for certain CustomResourceDefinition features. -{{< note >}} -OpenAPI v3 validation is available as beta. The -`CustomResourceValidation` feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. -{{< /note >}} - The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object: @@ -585,7 +640,46 @@ CustomResourceDefinition applies the following validations on the custom object: Save the CustomResourceDefinition to `resourcedefinition.yaml`: +{{< tabs name="CustomResourceDefinition_validation" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + versions: + - name: v1 + served: true + storage: true + schema: + # openAPIV3Schema is the schema for validating custom objects. + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' + replicas: + type: integer + minimum: 1 + maximum: 10 + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -620,6 +714,8 @@ spec: minimum: 1 maximum: 10 ``` +{{% /tab %}} +{{< /tabs >}} And create it: @@ -685,18 +781,16 @@ crontab "my-new-cron-object" created ### Defaulting -{{< feature-state state="alpha" for_kubernetes_version="1.15" >}} +{{< feature-state state="beta" for_kubernetes_version="1.16" >}} {{< note >}} -Defaulting is available as alpha since 1.15. It is disabled by default and can be enabled via the `CustomResourceDefaulting` feature gate. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. - -Defaulting also requires a structural schema and pruning. +Defaulting is available as beta since 1.16 in `apiextensions.k8s.io/v1` CustomResourceDefinitions, and hence enabled by default for most clusters (feature gate `CustomResourceDefaulting`, refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation). {{< /note >}} Defaulting allows to specify default values in the [OpenAPI v3 validation schema](#validation): ```yaml -apiVersion: apiextensions.k8s.io/v1beta1 +apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com @@ -706,34 +800,32 @@ spec: - name: v1 served: true storage: true - version: v1 + schema: + # openAPIV3Schema is the schema for validating custom objects. + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' + default: "5 0 * * *" + image: + type: string + replicas: + type: integer + minimum: 1 + maximum: 10 + default: 1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - - ct - preserveUnknownFields: false - validation: - # openAPIV3Schema is the schema for validating custom objects. - openAPIV3Schema: - type: object - properties: - spec: - type: object - properties: - cronSpec: - type: string - pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' - default: "5 0 * * *" - image: - type: string - replicas: - type: integer - minimum: 1 - maximum: 10 - default: 1 + - ct ``` With this both `cronSpec` and `replicas` are defaulted: @@ -762,15 +854,19 @@ spec: Note that defaulting happens on the object -* in the request to the API server using the request version defaults -* when reading from etcd using the storage version defaults +* in the request to the API server using the request version defaults, +* when reading from etcd using the storage version defaults, * after mutating admission plugins with non-empty patches using the admission webhook object version defaults. -Note that defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd. +Defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd. + +Default values must be pruned (with the exception of defaults for `metadata` fields) and must validate against a provided schema. + +Default values for `metadata` fields of `x-kubernetes-embedded-resources: true` nodes (or parts of a default value covering `metadata`) are not pruned during CustomResourceDefinition creation, but through the pruning step during handling of requests. ### Publish Validation Schema in OpenAPI v2 -{{< feature-state state="beta" for_kubernetes_version="1.15" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} {{< note >}} OpenAPI v2 Publishing is available as beta since 1.15, and as alpha since 1.14. The @@ -801,34 +897,98 @@ CustomResourceDefinition. The following example adds the `Spec`, `Replicas`, and columns. 1. Save the CustomResourceDefinition to `resourcedefinition.yaml`. - ```yaml - apiVersion: apiextensions.k8s.io/v1beta1 - kind: CustomResourceDefinition - metadata: - name: crontabs.stable.example.com - spec: - group: stable.example.com - version: v1 - scope: Namespaced - names: - plural: crontabs - singular: crontab - kind: CronTab - shortNames: - - ct - additionalPrinterColumns: - - name: Spec - type: string - description: The cron spec defining the interval a CronJob is run - JSONPath: .spec.cronSpec - - name: Replicas - type: integer - description: The number of jobs launched by the CronJob - JSONPath: .spec.replicas - - name: Age - type: date - JSONPath: .metadata.creationTimestamp - ``` + {{< tabs name="CustomResourceDefinition_printer_columns" >}} + {{% tab name="apiextensions.k8s.io/v1" %}} +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + additionalPrinterColumns: + - name: Spec + type: string + description: The cron spec defining the interval a CronJob is run + jsonPath: .spec.cronSpec + - name: Replicas + type: integer + description: The number of jobs launched by the CronJob + jsonPath: .spec.replicas + - name: Age + type: date + jsonPath: .metadata.creationTimestamp +``` + {{% /tab %}} + {{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + version: v1 + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct + validation: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + additionalPrinterColumns: + - name: Spec + type: string + description: The cron spec defining the interval a CronJob is run + JSONPath: .spec.cronSpec + - name: Replicas + type: integer + description: The number of jobs launched by the CronJob + JSONPath: .spec.replicas + - name: Age + type: date + JSONPath: .metadata.creationTimestamp +``` + {{% /tab %}} + {{< /tabs >}} 2. Create the CustomResourceDefinition: @@ -892,7 +1052,7 @@ The column's `format` controls the style used when `kubectl` prints the value. ### Subresources -{{< feature-state state="beta" for_kubernetes_version="1.11" >}} +{{< feature-state state="stable" for_kubernetes_version="1.16" >}} Custom resources support `/status` and `/scale` subresources. @@ -972,7 +1132,63 @@ In the following example, both status and scale subresources are enabled. Save the CustomResourceDefinition to `resourcedefinition.yaml`: +{{< tabs name="CustomResourceDefinition_scale" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + status: + type: object + properties: + replicas: + type: integer + labelSelector: + type: string + # subresources describes the subresources for custom resources. + subresources: + # status enables the status subresource. + status: {} + # scale enables the scale subresource. + scale: + # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas. + specReplicasPath: .spec.replicas + # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas. + statusReplicasPath: .status.replicas + # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector. + labelSelectorPath: .status.labelSelector + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -990,6 +1206,26 @@ spec: kind: CronTab shortNames: - ct + validation: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + status: + type: object + properties: + replicas: + type: integer + labelSelector: + type: string # subresources describes the subresources for custom resources. subresources: # status enables the status subresource. @@ -1003,6 +1239,8 @@ spec: # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector. labelSelectorPath: .status.labelSelector ``` +{{% /tab %}} +{{< /tabs >}} And create it: @@ -1068,7 +1306,47 @@ and illustrates how to output the custom resource using `kubectl get all`. Save the following CustomResourceDefinition to `resourcedefinition.yaml`: +{{< tabs name="CustomResourceDefinition_categories" >}} +{{% tab name="apiextensions.k8s.io/v1" %}} ```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct + # categories is a list of grouped resources the custom resource belongs to. + categories: + - all +``` +{{% /tab %}} +{{% tab name="apiextensions.k8s.io/v1beta1" %}} +```yaml +# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: @@ -1079,6 +1357,19 @@ spec: - name: v1 served: true storage: true + validation: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer scope: Namespaced names: plural: crontabs @@ -1090,6 +1381,8 @@ spec: categories: - all ``` +{{% /tab %}} +{{< /tabs >}} And create it: @@ -1134,7 +1427,7 @@ crontabs/my-new-cron-object 3s {{% capture whatsnext %}} -* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions-k8s-io). +* See [CustomResourceDefinition](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1-apiextensions-k8s-io). * Serve [multiple versions](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/) of a CustomResourceDefinition.