CRD system for OpenShift resources not interpreted as expected

[mjovanovic0]

Hi all,

I have the following structure:

├── base
│   ├── 001-image-stream.yaml
│   ├── crd
│   │   ├── imagestream.json
│   └── kustomization.yaml
└── test
    ├── 001-image-stream.yaml
    └── kustomization.yaml

base/001-image-stream.yaml:

apiVersion: v1
kind: ImageStream
metadata:
  labels:
    app: app
  name: image-name
spec:
  lookupPolicy:
    local: false
  tags:
    - from:
        kind: DockerImage
        name: {to-be-replaced}
      name: image-name
      referencePolicy:
        type: Source

base/kustomization.yaml:

crds:
  - crd/imagestream.json
resources:
  - 001-image-stream.yaml

test/001-image-stream.yaml:

apiVersion: v1
kind: ImageStream
metadata:
  name: image-name
spec:
  tags:
    - name: image-name
      from:
        name: docker.io/hello-world

base/kustomization.yaml:

namespace: test-namespace-name
namePrefix: test-
resources:
  - ../base
patchesStrategicMerge:
  - 001-image-stream.yaml

base/crd/imagestream.json: (Extracted and coverted parts from OpenShift Swagger API)

{
  "github.com/openshift/api/image/v1.ImageStream": {
    "Schema": {
      "description": "ImageStream stores a mapping of tags to images, metadata overrides that are applied when images are tagged in a stream, and an optional reference to a container image repository on a registry.",
      "properties": {
        "apiVersion": {
          "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources",
          "type": "string"
        },
        "kind": {
          "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds",
          "type": "string"
        },
        "metadata": {
          "description": "Standard object's metadata.",
          "$ref": "k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta"
        },
        "spec": {
          "description": "Spec describes the desired state of this stream",
          "$ref": "github.com/openshift/api/image/v1.ImageStreamSpec"
        },
        "status": {
          "description": "Status describes the current state of this stream",
          "$ref": "github.com/openshift/api/image/v1.ImageStreamStatus"
        }
      }
    },
    "x-kubernetes-group-version-kind": [
      {
        "group": "",
        "kind": "ImageStream",
        "version": "v1"
      },
      {
        "group": "image.openshift.io",
        "kind": "ImageStream",
        "version": "v1"
      }
    ]
  },
  "github.com/openshift/api/image/v1.ImageStreamSpec": {
    "Schema": {
      "description": "ImageStreamSpec represents options for ImageStreams.",
      "properties": {
        "dockerImageRepository": {
          "description": "dockerImageRepository is optional, if specified this stream is backed by a container repository on this server Deprecated: This field is deprecated as of v3.7 and will be removed in a future release. Specify the source for the tags to be imported in each tag via the spec.tags.from reference instead.",
          "type": "string"
        },
        "lookupPolicy": {
          "description": "lookupPolicy controls how other resources reference images within this namespace.",
          "$ref": "github.com/openshift/api/image/v1.ImageLookupPolicy"
        },
        "tags": {
          "description": "tags map arbitrary string values to specific image locators",
          "type": "array",
          "items": {
            "$ref": "github.com/openshift/api/image/v1.TagReference"
          },
          "x-kubernetes-patch-merge-key": "name",
          "x-kubernetes-patch-strategy": "merge"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.ImageStreamStatus": {
    "Schema": {
      "description": "ImageStreamStatus contains information about the state of this image stream.",
      "required": [
        "dockerImageRepository"
      ],
      "properties": {
        "dockerImageRepository": {
          "description": "DockerImageRepository represents the effective location this stream may be accessed at. May be empty until the server determines where the repository is located",
          "type": "string"
        },
        "publicDockerImageRepository": {
          "description": "PublicDockerImageRepository represents the public location from where the image can be pulled outside the cluster. This field may be empty if the administrator has not exposed the integrated registry externally.",
          "type": "string"
        },
        "tags": {
          "description": "Tags are a historical record of images associated with each tag. The first entry in the TagEvent array is the currently tagged image.",
          "type": "array",
          "items": {
            "$ref": "github.com/openshift/api/image/v1.NamedTagEventList"
          },
          "x-kubernetes-patch-merge-key": "tag",
          "x-kubernetes-patch-strategy": "merge"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.ImageLookupPolicy": {
    "Schema": {
      "description": "ImageLookupPolicy describes how an image stream can be used to override the image references used by pods, builds, and other resources in a namespace.",
      "required": [
        "local"
      ],
      "properties": {
        "local": {
          "description": "local will change the docker short image references (like \"mysql\" or \"php:latest\") on objects in this namespace to the image ID whenever they match this image stream, instead of reaching out to a remote registry. The name will be fully qualified to an image ID if found. The tag's referencePolicy is taken into account on the replaced value. Only works within the current namespace.",
          "type": "boolean"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.TagReference": {
    "Schema": {
      "description": "TagReference specifies optional annotations for images using this tag and an optional reference to an ImageStreamTag, ImageStreamImage, or DockerImage this tag should track.",
      "required": [
        "name"
      ],
      "properties": {
        "annotations": {
          "description": "Optional; if specified, annotations that are applied to images retrieved via ImageStreamTags.",
          "type": "object",
          "additionalProperties": {
            "type": "string"
          }
        },
        "from": {
          "description": "Optional; if specified, a reference to another image that this tag should point to. Valid values are ImageStreamTag, ImageStreamImage, and DockerImage.  ImageStreamTag references can only reference a tag within this same ImageStream.",
          "$ref": "k8s.io/api/core/v1.ObjectReference"
        },
        "generation": {
          "description": "Generation is a counter that tracks mutations to the spec tag (user intent). When a tag reference is changed the generation is set to match the current stream generation (which is incremented every time spec is changed). Other processes in the system like the image importer observe that the generation of spec tag is newer than the generation recorded in the status and use that as a trigger to import the newest remote tag. To trigger a new import, clients may set this value to zero which will reset the generation to the latest stream generation. Legacy clients will send this value as nil which will be merged with the current tag generation.",
          "type": "integer",
          "format": "int64"
        },
        "importPolicy": {
          "description": "ImportPolicy is information that controls how images may be imported by the server.",
          "$ref": "github.com/openshift/api/image/v1.TagImportPolicy"
        },
        "name": {
          "description": "Name of the tag",
          "type": "string"
        },
        "reference": {
          "description": "Reference states if the tag will be imported. Default value is false, which means the tag will be imported.",
          "type": "boolean"
        },
        "referencePolicy": {
          "description": "ReferencePolicy defines how other components should consume the image.",
          "$ref": "github.com/openshift/api/image/v1.TagReferencePolicy"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.NamedTagEventList": {
    "Schema": {
      "description": "NamedTagEventList relates a tag to its image history.",
      "required": [
        "tag",
        "items"
      ],
      "properties": {
        "conditions": {
          "description": "Conditions is an array of conditions that apply to the tag event list.",
          "type": "array",
          "items": {
            "$ref": "github.com/openshift/api/image/v1.TagEventCondition"
          }
        },
        "items": {
          "description": "Standard object's metadata.",
          "type": "array",
          "items": {
            "$ref": "github.com/openshift/api/image/v1.TagEvent"
          }
        },
        "tag": {
          "description": "Tag is the tag for which the history is recorded",
          "type": "string"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.TagImportPolicy": {
    "Schema": {
      "description": "TagImportPolicy controls how images related to this tag will be imported.",
      "properties": {
        "insecure": {
          "description": "Insecure is true if the server may bypass certificate verification or connect directly over HTTP during image import.",
          "type": "boolean"
        },
        "scheduled": {
          "description": "Scheduled indicates to the server that this tag should be periodically checked to ensure it is up to date, and imported",
          "type": "boolean"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.TagReferencePolicy": {
    "Schema": {
      "description": "TagReferencePolicy describes how pull-specs for images in this image stream tag are generated when image change triggers in deployment configs or builds are resolved. This allows the image stream author to control how images are accessed.",
      "required": [
        "type"
      ],
      "properties": {
        "type": {
          "description": "Type determines how the image pull spec should be transformed when the image stream tag is used in deployment config triggers or new builds. The default value is `Source`, indicating the original location of the image should be used (if imported). The user may also specify `Local`, indicating that the pull spec should point to the integrated container image registry and leverage the registry's ability to proxy the pull to an upstream registry. `Local` allows the credentials used to pull this image to be managed from the image stream's namespace, so others on the platform can access a remote image but have no access to the remote secret. It also allows the image layers to be mirrored into the local registry which the images can still be pulled even if the upstream registry is unavailable.",
          "type": "string"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.TagEventCondition": {
    "Schema": {
      "description": "TagEventCondition contains condition information for a tag event.",
      "required": [
        "type",
        "status",
        "generation"
      ],
      "properties": {
        "generation": {
          "description": "Generation is the spec tag generation that this status corresponds to",
          "type": "integer",
          "format": "int64"
        },
        "lastTransitionTime": {
          "description": "LastTransitionTIme is the time the condition transitioned from one status to another.",
          "$ref": "k8s.io/apimachinery/pkg/apis/meta/v1.Time"
        },
        "message": {
          "description": "Message is a human readable description of the details about last transition, complementing reason.",
          "type": "string"
        },
        "reason": {
          "description": "Reason is a brief machine readable explanation for the condition's last transition.",
          "type": "string"
        },
        "status": {
          "description": "Status of the condition, one of True, False, Unknown.",
          "type": "string"
        },
        "type": {
          "description": "Type of tag event condition, currently only ImportSuccess",
          "type": "string"
        }
      }
    }
  },
  "github.com/openshift/api/image/v1.TagEvent": {
    "Schema": {
      "description": "TagEvent is used by ImageStreamStatus to keep a historical record of images associated with a tag.",
      "required": [
        "created",
        "dockerImageReference",
        "image",
        "generation"
      ],
      "properties": {
        "created": {
          "description": "Created holds the time the TagEvent was created",
          "$ref": "k8s.io/apimachinery/pkg/apis/meta/v1.Time"
        },
        "dockerImageReference": {
          "description": "DockerImageReference is the string that can be used to pull this image",
          "type": "string"
        },
        "generation": {
          "description": "Generation is the spec tag generation that resulted in this tag being updated",
          "type": "integer",
          "format": "int64"
        },
        "image": {
          "description": "Image is the image",
          "type": "string"
        }
      }
    }
  }
}

And when I execute:

kustomize build test

This is output:

apiVersion: v1
kind: ImageStream
metadata:
  labels:
    app: app
  name: test-image-name
  namespace: test-namespace-name
spec:
  lookupPolicy:
    local: false
  tags:
  - from:
      name: docker.io/hello-world
    name: image-name

While expected output is:

apiVersion: v1
kind: ImageStream
metadata:
  labels:
    app: app
  name: test-image-name
  namespace: test-namespace-name
spec:
  lookupPolicy:
    local: false
  tags:
    - from:
        kind: DockerImage
        name: docker.io/hello-world
      name: image-name
      referencePolicy:
        type: Source

Now if I understand CRD mechanism in Kustomize, for him to know how to merge in CRD definitions needs to be annotations:

• "x-kubernetes-patch-merge-key": "tag"
• "x-kubernetes-patch-strategy": "merge"

Which are defined in imagestream.json CRD.

Can someone say me what I'm doing wrong or if this is not supported or is it a bug?

Thanks

[jbrette]

@mjovanovic0 The is linked to the fact that JMP is invoked instead of StrategicMergePatch. Have a look at #1510 #1510.
Have a look at the registry mechanism using scheme.register for Argo Rollout CRD. I'll try something similar with OpenShit CRD, Istio

[jbrette]

@mjovanovic0: Have a look at that environment; CRDRegister

By using the kustomize v3 external go plugin, you can kind of solve your issue

kustomize build overlay/ --enable_alpha_plugins

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  labels:
    app: app
  name: test-image-name
  namespace: test-namespace-name
spec:
  lookupPolicy:
    local: false
  tags:
  - from:
      kind: DockerImage
      name: docker.io/hello-world
    name: image-name
    referencePolicy:
      type: Source

We will most likely have to write to KEP to do that efficiently

/cc @monopole @Liujingfang1 @liggitt @jessesuen

[mjovanovic0]

I'm having trouble compile-ing OpenShiftCRDRegister.so for Kustomize v3.1.0 - I get error:

Error: accumulating resources: recursed accumulation of path '../base': 
plugin
  /Users/mjovanovic/.config/kustomize/plugin/api.openshift.io/v1/openshiftcrdregister/OpenShiftCRD
Register fails to load: 
  plugin.Open("/Users/mjovanovic/.config/kustomize/plugin/api.openshift.io/v1/openshiftcrdregister/OpenShiftCRDRegister"):
plugin was built with a different version of package internal/cpu

Are you able to instruct what I need to do to compile it properly?

[jbrette]

@mjovanovic0 Thanks, one more section to put in the KEP: How to compile a plugin which go module dependencies are overlapping with the default kustomize modules

I just realized that if I try to build outside my GOPATH, it is does not work for me either, and I have the same issue that you listed.

But if you build from your GOPATH as described bellow, it seems to work:

myuserid@airship:~$ echo $GOPATH
/home/myuserid
myuserid@airship:~$ cd src
myuserid@airship:~/src$ cd sigs.k8s.io/
myuserid@airship:~/src/sigs.k8s.io$ git clone -b armadacrd https://github.com/keleustes/kustomize
Cloning into 'kustomize'...
remote: Enumerating objects: 439, done.
remote: Counting objects: 100% (439/439), done.
remote: Compressing objects: 100% (217/217), done.
remote: Total 25845 (delta 255), reused 373 (delta 214), pack-reused 25406
Receiving objects: 100% (25845/25845), 25.59 MiB | 1.67 MiB/s, done.
Resolving deltas: 100% (12726/12726), done.
Checking connectivity... done.
myuserid@airship:~/src/sigs.k8s.io$ cd kustomize

myuserid@airship:~/src/sigs.k8s.io/kustomize$ make install build-plugins
go install /home/myuserid/src/sigs.k8s.io/kustomize/cmd/kustomize
./plugin/buildPlugins.sh /home/myuserid
Generating linkable plugins...
plugin/armada.airshipit.org/v1alpha1/armadacrdregister/ArmadaCRDRegister
plugin/rollouts.argoproj.io/v1alpha1/rolloutcrdregister/RolloutCRDRegister
plugin/api.openshift.io/v1/openshiftcrdregister/OpenShiftCRDRegister
plugin/argo.argoproj.io/v1alpha1/workflowcrdregister/WorkflowCRDRegister
plugin/networking.istio.io/v1alpha3/networkingcrdregister/NetworkingCRDRegister
All done.

myuserid@airship:~/src/sigs.k8s.io/kustomize$ cd examples/crds/openshift-api/
myuserid@airship:~/src/sigs.k8s.io/kustomize/examples/crds/openshift-api$ which kustomize
/home/myuserid/bin/kustomize
myuserid@airship:~/src/sigs.k8s.io/kustomize/examples/crds/openshift-api$ ls -lt /home/myuserid/.config/kustomize/plugin/
total 20
drwxr-xr-x 3 myuserid alemax 4096 Sep 14 11:39 networking.istio.io
drwxr-xr-x 3 myuserid alemax 4096 Sep 14 11:39 argo.argoproj.io
drwxr-xr-x 3 myuserid alemax 4096 Sep 14 11:39 api.openshift.io
drwxr-xr-x 3 myuserid alemax 4096 Sep 14 11:39 rollouts.argoproj.io
drwxr-xr-x 3 myuserid alemax 4096 Sep 14 11:39 armada.airshipit.org

myuserid@airship:~/src/sigs.k8s.io/kustomize/examples/crds/openshift-api$ kustomize build overlay/ --enable_alpha_plugins
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  labels:
    app: app
  name: test-image-name
  namespace: test-namespace-name
spec:
  lookupPolicy:
    local: false
  tags:
  - from:
      kind: DockerImage
      name: docker.io/hello-world
    name: image-name
    referencePolicy:
      type: Source

[mjovanovic0]

Awesome man. Thank you very much :)
Works like a charm :)