Composition creates duplicate API GW integrations

[alexandru-crossover]

What happened?

Composition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: api-route-composition
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: core.test.io/v1
    kind: CompositeApiRoute
  resources:
    - name: integration
      base:
        apiVersion: apigatewayv2.aws.crossplane.io/v1alpha1
        kind: Integration
        spec: {}
      patches:
        - fromFieldPath: "spec.resourceName"
          toFieldPath: "metadata.name"
          policy:
            fromFieldPath: Required
        - fromFieldPath: "spec.apiId"
          toFieldPath: "spec.forProvider.apiId"
        - fromFieldPath: "spec.integrationType"
          toFieldPath: "spec.forProvider.integrationType"
          policy:
            fromFieldPath: Required
        - fromFieldPath: "spec.integrationUri"
          toFieldPath: "spec.forProvider.integrationURI"
        - fromFieldPath: "spec.payloadFormatVersion"
          toFieldPath: "spec.forProvider.payloadFormatVersion"
        - fromFieldPath: "spec.requestParameters"
          toFieldPath: "spec.forProvider.requestParameters"
        - fromFieldPath: "spec.region"
          toFieldPath: "spec.forProvider.region"
          policy:
            fromFieldPath: Required
        - type: ToCompositeFieldPath
          fromFieldPath: "status.atProvider.integrationID"
          toFieldPath: "spec._integrationID"
    - name: route
      base:
        apiVersion: apigatewayv2.aws.crossplane.io/v1alpha1
        kind: Route
        spec:
          forProvider:
            target: integrations/unknown
      patches:
        - fromFieldPath: "spec.resourceName"
          toFieldPath: "metadata.name"
          policy:
            fromFieldPath: Required
        - fromFieldPath: "spec.apiId"
          toFieldPath: "spec.forProvider.apiId"
        - fromFieldPath: "spec.region"
          toFieldPath: "spec.forProvider.region"
          policy:
            fromFieldPath: Required
        - fromFieldPath: "spec.routeKey"
          toFieldPath: "spec.forProvider.routeKey"
          policy:
            fromFieldPath: Required
        - fromFieldPath: "spec._integrationID"
          toFieldPath: "spec.forProvider.target"
          transforms:
            - type: string
              string:
                fmt: "integrations/%s"
        - type: ToCompositeFieldPath
          fromFieldPath: "status.atProvider.routeID"
          toFieldPath: "status.routeID"

Definition:

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: compositeapiroutes.core.test.io
spec:
  defaultCompositionRef:
    name: api-route-composition
  group: core.test.io
  names:
    kind: CompositeApiRoute
    plural: compositeapiroutes
  claimNames:
    kind: ApiRoute
    plural: apiroutes
  versions:
    - name: v1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                resourceName:
                  type: string
                apiId:
                  type: string
                integrationType:
                  type: string
                  enum:
                    - AWS
                    - AWS_PROXY
                    - HTTP
                    - HTTP_PROXY
                    - MOCK
                integrationUri:
                  type: string
                payloadFormatVersion:
                  type: string
                region:
                  type: string
                requestParameters:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                routeKey:
                  type: string
                _integrationID:
                  type: string
              required:
                - integrationType
                - region
                - routeKey
            status:
              type: object
              properties:
                integrationID:
                  type: string
                routeID:
                  type: string

Sometimes, the above composition generates 2 integrations for the same resource. We have identified the creation of duplicate resources from CloudTrail events. We are seeing this only for integrations, routes are always created only once.

This leads to orphan integrations when the composite resource gets deleted, because only one of the 2 integrations is removed.

This is similar to #378 and #590, but we are not copying the annotations.

How can we reproduce it?

It's unclear under what conditions this happens, though it appears to be related to the load from the cluster.

What environment did it happen in?

Crossplane version: 1.1.1
AWS provider version: v0.20.0-rc.0.80.ga4a50b5d
Kubernetes version: 1.19
Kubernetes distribution: EKS

[chlunde]

Might be related: #825

[muvaf]

@chlunde Is this still relevant?

[chlunde]

I believe this is fixed now, please report if you can reproduce this on the latest release, thanks!