Add OpenSearch Ingestion Pipeline Support

[blakeromano]

Description of your changes

Fixes #1446

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

I took the example and modified line 26 to change it to my own AWS Account ID. I was able to validate the command:

make e2e UPTEST_EXAMPLE_LIST=examples/osis/v1beta1/pipeline.yaml

---
apiVersion: osis.aws.upbound.io/v1beta1
kind: Pipeline
metadata:
  annotations:
    crossplane.io/external-create-pending: "2024-09-12T16:23:43Z"
    crossplane.io/external-create-succeeded: "2024-09-12T16:23:43Z"
    crossplane.io/external-name: example
    crossplane.io/paused: "false"
    meta.upbound.io/example-id: osis/v1beta1/pipeline
    upjet.upbound.io/test: "true"
    uptest-old-id: example
  creationTimestamp: "2024-09-12T16:23:42Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-09-12T16:29:49Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 3
  labels:
    testing.upbound.io/example-name: example
  name: example
  resourceVersion: "2581"
  uid: 351b183c-0dac-430e-ba5f-90aec59e402e
spec:
  deletionPolicy: Delete
  forProvider:
    maxUnits: 1
    minUnits: 1
    pipelineConfigurationBody: |
      version: "2"
      example-pipeline:
          source:
              http:
                  path: "/example"
          sink:
          - s3:
              aws:
                  sts_role_arn: "arn:aws:iam::ACCOUNT_ID:role/example-osi-pipeline-role"
                  region: "us-west-1"
              bucket: "example-upbound-osis-pipeline-example-blake"
              threshold:
                  event_collect_timeout: "60s"
              codec:
                  ndjson:
    pipelineName: example
    region: us-west-1
    tags:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:
    id: example
    ingestEndpointUrls:
    - example-7myxz74qalrcrk5ilmygdzco2u.us-west-1.osis.amazonaws.com
    maxUnits: 1
    minUnits: 1
    pipelineArn: arn:aws:osis:us-west-1:ACCOUNT_ID:pipeline/example
    pipelineConfigurationBody: |
      version: "2"
      example-pipeline:
          source:
              http:
                  path: "/example"
          sink:
          - s3:
              aws:
                  sts_role_arn: "arn:aws:iam::ACCOUNT_ID:role/example-osi-pipeline-role"
                  region: "us-west-1"
              bucket: "example-upbound-osis-pipeline-example-blake"
              threshold:
                  event_collect_timeout: "60s"
              codec:
                  ndjson:
    pipelineName: example
    tags:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
    tagsAll:
      crossplane-kind: pipeline.osis.aws.upbound.io
      crossplane-name: example
      crossplane-providerconfig: default
  conditions:
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: UpToDate
    status: "True"
    type: Test
  - lastTransitionTime: "2024-09-12T16:29:42Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[blakeromano]

So looking at logs https://github.com/crossplane-contrib/provider-upjet-aws/actions/runs/10835115001/job/30065908399 it seems like it actually does get ready but maybe I am missing something in terms of it not working or maybe it is just timing out too soon?

[mergenci]

First, I'm not an expert on Uptest logs 🙂 Second, the following line tells me that the Pipeline resource is not ready:

2024-09-12T17:13:53.1182356Z     case.go:363: command "${KUBECTL} wait pipeline.osis.aws.upbound.io/example --for=condition=Test --timeout 10s" exceeded 7 sec timeout, context deadline exceeded

When I check the resource output by grepping osis.aws.upbound.io in the Uptest log file I downloaded, I see that the resource is not ready:

- apiVersion: osis.aws.upbound.io/v1beta1
  kind: Pipeline
  metadata:
    annotations:
      crossplane.io/external-create-pending: "2024-09-12T17:13:44Z"
      crossplane.io/external-create-succeeded: "2024-09-12T17:13:44Z"
      crossplane.io/external-name: example
      meta.upbound.io/example-id: osis/v1beta1/pipeline
      upjet.upbound.io/test: "true"
    creationTimestamp: "2024-09-12T16:53:53Z"
    finalizers:
    - finalizer.managedresource.crossplane.io
    generation: 2
    labels:
      testing.upbound.io/example-name: example
    name: example
    resourceVersion: "4358"
    uid: 8b15e491-dd01-4096-9084-b373f7027be0
  spec:
    deletionPolicy: Delete
    forProvider:
      maxUnits: 1
      minUnits: 1
      pipelineConfigurationBody: |
        version: "2"
        example-pipeline:
            source:
                http:
                    path: "/example"
            sink:
            - s3:
                aws:
                    sts_role_arn: "arn:aws:iam::609897127049:role/example-osi-pipeline-role"
                    region: "us-west-1"
                bucket: "example-upbound-osis-pipeline-example"
                threshold:
                    event_collect_timeout: "60s"
                codec:
                    ndjson:
      pipelineName: example
      region: us-west-1
      tags:
        crossplane-kind: pipeline.osis.aws.upbound.io
        crossplane-name: example
        crossplane-providerconfig: default
    initProvider: {}
    managementPolicies:
    - '*'
    providerConfigRef:
      name: default
  status:
    atProvider: {}
    conditions:
    - lastTransitionTime: "2024-09-12T16:53:54Z"
      reason: Creating
      status: "False"
      type: Ready
    - lastTransitionTime: "2024-09-12T16:53:54Z"
      reason: ReconcileSuccess
      status: "True"
      type: Synced
    - lastTransitionTime: "2024-09-12T16:53:55Z"
      reason: Success
      status: "True"
      type: LastAsyncOperation

[blakeromano]

@mergenci sorry missed that 🤦‍♂️ what I am still confused by is why it is still in a creating state. I know OSI pipelines do take a long time to come up but I am able to get this same uptest command work locally on my AWS environment by just swapping the AWS Account ID in the pipeline body. Like I said I wonder if there we are just timing out on the test and it isn't an actual failure.

[turkenf]

Please rebase the PR and trigger again to get this fix, so that we can see if there is an error.

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[blakeromano]

It looks like it is failing with this error:

AccessDeniedException: User: arn:aws:iam::153891904029:user/official-provider-testing is not authorized to perform: osis:CreatePipeline on resource: arn:aws:osis:us-west-1:153891904029:pipeline/example

So we may need to tweak the permissioning on the user to allow the test to pass.

[jeanduplessis]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[jeanduplessis]

@blakeromano I've added the following policies to the test user:

• AmazonOpenSearchIngestionFullAccess
• AmazonOpenSearchServiceFullAccess

[blakeromano]

@jeanduplessis I still see same error on Uptest. Is there an SCP or Tag Policy or anything that is giving it the explicit deny? I just can't see anything from my POV on why this would be failing...

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[turkenf]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[turkenf]

Thank you for your patient and effort in this PR @blakeromano. I left a few comments for you to consider.

[blakeromano]

/test-examples="examples/osis/v1beta1/pipeline.yaml"

[turkenf]

Because of this issue we could not convert the singleton list to embedded object in the osis Pipeline resource. This issue is not currently blocking and will be fixed with new versions of the resource.

[turkenf]

Thank you @blakeromano, LGTM.