Propose better integration test that exercise real cloud plugins

[anasalkouz]

Is your feature request related to a problem? Please describe.
We need to enhance our integration tests to exercise the usage of real cloud plugins Like Azure, Google Cloud ..etc. This is was clearly needed after we received the repository-azure plugin issue as regression if OpenSearch 1.2 release.

1. Define account creds in Jenkins secret store. Export same in jenkins file as environment variables so that tests can consume/use them.
2. Create different cloud providers account.
3. Create jenkins setup locally to verify tests works.
4. Add new account creds in Jenkins secret store for prod jenkins.

Additional context
More details: #1734 #1740

[dreamer-89]

Looking into it.

[reta]

@dreamer-89 the XxxThirdPartyTests in the repository plugins are runnable against real cloud storages (Azure / S3 / GCS), only a matter of system properties, but there are no CI integrations + test accounts at the moment for that.

[dreamer-89]

@reta: Thank you for sharing this.

[dreamer-89]

Tests

Checked different repository actions and corresponding integration tests already exists for all supported repository types here. I have verified that providing aws credentails as enviornment variables; the tests were able to use actual S3 end-point and perform different actions.

Infra Setup

The actual choice of infra to store credentials (Jenkins Credentials or Github secrets) depends on Issue 136; which looks like is weighing in towards current Jenkins set up we have. I am biased towards GHA workflows for multitude of reasons already mentioned in 136 :)

[peterzhuamazon]

Meeting notes:

• @dreamer-89 wants to check whether we can have a system to connect to third party cloud and test.
• Seems like we need to do external communication to Azure and Google Cloud.
• A one-pager will be prepared to get some high level understanding of what is needed, and what port to open and what service to use, etc.
• Security Practices

[dreamer-89]

Choice of host for running test

AWS Ec2 via Jenkins

• Pros
  • Faster
  • Existing gradle check already running on Jenkins agents (EC2 host), no extra setup required other than feeding credentials
• Cons
  • May need to get AppSec approval and it involves communication with external cloud providers

Azure via Github Actions

• Pros
  • No need to go through AppSec
  • No need to maintain infra for running the test
• Cons
  • Need to maintain a separate GHA for repository tests
  • Carve out repository tests as separate test.

[dreamer-89]

Waiting on Issue 136 for choice of infra (Jenkins Vs CodeBuild Vs GHA) and review from infra team on requirements.

[owaiskazi19]

Hey @dreamer-89! Issue 136 is closed. Jenkinsfile is added in OpenSearch for gradle check. We are going with public Jenkins.

[dreamer-89]

Listing down pre and post test coverage with introduction of AWS credentials for running repository-s3 plugin tests and get feedback.

plugin/repository-s3

Today, tests use mocked servers (S3HttpHandler, Minio server) providing S3 compatible APIs for running the S3 integration tests. repository-s3 gradle uses useFixture var (true when permanent creds are provided) to govern whether to use mocked servers or actual S3 service.

Below are the set of tasks which runs today as part of CI gradle check.

• yamlRestTestMinio. Runs 20_repository_permanent_credentials test using minio server.
• yamlRestTestECS. Runs 50_repository_ecs_credentials using s3-fixture-with-ecs (S3HttpHandler mock s3 server).
• yamlRestTest. Runs 20_repository_permanent_credentials, 30_repository_temporary_credentials, 40_repository_ec2_credentials using respective fixtures.
• testRepositoryCreds. Runs RepositoryCredentialsTests
• s3ThirdPartyTest. runs S3RepositoryThirdPartyTests
• internalClusterTest. runs S3BlobStoreRepositoryTests
• test. runs all tests under test package.

When permanent credentials are provided:

1. s3ThirdPartyTest tests and 20_repository_permanent_credentials in yamlRestTest uses external S3 end-points.
2. Tasks yamlRestTestMinio and yamlRestTestECS are skipped.
3. yamlRestTest only runs tests in 20_repository_permanent_credentials. This (and 2 above) are fine and probably intentional as tests are supposed to verify integration with AWS S3 on basic operations and not the underlying authentication. Though, running these missing test/task should be straightforward, by using a new var here (e.g. useFixtureECS) to govern whether to run test fixtures or use actual service end-points for temporary, ec2 and ecs cred based yaml tests.

[Edit]: Just verified Ec2/ECS credentails (InstanceProfileCredentialsProvider) comes after basic key/secret credentials in cred provider chain; which means ec2/ecs yaml tests will use permanent creds only.

@reta @andrross : WDYT ?

[reta]

@dreamer-89 thanks for the summary, afaik, the most interesting tests for us are xxxThirdPartyTest (for S3, s3ThirdPartyTest but others have the analogous test suites), those exercise end-2-end flows and accept external endpoints.

[dreamer-89]

@dreamer-89 thanks for the summary, afaik, the most interesting tests for us are xxxThirdPartyTest (for S3, s3ThirdPartyTest but others have the analogous test suites), those exercise end-2-end flows and accept external endpoints.

Thanks @reta for the reply. Yes, I verified that s3ThirdPartyTest uses actual external service end-points. Verified that tests create relevant artefacts under provided S3 bucket and base path.

[dreamer-89]

Jenkins gradle job is updated to export permanent aws account credentials. This results in s3ThirdPartyTest tests (and yamlRestTest 20_repository_permanent_credentials) using actual s3 end-points.

[dreamer-89]

Jenkins gradle job is updated with Azure credentials. Gradle check is now running repository-azure tests using actual azure end-points.

[dreamer-89]

Closing this issue and creating new one to track GCP and Azure integration and enabling back Ec2/ECS credentials related repository-s3 integration tests discussed in previous comment here