[Validation Blocking][Bug]: Integration Tests are failing on Jenkins

[owaiskazi19]

Describe the bug

Integ tests are failing on jenkins for all the plugins.
Report:

2021-11-11 19:56:44 ERROR    | alerting             | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | alerting             | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | anomaly-detection    | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | anomaly-detection    | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | asynchronous-search  | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | asynchronous-search  | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | dashboards-reports   | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | index-management     | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | index-management     | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | k-NN                 | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | k-NN                 | without-security     | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | sql                  | with-security        | FAIL  |    1 |
2021-11-11 19:56:44 ERROR    | sql                  | without-security     | FAIL  |    1 |

To reproduce

Run integ-test job on Jenkins with the right params.

Expected behavior

Integration tests should run successfully.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[owaiskazi19]

Alerting:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':alerting-core:compileKotlin'.
> Could not resolve all files for configuration ':alerting-core:compileClasspath'.
   > Could not find org.opensearch:common-utils:1.2.0.0.
     Searched in the following locations:
       - file:/home/ec2-user/.m2/repository/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://jcenter.bintray.com/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
     Required by:
         project :alerting-core

[owaiskazi19]

Asynchronous Search:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:common-utils:1.2.0.0.
     Searched in the following locations:
       - file:/home/ec2-user/.m2/repository/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://jcenter.bintray.com/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
     Required by:
         project :

[owaiskazi19]

Index Management:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileKotlin'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:notification:1.2.0.0.
     Searched in the following locations:
       - https://jcenter.bintray.com/org/opensearch/notification/1.2.0.0/notification-1.2.0.0.pom
       - file:/home/ec2-user/.m2/repository/org/opensearch/notification/1.2.0.0/notification-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/notification/1.2.0.0/notification-1.2.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch:common-utils:1.2.0.0.
     Searched in the following locations:
       - https://jcenter.bintray.com/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - file:/home/ec2-user/.m2/repository/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch:opensearch-job-scheduler-spi:1.2.0.0.
     Searched in the following locations:
       - https://jcenter.bintray.com/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - file:/home/ec2-user/.m2/repository/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
     Required by:
         project :

[owaiskazi19]

K-NN:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> Could not resolve all files for configuration ':opensearch_distro_extracted_testclusters--integTest-0-1.2.0-'.
   > Could not resolve opensearch-distribution:opensearch:1.2.0.
     Required by:
         project :
      > Could not resolve opensearch-distribution:opensearch:1.2.0.
         > Could not get resource 'https://artifacts.opensearch.org/releases/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/releases/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden
      > Could not resolve opensearch-distribution:opensearch:1.2.0.
         > Could not get resource 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden

[owaiskazi19]

Anomaly Detection:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:common-utils:1.2.0.0.
     Searched in the following locations:
       - file:/home/ec2-user/.m2/repository/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://jcenter.bintray.com/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch:opensearch-job-scheduler-spi:1.2.0.0.
     Searched in the following locations:
       - file:/home/ec2-user/.m2/repository/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://repo.maven.apache.org/maven2/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://plugins.gradle.org/m2/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://jcenter.bintray.com/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
     Required by:
         project :

[owaiskazi19]

SQL:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integ-test:integTest'.
> Could not resolve all files for configuration ':integ-test:opensearch_distro_extracted_testclusters-integ-test-integTest-0-1.2.0-'.
   > Could not resolve opensearch-distribution:opensearch:1.2.0.
     Required by:
         project :integ-test
      > Could not resolve opensearch-distribution:opensearch:1.2.0.
         > Could not get resource 'https://artifacts.opensearch.org/releases/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/releases/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden
      > Could not resolve opensearch-distribution:opensearch:1.2.0.
         > Could not get resource 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.2.0/opensearch-min-1.2.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden

[owaiskazi19]

dashboard-reports:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileKotlin'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.opensearch:common-utils:1.2.0.0.
     Searched in the following locations:
       - https://jcenter.bintray.com/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - file:/home/ec2-user/.m2/repository/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/common-utils/1.2.0.0/common-utils-1.2.0.0.pom
     Required by:
         project :
   > Could not find org.opensearch:opensearch-job-scheduler-spi:1.2.0.0.
     Searched in the following locations:
       - https://jcenter.bintray.com/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - file:/home/ec2-user/.m2/repository/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
       - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch-job-scheduler-spi/1.2.0.0/opensearch-job-scheduler-spi-1.2.0.0.pom
     Required by:
         project :

[saratvemulapalli]

Looking at this, looks like dependency installer (https://github.com/opensearch-project/opensearch-build/blob/main/src/test_workflow/dependency_installer.py) is unable to pull the right dependencies for plugins to run the integration tests.

[dblock]

What's in the dependency installer pull log? What is it bringing to maven local?

[dblock]

Looks like the build is missing .pom files, the only maven artifacts downloaded are .jar(s):

2021-11-11 19:45:45 INFO     Downloading to /home/ec2-user/.m2/repository ...
2021-11-11 19:45:45 INFO     Downloading https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/962/linux/x64/builds/maven/org/opensearch/common-utils/common-utils-1.2.0.0.jar into /home/ec2-user/.m2/repository/org/opensearch/common-utils/common-utils-1.2.0.0.jar ...
2021-11-11 19:45:45 INFO     Downloading https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/962/linux/x64/builds/maven/org/opensearch/common-utils/common-utils-1.2.0.0-javadoc.jar into /home/ec2-user/.m2/repository/org/opensearch/common-utils/common-utils-1.2.0.0-javadoc.jar ...
2021-11-11 19:45:45 INFO     Downloading https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/962/linux/x64/builds/maven/org/opensearch/common-utils/common-utils-1.2.0.0-sources.jar into /home/ec2-user/.m2/repository/org/opensearch/common-utils/common-utils-1.2.0.0-sources.jar ...

[bbarani]

Closing this issue as it was resolved and we have already released 1.2.0.