BWC Tests are failing on main

[cwperks]

See a recent example of BWC test failure here: https://github.com/opensearch-project/security/actions/runs/3372773855/jobs/5611052811

This is blocking multiple PRs.

* What went wrong:
Execution failed for task ':securityBwcCluster#mixedClusterTask'.
> `cluster{::securityBwcCluster0}` failed to wait for cluster health yellow after 40 SECONDS
    IO error while waiting cluster
    503 Service Unavailable

I will add more information to the issue as its discovered.

[cwperks]

Could be a transient issue until new artifacts are available. The latest distribution build is not available: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/6385/linux/x64/tar/builds/opensearch/plugins/opensearch-security-2.4.0.0.zip (Access denied at the time of this writing)

[peternied]

The latest distribution build is not available

Can you file a bug on the OpenSearch-build repository, those URLs should always have a healthy target AFAIK. If we are using the incorrect URL we should change it so we keep our main builds stable

[cwperks]

securityBwcCluster0-0 is the node (1 out of 3) that is upgraded in the mixedClusterTask and the node is failing to connect to the cluster.

2 / 3 of the nodes on 2.4.0

1 node upgraded from 2.4.0 -> 3.0.0 before the task starts

[cwperks]

I don't think this is related to any change on the security repo. I created a branch and went back to a commit where BWC tests were working and now they are reported to be failing.

[cwperks]

@VachaShah The security repo's bwcTests started failing yesterday and I am trying to determine why and how to get the check working as a number of PRs are affected. On commits that bwcTests worked on previously, the check is now failing so it looks like the check is failing due to a change outside of this repo. Do you have any ideas why the bwcTests started failing? The failing task is mixedClusterTask

[peternied]

Looks like we have two issues are tracking the same problem:

• BWC Tests are failing on main #2221
• [BUG] Errors/Broken operations during rolling upgrade of clusters from 1.3 to 2.0 #2228

[Update] These issues are not related

[peternied]

@cwperks Seems like something changed in OpenSearch causing this failure - can you create an issue in that repo to get traction?

[cwperks]

@peternied Filed an issue in core with the information I currently have:opensearch-project/OpenSearch#5076

[cwperks]

[Triage] Closing this issue. The automation that broke when creating the next version manifest can be tracked here: opensearch-project/OpenSearch#5091

[peternied]

Reopening, this is still broken [1] on a build from 17 minutes ago, until this is fixed in main we should keep this issue open.

[1] https://github.com/opensearch-project/security/actions/runs/3422233118/jobs/5699382046

[peternied]

@DarshitChanpura Were you looking into this issue/related issues?

[DarshitChanpura]

@DarshitChanpura Were you looking into this issue/related issues?

Following up with the team: opensearch-project/opensearch-build#2868

[DarshitChanpura]

#2253 fixes the BWC test CI

[DarshitChanpura]

Closing this as #2253 is merged