Add support for automated backwards-compatibility tests for plugins

[saratvemulapalli]

OpenSearch has bwc tests which are enabled via gradle plugin.
We'd like to extend the same plugin to work for OpenSearch plugins.
This will solve the problem of automating bwc tests for plugins to verify functionality against older version of plugins and run these checks as integration tests in CI.

We decided to extend the existing framework because:
a. We would have only one place to manage the cluster for testing and not duplicating code in multiple places, extended via gradle plugin. (Probably eventually we could stub this out of OpenSearch repo).
b. Plugins already have a dependency on opensearch gradle plugin, so this is not a new dependency without which the plugin cannot build.
c. This will allow plugins to run bwc in their CI's without external solution to manage the cluster upgrade.
d. Most of the framework already exists and we do not need to reinvent it.

[saratvemulapalli]

Updating this issue.
I took a stab at this and was able setup OpenDistro cluster with its plugin within the OpenSearch plugin repository.
The challenge is the frame work today doesn't allow different versions of the plugins to be installed so it tries to install OpenDistro plugin on OpenSearch and it fails. This needs a framework change to OpenSearch to support it.

Failure:

> Task :sarat1 FAILED
Exec output and error:
| Output for ./bin/opensearch-plugin:-> Installing file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Downloading file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Failed installing file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Rolling back file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Rolled back file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| Exception in thread "main" java.lang.IllegalArgumentException: property [opensearch.version] is missing for plugin [opendistro-job-scheduler]
|       at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:194)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:797)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:856)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:263)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
|       at org.opensearch.cli.Command.main(Command.java:103)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

=== Standard output of node `node{::adMixedCluster-1}` ===

Here is a sample(hacky) way to set it up:

testClusters {
    adMixedCluster {
        testDistribution = "ARCHIVE"
	    versions = ["7.10.2","1.0.0-rc1"]
	    numberOfNodes = 3
        setting 'http.content_type.required', 'true'
        /*plugin(provider(new Callable<RegularFile>(){
            @Override
            RegularFile call() throws Exception {
                return new RegularFile() {
                    @Override
                    File getAsFile() {
                        return fileTree("src/test/resources/job-scheduler/1.13.2").getSingleFile()
                    }
                }
            }
        }))
        plugin(provider(new Callable<RegularFile>(){
            @Override
            RegularFile call() throws Exception {
                return new RegularFile() {
                    @Override
                    File getAsFile() {
                        return fileTree("src/test/resources/anomaly-detection/").getSingleFile()
                    }
                }
            }
        }))*/
        println "Job Scheduler installed!!"
        setting 'repositories.url.allowed_urls', 'http://snapshot.test*'
        setting 'path.repo', "${buildDir}/cluster/shared/repo/adMixedCluster"
        setting 'http.content_type.required', 'true'
    }
}

task sarat(type: StandaloneRestIntegTestTask) {
    useCluster testClusters.adMixedCluster
    if (System.getProperty("mixedCluster") != null) {
        filter {
            includeTestsMatching "org.opensearch.ad.bwc.*IT"
        }
    }
    install_plugin(true,0)
    String baseName = "adMixedCluster"
    nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}".allHttpSocketURI.join(",")}")
    nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}".getName()}")
}

task sarat1(type: StandaloneRestIntegTestTask) {
    dependsOn sarat
    String baseName = "adMixedCluster"
    useCluster testClusters."${baseName}"
    doFirst {
      testClusters.adMixedCluster.nextNodeToNextVersion()
      testClusters.adMixedCluster.nodes[1].plugin(project.tasks.bundlePlugin.archiveFile)
    }
    if (System.getProperty("mixedCluster") != null) {
        filter {
            includeTestsMatching "org.opensearch.ad.bwc.*IT"
        }
    }
    nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}".allHttpSocketURI.join(",")}")
    nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}".getName()}")
}

[saratvemulapalli]

Created an issue on OpenSearch: opensearch-project/OpenSearch#1002

[saratvemulapalli]

Closing this issue as tests have been pushed to AD.