Parameterized test for SDK live test

Motivation

Azure SDK uses SPI for Http clients which allows user to choose the library to use. Azure core itself provides two HTTP libraries (Netty, Okhttp). Originally, we only have tests running on Netty HTTP client. In order to test over how SDK APIs work on different HTTP clients built by Azure SDK team, we decided to use parameterized test for packing more variables we want to test on (e.g. HTTP client, service version).

Current setting

For tests enable parameterized test, here is what current pipeline is working.

	HTTP client	Service Version
Linux (8)	Netty	Latest
MacOS(8)	Okhttp	Latest
Windows(8)	Netty	Latest
Linux (11)	Okhttp	Latest
MacOS(11)	Netty	Latest
Windows(11)	Okhttp	Latest

If any client libraries want to test some other values other than the table, we can setup a separate pipeline on running specific case.

Here lists the variables can be set on new pipeline:

• HTTP clients environment variable

AZURE_TEST_HTTP_CLIENTS

Allowed values (case insensitive):

1. Enable all clients: all
2. Enable one client only (Use partial identifier on client class name): netty(NettyAsyncHttpClient) or okhttp (OkHttpAsyncHttpClient)
3. Enable a list, client names separated by comma: netty, okhttp

• Service version environment variable

The allowed values for environment variable is what getValue returned from ServiceVersion of each clients. E.g. App configuration.

set AZURE_APPCONFIG_TEST_SERVICE_VERSIONS = "1.0"

	Service Version Env name
AppConfig	AZURE_APPCONFIG_TEST_SERVICE_VERSIONS
KeyVault Keys	AZURE_KEYVAULT_TEST_KEYS_SERVICE_VERSIONS
KeyVault Cryptography	AZURE_KEYVAULT_TEST_CRYPTOGRAPHY_SERVICE_VERSIONS
KeyValue Certificates	AZURE_KEYVAULT_TEST_CERTIFICATE_SERVICE_VERSIONS
KeyVault Secrets	AZURE_KEYVAULT_TEST_SECRETS_SERVICE_VERSIONS
To be continued..

Steps to enable to parameterzied test

Here are the steps for new client which needs to adopt parameterized tests

1. Adding test scope dependencies of all HTTP client libraries you want to test in pom.xml. e.g.

    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-core-http-netty</artifactId>
      <version>1.5.1</version> 
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-core-http-okhttp</artifactId>
      <version>1.2.2</version> 
      <scope>test</scope>
    </dependency>

1. Adding your method to return your argument list. e.g. Code example

    static final String DISPLAY_NAME_WITH_ARGUMENTS = "{displayName} with [{arguments}]";
    private static final String SERVICE_VERSION_FROM_ENV = Configuration.getGlobalConfiguration().get("{ServiceVersionEnvVariableName}");

    static Stream<Arguments> getTestParameters() {
        List<Arguments> argumentsList = new ArrayList<>();

        getHttpClients()
            .forEach(httpClient -> {
                Arrays.stream({ClientServiceVersion}.values()).filter({TestClass}::shouldServiceVersionBeTested)
                    .forEach(serviceVersion -> argumentsList.add(Arguments.of(httpClient, serviceVersion)));
            });
        return argumentsList.stream();
    }

    // Helper method to filter service version
`   private static boolean shouldServiceVersionBeTested({ClientServiceVersion} serviceVersion) {
        if (CoreUtils.isNullOrEmpty(SERVICE_VERSION_FROM_ENV)) {
            return {ClientServiceVersion}.getLatest().equals(serviceVersion);
        }
        if (AZURE_TEST_SERVICE_VERSIONS_VALUE_ALL.equalsIgnoreCase(SERVICE_VERSION_FROM_ENV)) {
            return true;
        }
        String[] configuredServiceVersionList = SERVICE_VERSION_FROM_ENV.split(",");
        return Arrays.stream(configuredServiceVersionList).anyMatch(configuredServiceVersion ->
            serviceVersion.getVersion().equals(configuredServiceVersion.trim()));
    }

1. Change the test annotations

Change the annotation and test method signature. e.g. Code example

    @ParameterizedTest(name = DISPLAY_NAME_WITH_ARGUMENTS)
    @MethodSource("{classpath}#getTestParameters")
    public void testYourAPI(HttpClient httpClient, ConfigurationServiceVersion serviceVersion)

1. Build your client in every test method using the httpClient and serviceVersion returned from getTestParamters

e.g. Code example

    private {SomeClient} getSomeClient(HttpClient httpClient, {ClientServiceVersion} serviceVersion) {
        return clientSetup(credentials -> {
            {SomeClientBuilder} builder = new {SomeClientBuilder} ()
                .connectionString(connectionString)
                .httpClient(httpClient == null ? interceptorManager.getPlaybackClient() : httpClient)
                .serviceVersion(serviceVersion)
                .httpLogOptions(new HttpLogOptions().setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS));
            if (getTestMode() != TestMode.PLAYBACK) {
                builder.addPolicy(interceptorManager.getRecordPolicy())
            }
            return builder.buildClient();
        });
    }

    @ParameterizedTest
    @MethodSource("{classpath}#getTestParameters")
    public void testSomeAPI(HttpClient httpClient, {SomeServiceVersion} serviceVersion) {
        {SomeClient} client = getSomeClient(httpClient, serviceVersion);
    }

Now you can try your parameterized test by setting env variables.