Test Proxy Migration

Test Proxy Server

It is common practice in many of our SDKs to test service client code by recording HTTP requests and responses during a test run against a live endpoint and then playing back the matching responses to requests in subsequent runs.

The basic idea is to have a test server that sits between the client being tested and the live endpoint. Instead of mocking out the communication with the server, the communication can be redirected to a test server.

Major benefits

1. Repo size - Our repos are getting big and the biggest contributor to this issue are recordings. Consolidating on a single solution and recording format makes it easier to have the storage for recordings moved outside of the main repo.
2. Share Code - reusing/centralizing the general code for recording and playback. eg: test server is hard-wired for sanitization/redirection defaults
3. Performance testing - eliminate server-side bottlenecks from a benchmark so instead of contacting the live service, a test service can respond with recorded/cached responses.

Steps:

The roll-out strategy can be split into below phases:

1. Record test recordings with the test-proxy integration
2. Migrate updated recordings to assets repo
3. Using test proxy going forward

1) Record test recordings with the test-proxy integration

Each SDK needs to re-record its test recordings using the test-proxy integration to ensure a consolidated recording format with serialized/sanitized requests and their matching responses.

Steps:

1. Delete old recordings from under src/test/resources/session-records.

2. To use the proxy, test classes should extend from TestProxyTestBase instead of TestBase

   public abstract class DocumentAnalysisClientTestBase extends TestProxyTestBase {}

3. Run tests in Record mode to get updated recordings.

4. Sanitize secrets if required:

   Default sanitizers, similar to the use of the RecordingRedactor are already registered in the TestProxyUtils for default redactions.

   Custom sanitizers can be added using TestProxySanitizer & interceptor.addSanitizer() method for addressing specific service sanitization needs.

   For example, registering a custom sanitizer for redacting the value of json key modelId from the response body looks like the following:

   @Override
   protected void beforeTest() {
       List<TestProxySanitizer> customSanitizer = new ArrayList<>();
       // sanitize value for key: "modelId" in response json body
       customSanitizer.add(new TestProxySanitizer("$..modelId", REPLACEMENT_TEXT, TestProxySanitizerType.BODY_KEY));
       // add sanitizer to Test Proxy Policy
       interceptorManager.addSanitizers(customSanitizer);
   }

   Note: Sanitizers must only be added once the playback client or record policy is registered. Look at the TableClientTestBase class for example.

5. After running tests in record mode, the newly updated recordings no longer be in the azure-sdk-for-java repo. These updates will be reflected in a git-excluded .assets folder at the root of the repo.

   To push these recordings to the assets repo follow the steps here 2)

   If you do not have an existing asset.json file in your serviceDirectory/LibraryLevel/ follow the steps here

2) Migrate updated recordings to assets repo

This is only a one-time setup. You only need to do this if there isn't an existing asset.json file in your serviceDirectory/LibraryLevel/

Migrating the test recordings to the asset repo will enable the test proxy to work against repositories and will not require them to emplace their test recordings directly alongside their test implementations.

Reference PR for details

Prerequisites

Migration script prerequisites

• The targeted library is already migrated to use the test proxy.
• Git version > 2.25.0 is to on the machine and in the path. Git is used by the script and test proxy.
• PowerShell Core >= 7.0 is installed.
• Global git config settings are configured for user.name and user.email.
  • These settings can be overridden with environment variables GIT_COMMIT_OWNER and GIT_COMMIT_EMAIL, respectively.
• Membership in the azure-sdk-write GitHub group.

Steps:

1. Create an asset.json file for each SDK i.e sdk/tables/azure-data-tables/assets.json using the generate-assets-json.ps1 The asset.json file basically will allow the test-proxy to restore a set of recordings to a path, then load the recording from that newly gathered data.

   Running the script:

   The script needs to be executed inside an sdk/ServiceDirectory/Library level.

       C:/repo/sdk-for-java/sdk/formrecognizer/azure-ai-formrecognizer> ..\..\..\eng\common\testproxy\onboarding\generate-assets-json.ps1 -InitialPush

   Running the script without the -InitialPush option will just create the assets.json with an empty tag. No data movement.

2. After running a script, executing a git status from within the language repo, where the script was invoked from, will reflect two primary results:

   • A new assets.json will be present in the directory from which they invoked the transition script.
   • A bunch of deleted files from where their recordings were before they were pushed to the assets repo.

   Check in these two changes to the language repo.

3) Using test proxy going forward

After moving recordings to the asset repo, live and playback testing will be the same as it was in the past.

Running tests in Playback mode

1. When running tests in Playback mode, the test-proxy automatically checks out the appropriate tag in each local assets repo and performs testing.
   Here is the result of running tests in playback-mode for a couple of packages within the Java repo:

   

2. To find the location test-proxy is storing your files locally, you can run:

test-proxy config  locate -a .\assets.json

Running tests in Record mode

Prerequisites here

1. After running tests in record mode, the newly updated recordings no longer be in the azure-sdk-for-java repo. These updates will be reflected in a git-excluded .assets folder at the root of the repo.

2. You can cd into the folder containing your package's recordings (see above for finding it) and use git status to view the recording updates. Verify the updates, and use the following command to push these recordings to the azure-sdk-assets repo:

       C:/repo/sdk-for-java/>test-proxy push -a sdk/tables/azure-data-tables/assets.json

   or running the command providing the context directory

       test-proxy push -a C:/repo/sdk-for-python/sdk/tables/azure-data-tables/assets.json

   How to set up and use the proxy can be found here.

3. After pushing your recordings, the assets.json file for your package will be updated to point to a new Tag that contains the updates. Include this assets.json update in any pull request to update the recordings pointer in the upstream repo.

   Find more details on the asset-sync feature here.

Potential gotchas!

1) Symmetry

The test proxy is much more exact about matching recordings than the old recorder. You may get errors or strange behavior when you try to playback recording, such as output like this:

Header differences:
    <Ocp-Apim-Subscription-Key> is absent in record, value <REDACTED>
    <Authorization> is absent in request, value <Sanitized>

This indicates that there is something different about your test in playback and record modes. Look for places where you're doing something conditional and ensure it is doing the same thing in both modes. If you think that your test is doing the same thing and you're still hitting problems, please reach out.

Example: Mismatched headers

Header differences:
    <Authorization> is absent in request, value <Sanitized>

Indicates the recording contains an "Authorization" header, but the playback HTTP request does not. When running in RECODE mode, the credential was available to calculate an Authorization header value. But in PLAYBACK mode, there is no credential to calculate the Authorization header value.

Code: Broken

if (getTestMode() == RECORD || getTestMode() == LIVE) {
   builder.credential(new DefaultCredentialBuilder().build())
       .endpoint(endpoint)
       .buildClient();
} else {
   builder.endpoint(fakeEndpoint)
       .buildCLient();
}

Code: Fixed

if (getTestMode() == RECORD || getTestMode() == LIVE) {
   builder.credential(new DefaultCredentialBuilder().build())
       .endpoint(endpoint)
       .buildClient();
} else {
   var mockedTokenCredential = mock(TokenCredential.class);
   when(mockedTokenCredential.getToken(any()).return(.....);

   builder.credential(mockedTokenCredential)
       .endpoint(fakeEndpoint)
       .buildClient();
}

2) New pattern for getting the HttpClient

Previously, in lots of tests you might have seen something like this:

SomeClient builder = new SomeClientBuilder()
    // httpClient being an injected parameter, etc.
    .httpClient(httpClient == null ? interceptorManager.getPlaybackClient() : httpClient)
    .build();

This worked because in PLAYBACK mode, TestBase.getHttpClients() would return null. In the new path we want to use the same HttpClient for record and playback. This means getHttpClients() now always returns a client. Thus, this code should change to look like this:

SomeClient builder = new SomeClientBuilder()
    // httpClient being an injected parameter, etc.
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient)
    .build();

3) Adding record polices where they weren't needed

Consider:

SomeClientBuilder builder = new SomeClientBuilder()
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient);
if (getTestMode() != TestMode.PLAYBACK) {
    builder.addPolicy(interceptorManager.getRecordPolicy());
}
return builder.buildClient();

Previously, because RecordNetworkCallPolicy had a check for RECORD mode this turned out to be a no-op in LIVE mode. Arguably this is a bug, and is prohibited in the test proxy path. Simply change this to only happen in RECORD:

SomeClientBuilder builder = new SomeClientBuilder()
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient);
if (getTestMode() == TestMode.RECORD) {
    builder.addPolicy(interceptorManager.getRecordPolicy());
}
return builder.buildClient();

4) Use of @BeforeEach

Override beforeTest() instead of overriding BeforeEach to perform any set-up before each test case. Any initialization that occurs in TestBase occurs first before this. Overriding the BeforeEach in test classes with Junit5 will result in it not being inherited.

4) Adding Sanitizers before registering Playback Client or Record Policy

It is required to register a Playback Client or Record Policy before adding sanitizers so that the sanitizers are correctly attached to the respective client when running tests.

Test Proxy CLI commands.

The test proxy enables CLI interactions with the external assets repository using the sdk/<sdk-name>/asset.json file.

Restore operation

To locally clone or pull assets/test-recording associated with a particular asset.json file.

test-proxy restore --assets-json-path <assetsJsonPath>

Reset operation

To reset the local copy of the files back to the version targeted in the given assets.json file.

test-proxy reset --assets-json-path <assetsJsonPath>

Push operation

To push the updated/local assets to the assets repo.

test-proxy push --assets-json-path <assetsJsonPath>

Test Proxy local assets clone repo structure

Test-Proxy maintains a separate clone for each assets.json. The recording files will be located under your repo root under the .assets folder.

    +-------------------------------+
    |  azure-sdk-for-java/        |
    |    sdk/                       |
    |      storage/                 |
    | +------assets.json            |
    | |    appconfiguration/        |
    | | +----assets.json            |
    | | |  keyvault/                |
    | | |    azure-keyvault-secrets |
    | | |      assets.json-------+  |
    | | |    azure-keyvault-keys |  |
    | | |      assets.json---+   |  |
    | | |                    |   |  |
    | | |.assets/            |   |  |
    | | +--AuN9me8zrT/       |   |  |
    | |      <sparse clone>  |   |  |
    | +----5hgHKwvMaN/       |   |  |
    |        <sparse clone>  |   |  |
    |      AuN9me8zrT--------+   |  |
    |        <sparse clone>      |  |
    |      BSdGcyN2XL------------+  |
    |        <sparse clone>         |
    +-------------------------------+
   

For more details/doubts on test-proxy workings, follow Teams channel