Skip to content

Commit

Permalink
Minor fixes for test proxy asset sync container mode issues (#4447)
Browse files Browse the repository at this point in the history
* Support ssh key auth in test proxy docker context

* Use switch parameter types for proxy transition script
  • Loading branch information
benbp authored Oct 17, 2022
1 parent c34ae27 commit 4fbc7ea
Show file tree
Hide file tree
Showing 5 changed files with 24 additions and 14 deletions.
20 changes: 13 additions & 7 deletions tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,49 +75,55 @@ dotnet tool update azure.sdk.tools.testproxy --global --add-source https://pkgs.

To uninstall an existing test-proxy
```powershell
> dotnet tool uninstall --global azure.sdk.tools.testproxy
dotnet tool uninstall --global azure.sdk.tools.testproxy
```

The test-proxy is also available from the [azure-sdk-for-net public feed](https://dev.azure.com/azure-sdk/public/_artifacts/feed/azure-sdk-for-net)

After successful installation, run the tool:

```powershell
> test-proxy --storage-location <location>
test-proxy --storage-location <location>
```

If you've already installed the tool, you can always check the installed version by invoking:

```powershell
> test-proxy --version
test-proxy --version
```

### Via Docker Image

The Azure SDK Team maintains a public Azure Container Registry.

```powershell
> docker run -v <your-volume-name-or-location>:/srv/testproxy/ -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/test-proxy:latest
docker run -v <your-volume-name-or-location>:/srv/testproxy/ -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/test-proxy:latest
```

For example, to save test recordings to disk in your repo's `/sdk/<service>/tests/recordings` directory, provide the path to the root of the repo:

```powershell
> docker run -v C:\\repo\\azure-sdk-for-<language>:/srv/testproxy/ -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/test-proxy:latest
docker run -v C:\\repo\\azure-sdk-for-<language>:/srv/testproxy/ -p 5001:5001 -p 5000:5000 azsdkengsys.azurecr.io/engsys/test-proxy:latest
```

Note the **port and volume mapping** as arguments! Any files that exist in this volume locally will only be appended to/updated in place. It is a non-destructive initialize.

Within the container, recording outputs are written within the directory `/srv/testproxy/`.

NOTE: if you are authenticated to github via SSH keys instead of a credential manager with https, you must mount your ssh credentials into docker. The following command shows an example mounting the default ssh key ~/.ssh/id_rsa on linux:

```bash
docker run -v /home/ben/.ssh:/root/.ssh -v /home/ben/sdk/azure-sdk-for-go:/srv/testproxy --add-host=host.docker.internal:host-gateway -p 5001:5001 -p 5000:5000 testproxy bash -c 'eval `ssh-agent` && ssh-add /root/.ssh/id_rsa && test-proxy start --dump'
```

#### A note about docker caching

The azure-sdk team regularly update the image associated with the `latest` tag. Combined with the fact that docker will aggressively cache if possible, it is very possible that developers' local machines may be running outdated versions of the test-proxy.

To ensure that your local copy is up to date, run:

```powershell
> docker pull azsdkengsys.azurecr.io/engsys/test-proxy:latest
docker pull azsdkengsys.azurecr.io/engsys/test-proxy:latest
```

## Command line arguments
Expand Down Expand Up @@ -576,7 +582,7 @@ There are two options here, generate your own SSL Cert, or import an existing on
Invoke the command:

```powershell
> dotnet dev-certs https --trust
dotnet dev-certs https --trust
```

This will be automatically retrieved if you run the nuget installed version of the tool. You may optionally use `openssl` [like so](https://raw.githubusercontent.com/BorisWilhelms/create-dotnet-devcert/f3b5da6f9107834eb31ea5ba7c0583e14cda6b31/create-dotnet-devcert.sh) to generate a certificate. Note that this shell script creates a dev cert that is compatible with ubuntu.
Expand Down
4 changes: 4 additions & 0 deletions tools/test-proxy/Azure.Sdk.Tools.TestProxy/Store/GitStore.cs
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,10 @@ public void CheckoutRepoAtConfig(GitAssetsConfiguration config)

try
{
// Workaround for git directory ownership checks that may fail when running in a container as a different user.
if ("true" == Environment.GetEnvironmentVariable("TEST_PROXY_CONTAINER")) {
GitHandler.Run($"config --global --add safe.directory {config.AssetsRepoLocation}", config);
}
// Always retrieve latest as we don't know when the last time we fetched from origin was. If we're lucky, this is a
// no-op. However, we are only paying this price _once_ per startup of the server (as we cache assets.json status remember!).
GitHandler.Run("fetch --tags origin", config);
Expand Down
5 changes: 3 additions & 2 deletions tools/test-proxy/docker/dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ ENV \
Logging__LogLevel__Default=Information \
Logging__LogLevel__Microsoft=Information \
TEST_PROXY_FOLDER="/srv/testproxy/" \
PATH="/proxyserver:${PATH}"
PATH="/proxyserver:${PATH}" \
TEST_PROXY_CONTAINER=true

# dotnet-dev-certs tool is not included in aspnet image, so manually copy from sdk image
COPY --from=build /usr/share/dotnet/sdk/6.0.102/DotnetTools/dotnet-dev-certs/6.0.2-servicing.22064.12/tools/net6.0/any/ /dotnet-dev-certs
Expand All @@ -31,7 +32,7 @@ COPY docker_build/$CERT_IMPORT_SH docker_build/dotnet-devcert.pfx docker_build/d
ADD run_proxy.sh /proxyserver/test-proxy

RUN \
apk add --no-cache bash nss-tools git \
apk add --no-cache bash nss-tools openssh git \
# Fix line endings
&& sed -i -e 's/\r$//' $CERT_FOLDER/$CERT_IMPORT_SH \
&& sed -i -e 's/\r$//' /proxyserver/test-proxy \
Expand Down
4 changes: 2 additions & 2 deletions tools/test-proxy/scripts/transition-scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The script needs to be executed inside an `sdk/<ServiceDirectory>` or deeper, in
```powershell
# in practice, given local clones of azure-sdk-for-java and azure-sdk-tools
cd c:/src/azure-sdk-for-java/sdk/attestation
c:/src/azure-sdk-tools/tools/test-proxy/transition-scripts/generate-assets-json.ps1 -InitialPush $true
c:/src/azure-sdk-tools/tools/test-proxy/transition-scripts/generate-assets-json.ps1 -InitialPush
```

After running a script, executing a `git status` from within the language repo, where the script was invoked from, will reflect two primary results:
Expand All @@ -70,7 +70,7 @@ Running the script without the `-InitialPush` option will just create the assets
Given the previous example of `sdk/attestation` transition script invocation, users should see the following:

- Creation of the assets.json file in the `sdk/attestation` directory.
- If `-InitialPush $true` has not been set, the script stops here and exits.
- If `-InitialPush` has not been specified, the script stops here and exits.
- A temp directory is created and the test-proxy's CLI restore is called on the current assets.json. Since there's nothing there, it'll just initialize an empty assets directory.
- The recordings are moved from their initial directories within the language repo into a temp directory that was created in the previous step.
- The relative paths from root are preserved.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@ Generated assets.json file contents
4. Tag: "" - Initially empty, as nothing has yet been pushed
#>
param(
[Parameter(Mandatory = $false)]
[bool] $InitialPush = $false,
[bool] $UseTestRepo = $false
[switch] $InitialPush,
[switch] $UseTestRepo
)

# Git needs to be in the path to determine the language and, if the initial push
Expand Down

0 comments on commit 4fbc7ea

Please sign in to comment.