Update transition script to allow docker/podman usage (#4427)

* add docker and podman support to the transition script. * update transition script to no longer use the temp directory, instead it will emplace new recordings under the local .assets folder. * update transition script readme to highlight some additional detail about using the script Co-authored-by: Ben Broderick Phillips <ben@benbp.net> Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
Azure · Oct 17, 2022 · 86cad9e · 86cad9e
1 parent 4fbc7ea
commit 86cad9e
Show file tree

Hide file tree

Showing 3 changed files with 148 additions and 81 deletions.
diff --git a/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs b/tools/test-proxy/Azure.Sdk.Tools.TestProxy/RecordingHandler.cs
@@ -504,7 +504,7 @@ public static string GetAssetsJsonLocation(string pathToAssetsJson, string conte
 
             if (!Path.IsPathFullyQualified(pathToAssetsJson))
             {
-                path = Path.Join(contextDirectory, pathToAssetsJson);
+                path = Path.Join(contextDirectory, pathToAssetsJson.Replace("\\", "/"));
             }
 
             return path;

diff --git a/tools/test-proxy/scripts/transition-scripts/README.md b/tools/test-proxy/scripts/transition-scripts/README.md
@@ -8,16 +8,46 @@ The script `generate-assets-json.ps1` will execute the initial migration of your
 
 The script is [generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/transition-scripts/generate-assets-json.ps1)
 
+### Download the transition script locally
+
+```powershell
+Invoke-WebRequest -OutFile "generate-assets-json.ps1" https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/tools/test-proxy/scripts/transition-scripts/generate-assets-json.ps1
+```
+
+```bash
+wget https://raw.githubusercontent.com/Azure/azure-sdk-tools/main/tools/test-proxy/scripts/transition-scripts/generate-assets-json.ps1 -o generate-assets-json.ps1
+```
+
 ## Setup
 
-Before running the script, understand that **only services that have migrated to use the `test-proxy` as their record/playback solution can store recordings into the external assets repository.** The CLI command commands are invoked from the `test-proxy` itself, so if it is NOT being used for record/playback, the work to transition to the `test-proxy` must be done before recordings can be moved.
+Before running the script, understand that **only services that have migrated to use the `test-proxy` as their record/playback solution can store recordings into the external assets repository.** The test-proxy itself contains the code for `restoring`/`push`ing recordings, so if it is NOT being used for record/playback, that work must be completed before recordings can be moved.
 
-Running the script requires the following:
+Running the script requires these base requirements.
 
 - [x] The targeted library is already migrated to use the test-proxy.
 - [x] Git version `>2.25.0` needs to be on the machine and in the path. Git is used by the script and test-proxy.
-- [x] Test-proxy needs to be on the machine and in the path. Instructions for that are [here](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#installation). The script uses test-proxy's CLI commands.
 - [x] [Powershell Core](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.2) at least version 7.
+- [x] Ensure global git config settings for `user.name` and `user.email` are updated. [Reference](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup)
+  - Override with environment variables `GIT_COMMIT_EMAIL` and `GIT_COMMIT_OWNER`. If either of these are set, they will override the default values pulled from `git config --global`.
+
+Once the above requirements are met, developers are welcome to choose one of the following paths.
+
+### `test-proxy` dotnet tool installed and called directly
+
+Provide `TestProxyExe` argument of `test-proxy` or leave it **blank**. This is the default use-case of this transition script.
+
+- [x] Test-proxy needs to be on the machine and in the path. Instructions for that are [here](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#installation).
+
+The newly installed test-proxy tool will be used during the recording migration portion of this script.
+
+### `docker` or `podman` invocation
+
+To utilize this methodology, the user must set input argument `TestProxyExe` to `docker` or `podman`.
+
+Other requirements:
+
+- [x] Install [docker](https://docs.docker.com/engine/install/) or [podman](https://podman.io/getting-started/installation.html)
+- [x] Set the environment variable `GIT_TOKEN` a valid token representing YOUR user
 
 ## Permissions
 
@@ -43,19 +73,27 @@ The location of the automatically restored assets is colloquially referred to as
 
 ## Running the script
 
-[generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/transition-scripts/generate-assets-json.ps1) is a standalone powershell script with no supporting script requirements. The easiest way to run the script would be to pull the <https://github.com/Azure/azure-sdk-tools> repository locally and run the script directly from there.
+[generate-assets-json.ps1](https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/transition-scripts/generate-assets-json.ps1) is a standalone powershell script with no supporting script requirements. The easiest way to run the script would be to use a one-liner [defined above](#download-the-transition-script-locally) to grab the file directly. **Please ensure you have the newest version of this script before continuing!**
 
 ```powershell
+# if downloading the file singly, cd to the directory containing generate-assets-json.ps1
 cd "<target-language-repo>/sdk/<service>"
-<language-repo-root>/tools/test-proxy/transition-scripts/generate-assets-json.ps1
+<path-to-transition-script>/generate-assets-json.ps1
 ```
 
-The script needs to be executed inside an `sdk/<ServiceDirectory>` or deeper, in an up to date language repository. A good rule here would be look at where the ci.yml is for an service directory. In the case where each library for a given service directory has their own pipelines, at the `sdk/<ServiceDirectory><Library>` level, it is recommended that the assets.json is created there. If the `ci.yml` exists deeper then the `sdk/<ServiceDirectory>/<Library>` level, then it is recommended to run the script from that directory.
+The script needs to be executed inside an `sdk/<ServiceDirectory>` or deeper and from within an up to date language repository. A good rule here would be look at where the ci.yml is for an service directory. In the case where each library for a given service directory has their own pipelines, at the `sdk/<ServiceDirectory>/<Library>` level, it is recommended that the assets.json is created there. If the `ci.yml` exists deeper than the `sdk/<ServiceDirectory>/<Library>` level, then it is recommended to run the script from that directory.
+
+```powershell
+# calling transition script against tool, given local clones of azure-sdk-for-java and azure-sdk-tools
+cd c:/src/azure-sdk-for-java/sdk/attestation
+<path-to-transition-script>/generate-assets-json.ps1 -InitialPush
+```
 
 ```powershell
-# in practice, given local clones of azure-sdk-for-java and azure-sdk-tools
+# calling transition script against docker, given local clones of azure-sdk-for-java and azure-sdk-tools
+$env:GIT_TOKEN="my git token"
 cd c:/src/azure-sdk-for-java/sdk/attestation
-c:/src/azure-sdk-tools/tools/test-proxy/transition-scripts/generate-assets-json.ps1 -InitialPush
+<path-to-transition-script>/generate-assets-json.ps1 -TestProxyExe "docker" -InitialPush
 ```
 
 After running a script, executing a `git status` from within the language repo, where the script was invoked from, will reflect two primary results:
@@ -71,7 +109,7 @@ Given the previous example of `sdk/attestation` transition script invocation, us
 
 - Creation of the assets.json file in the `sdk/attestation` directory.
   - If `-InitialPush` has not been specified, the script stops here and exits.
-- A temp directory is created and the test-proxy's CLI restore is called on the current assets.json. Since there's nothing there, it'll just initialize an empty assets directory.
+- test-proxy's CLI restore is called on the current assets.json. Since there's nothing there, it'll just initialize an empty assets directory under the `.assets` directory under repo root.
 - The recordings are moved from their initial directories within the language repo into a temp directory that was created in the previous step.
   - The relative paths from root are preserved.
   - For example, the recordings for `C:/src/azure-sdk-for-python/sdk/tables` live in the `azure-data-tables/tests/recordings` subdirectory and in the target repository they'll live in `python/sdk/tables/azure-data-tables/tests/recordings`. All the azure-sdk supported languages will leverage [Azure/azure-sdk-assets](https://github.com/Azure/azure-sdk-assets), so adding a prefix to the output path `python` ensures that these recordings can live alongside others in the assets repo.

diff --git a/tools/test-proxy/scripts/transition-scripts/generate-assets-json.ps1 b/tools/test-proxy/scripts/transition-scripts/generate-assets-json.ps1
@@ -9,33 +9,54 @@ Requirements:
 needs to be run at an sdk/<ServiceDirectory> or deeper. For example sdk/core if the assets.json is
 being created at the service directory level or sdk/core/<somelibrary> if the assets.json is being
 created at the library level. A good rule here would be to run this in the same directory where the ci.yml
-file lives. For most this is the sdk/<ServiceDirectory> but, in some service directories, each library
-has its own ci.yml and pipeline and for these the ci.yml would be in the sdk/<ServiceDirectory>/<library>.
+file lives. For most this is the sdk/<ServiceDirectory>, but some services emplace a ci.yml alongside each package.
+In that case, the assets.json should live alongside the ci.yml in the sdk/<ServiceDirectory>/<library> directory.
 
 Generated assets.json file contents
-1. AssetsRepo: "Azure/azure-sdk-assets" - This is the assets repository
-2. AssetsRepoPrefixPath: "<language>" - this is will be computed from repository it's being run in
-3. TagPrefix: "<language>/<ServiceDirectory>" or "<language>/<ServiceDirectory>/<library>" or deeper if things
-              are nested in such a manner.
-4. Tag: "" - Initially empty, as nothing has yet been pushed
+- AssetsRepo: "Azure/azure-sdk-assets" - This is the assets repository, aka where your recordings will live after this script runs.
+- AssetsRepoPrefixPath: "<language>" - this is will be computed from repository it's being run in. 
+- TagPrefix: "<language>/<ServiceDirectory>" or "<language>/<ServiceDirectory>/<library>" or deeper if things
+              are nested in such a manner. All tags created for this assets.json will start with this name.
+- Tag: "" - Initially empty, as nothing has yet been pushed.
+
+If flag InitialPush is set, recordings will be automatically pushed to the assets repo and the Tag property updated.
+
+.PARAMETER TestProxyExe
+The executable used during the "InitialPush" action. Defaults to the dotnet tool test-proxy, but also supports "docker" or "podman".
+
+.PARAMETER InitialPush
+Pass this flag to automagically move all recordings found UNDER your assets.json to an assets repo.
+
+Detailed process:
+- Create a temp directory.
+- Call "restore" against that assets directory to prepare it to receive updates.
+- Move all recordings found under the assets.json within the language repo to the assets directory prepared by the restore operation in the previous step.
+- Push moved recordings to the assets repo.
+- Update the assets.json with the new tag.
+
+.PARAMETER UseTestRepo
+Enabling this parameter will result in an assets.json that points at repo Azure/azure-sdk-assets-integration. This is the
+integration repo that the azure-sdk EngSys team uses to integration test this script and other asset-sync features.
+
+Most library devs should ignore this setting unless directed otherwise (or if they're curious!). Permissions to the integration
+repo are identical to the default assets repo.
+
 #>
 param(
+  [ValidateSet("test-proxy", "docker", "podman", IgnoreCase = $true)]
+  [Parameter(Mandatory = $false)]
+  [string] $TestProxyExe = "test-proxy",
   [switch] $InitialPush,
   [switch] $UseTestRepo
 )
 
 # Git needs to be in the path to determine the language and, if the initial push
 # is being performed, for the CLI commands to work
 $GitExe = "git"
-# If the initial push is being performed, test-proxy needs to be in the path in
-# order for the CLI commands to be executed
-$TestProxyExe = "test-proxy"
-
-$OriginalProxyAssetsFolder = $env:PROXY_ASSETS_FOLDER
 
 # The built test proxy on a dev machine will have the version 1.0.0-dev.20221013.1
 # whereas the one installed from nuget will have the version 20221013.1 (minus the 1.0.0-dev.)
-$MinTestProxyVersion = "20221012.1"
+$MinTestProxyVersion = "20221017.1"
 
 $DefaultAssetsRepo = "Azure/azure-sdk-assets"
 if ($UseTestRepo) {
@@ -228,8 +249,42 @@ Function New-Assets-Json-File {
 Function Invoke-ProxyCommand {
   param(
     [string] $TestProxyExe,
-    [string] $CommandArgs
+    [string] $CommandArgs,
+    [string] $TargetDirectory
   )
+  $updatedDirectory = $TargetDirectory.Replace("`\", "/")
+
+  if ($TestProxyExe -eq "docker" -or $TestProxyExe -eq "podman"){
+    $token = $env:GIT_TOKEN
+    $committer = $env:GIT_COMMIT_OWNER
+    $email = $env:GIT_COMMIT_EMAIL
+
+    if (-not $committer) {
+      $committer = & git config --global user.name
+    }
+
+    if (-not $email) {
+      $email = & git config --global user.email
+    }
+
+    if(-not $token -or -not $committer -or -not $email){
+      Write-Error "When running this transition script in `"docker`" or `"podman`" mode, " `
+        + "the environment variables GIT_TOKEN, GIT_COMMIT_OWNER, GIT_COMMIT_EMAIL must be set to reflect the appropriate user. "
+    }
+
+    $targetImage = if ($env:TRANSITION_SCRIPT_DOCKER_TAG) { $env:TRANSITION_SCRIPT_DOCKER_TAG } else { "azsdkengsys.azurecr.io/engsys/test-proxy:latest" }
+
+    $CommandArgs = @(
+      "run --rm --name transition.test.proxy",
+      "-v `"${updatedDirectory}:/srv/testproxy`"",
+      "-e `"GIT_TOKEN=${token}`"",
+      "-e `"GIT_COMMIT_OWNER=${committer}`"",
+      "-e `"GIT_COMMIT_EMAIL=${email}`"",
+      $targetImage,
+      "test-proxy",
+      $CommandArgs
+    ) -join " "
+  }
 
   Write-Host "$TestProxyExe $CommandArgs"
   [array] $output = & "$TestProxyExe" $CommandArgs.Split(" ")
@@ -239,46 +294,20 @@ Function Invoke-ProxyCommand {
   }
 }
 
-Function Get-TempPath {
-  return [System.IO.Path]::GetTempPath()
-}
-# Set the PROXY_ASSETS_FOLDER to [System.IO.Path]::GetTempPath()/<Guid>
-# This is a temporary directory that'll be used for the restore/push operatios
-# on the assets.json that was just created. This is temporary, as the original
-# PROXY_ASSETS_FOLDER value was saved at the beginning of the script and
-# the original value will be restored at the end of the script.
-Function Set-ProxyAssetsFolder {
-  $guid = [Guid]::NewGuid()
-  $tempPath = Get-TempPath
-  $proxyAssetsFolder = Join-Path -Path $tempPath -ChildPath $guid
-  New-Item -Type Directory -Force -Path $proxyAssetsFolder | Out-Null
-  $env:PROXY_ASSETS_FOLDER = $proxyAssetsFolder
-  return $proxyAssetsFolder
-}
-
 # Get the shorthash directory under PROXY_ASSETS_FOLDER
 Function Get-AssetsRoot {
   param(
     [string] $AssetsJsonFile
   )
-
-  $startingPath = $env:PROXY_ASSETS_FOLDER
-  # It's odd that $folder.Count and $folders.Lenght work and we need to do this
-  $numDirs = Get-ChildItem $startingPath -Directory | Measure-Object | ForEach-Object { $_.Count }
-  $folders = Get-ChildItem $startingPath -Directory
-  # There should only be one folder
-  if (1 -ne $numDirs) {
-    Write-Error "The assets directory ($startingPath) should only contain 1 subfolder not $numDirs ($folders -join $([Environment]::NewLine))"
-    exit 1
-  }
-  $assetsRoot = $folders[0].FullName
   $repoRoot = Get-Repo-Root
-  $assets = Get-Content $AssetsJsonFile | Out-String | ConvertFrom-Json
-  $assetsJsonPath = Split-Path -Path $AssetsJsonFile
-  $relPath = [IO.Path]::GetRelativePath($repoRoot, $assetsJsonPath)
-  $assetsRoot = Join-Path -Path $folders[0].FullName -ChildPath $assets.AssetsRepoPrefixPath -AdditionalChildPath $relPath
+  $relPath = [IO.Path]::GetRelativePath($repoRoot, $AssetsJsonFile).Replace("`\", "/")
+  $breadcrumbFile = Join-Path $repoRoot ".assets" ".breadcrumb"
 
-  return $assetsRoot
+  $breadcrumbString = Get-Content $breadcrumbFile | Where-Object { $_.StartsWith($relPath) }
+  $assetRepo = $breadcrumbString.Split(";;")[1]
+  $assetsPrefix = (Get-Content $AssetsJsonFile | Out-String | ConvertFrom-Json).AssetsRepoPrefixPath
+
+  return Join-Path $repoRoot ".assets" $assetRepo $assetsPrefix $relPath
 }
 
 Function Move-AssetsFromLangRepo {
@@ -290,25 +319,21 @@ Function Move-AssetsFromLangRepo {
   Write-Host "Get-ChildItem -Recurse -Filter ""*.json"" | Where-Object { `$_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains ""$filter"" }"
   $filesToMove = Get-ChildItem -Recurse -Filter "*.json" | Where-Object { $_.DirectoryName.Split([IO.Path]::DirectorySeparatorChar) -contains "$filter" }
   [string] $currentDir = Get-Location
+
   foreach ($fromFile in $filesToMove) {
+    Write-Host $fromFile
     $relPath = [IO.Path]::GetRelativePath($currentDir, $fromFile)
     $toFile = Join-Path -Path $AssetsRoot -ChildPath $relPath
     # Write-Host "Moving from=$fromFile"
     # Write-Host "          to=$toFile"
     $toPath = Split-Path -Path $toFile
+
+    Write-Host $toFile
     if (!(Test-Path $toPath)) {
       New-Item -Path $toPath -ItemType Directory -Force | Out-Null
     }
-    Move-Item -LiteralPath $fromFile -Destination $toFile
-  }
-}
-
-Function Remove-ProxyAssetsFolder {
-  if (![string]::IsNullOrWhitespace($env:DISABLE_INTEGRATION_BRANCH_CLEANUP)) {
-    return
+    Move-Item -LiteralPath $fromFile -Destination $toFile -Force
   }
-  Write-Host "cleaning up $env:PROXY_ASSETS_FOLDER"
-  Remove-Item -LiteralPath $env:PROXY_ASSETS_FOLDER -Force -Recurse
 }
 
 Test-Exe-In-Path -ExeToLookFor $GitExe
@@ -319,13 +344,18 @@ $language = Get-Repo-Language
 # directories
 if ($InitialPush) {
   Test-Exe-In-Path -ExeToLookFor $TestProxyExe
-  Test-TestProxyVersion -TestProxyExe $TestProxyExe
-  if (!$LangRecordingDirs.ContainsKey($language)) {
-    Write-Error "The language, $language, does not have an entry in the LangRecordingDirs dictionary."
-    exit 1
+
+  if ($TestProxyExe -eq "test-proxy") {
+    Test-TestProxyVersion -TestProxyExe $TestProxyExe
+    if (!$LangRecordingDirs.ContainsKey($language)) {
+      Write-Error "The language, $language, does not have an entry in the LangRecordingDirs dictionary."
+      exit 1
+    }
   }
 }
 
+$repoRoot = Get-Repo-Root
+
 # Create the assets-json file
 $assetsJsonFile = New-Assets-Json-File -Language $language
 
@@ -334,33 +364,32 @@ $assetsJsonFile = New-Assets-Json-File -Language $language
 # 2. Move all of the assets over, preserving the directory structure
 # 3. Push the repository which will update the assets.json with the new Tag
 if ($InitialPush) {
-
   try {
-    $proxyAssetsFolder = Set-ProxyAssetsFolder
-    Write-Host "proxyAssetsFolder=$proxyAssetsFolder"
+    $assetsJsonRelPath = [System.IO.Path]::GetRelativePath($repoRoot, $assetsJsonFile)
 
     # Execute a restore on the current assets.json, it'll prep the root directory that
     # the recordings need to be copied into
-    $CommandArgs = "restore --assets-json-path $assetsJsonFile"
-    Invoke-ProxyCommand -TestProxyExe $TestProxyExe -CommandArgs $CommandArgs
+    $CommandArgs = "restore --assets-json-path $assetsJsonRelPath"
+    Invoke-ProxyCommand -TestProxyExe $TestProxyExe -CommandArgs $CommandArgs -TargetDirectory $repoRoot
 
     $assetsRoot = Get-AssetsRoot -AssetsJsonFile $assetsJsonFile
     Write-Host "assetsRoot=$assetsRoot"
 
     Move-AssetsFromLangRepo -AssetsRoot $assetsRoot
 
-    $CommandArgs = "push --assets-json-path $assetsJsonFile"
-    Invoke-ProxyCommand -TestProxyExe $TestProxyExe -CommandArgs $CommandArgs
+    $CommandArgs = "push --assets-json-path $assetsJsonRelPath"
+    Invoke-ProxyCommand -TestProxyExe $TestProxyExe -CommandArgs $CommandArgs -TargetDirectory $repoRoot
 
     # Verify that the assets.json file was updated
     $updatedAssets = Get-Content $assetsJsonFile | Out-String | ConvertFrom-Json
     if ([String]::IsNullOrWhitespace($($updatedAssets.Tag))) {
-      Write-Error "AssetsJsonFile ($assetsJsonFile) did not have it's tag updated"
+      Write-Error "AssetsJsonFile ($assetsJsonFile) did not have it's tag updated. Check above output messages for erroneous git output."
       exit 1
     }
   }
-  finally {
-    Remove-ProxyAssetsFolder
-    $env:PROXY_ASSETS_FOLDER = $OriginalProxyAssetsFolder
+  catch {
+    $ex = $_
+    Write-Host $ex
+    exit 1
   }
 }