Skip to content

Commit

Permalink
[tests] Add Designer Integration Tests to Nightly Tests (#7835)
Browse files Browse the repository at this point in the history
Context: 618bd4a

Commit 618bd4a stated:

> Heads-up: xamarin-android/main is *becoming* .NET Android-only;
> Classic Xamarin.Android is only supported [through 2024-May-1][0];
> all current and future feature work is oriented toward .NET Android.
> Classic Xamarin.Android will only be fully buildable/supportable on
> the [d17-5 release branch][1].

This indirectly brings us to the [Xamarin.Android Designer][2]
Integration Tests:

 1. The Xamarin.Android Designer only works with
    "Classic" Xamarin.Android; it doesn't work with .NET Android.

    (It will likely *never* work with .NET Android, unless
    priorities change.)

    This is "fine" because Classic Xamarin.Android still works, but

 2. *How often* are the Xamarin.Android Designer Integration tests run?

(2) is of concern: the Designer Integration tests are normally only run
during PR builds and release branch builds.  However, the d17-5 release
branch is getting very little of either.  Meanwhile, the Designer
Integration tests live in xamarin/UITools, which is actively developed.

This presents a terrible awful no good very bad scenario: UITools
changes "something" which breaks the interaction between
Xamarin.Android and the Designer, it isn't caught in UITools, and it
isn't caught in xamarin-android because the tests in xamarin-android
aren't run frequently enough.

Address this scenario: add the Designer Integration tests to
`azure-pipelines-nightly.yaml`, which will cause them to be run
every night.

[0]: https://dotnet.microsoft.com/en-us/platform/support/policy/xamarin
[1]: https://github.com/xamarin/xamarin-android/commits/d17-5
[2]: https://learn.microsoft.com/en-us/xamarin/android/user-interface/android-designer/
  • Loading branch information
dellis1972 authored Mar 7, 2023
1 parent a200af1 commit 1fae01a
Showing 1 changed file with 163 additions and 0 deletions.
163 changes: 163 additions & 0 deletions build-tools/automation/azure-pipelines-nightly.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,16 @@ resources:
type: github
name: xamarin/monodroid
endpoint: xamarin
- repository: uitools
type: github
name: xamarin/UITools
ref: refs/heads/main
endpoint: xamarin

parameters:
- name: provisionatorChannel
type: string
default: latest # Support for launching a build against a Provisionator PR (e.g., pr/[github-account-name]/[pr-number]) as a means to test in-progress Provisionator changes

# Global variables
variables:
Expand Down Expand Up @@ -58,6 +68,159 @@ stages:
artifactName: Build Results - Nightly macOS
includeBuildResults: true

- stage: designer_tests
displayName: Designer Tests
dependsOn: mac_build
variables:
- group: Xamarin-Secrets
- group: xamops-azdev-secrets
jobs:
# Check - "Xamarin.Android (macOS > Tests > Designer Integration)"
- job: designer_integration_mac
condition: false #TODO: Enable once test issues are fixed.
displayName: macOS > Tests > Designer Integration
pool:
vmImage: $(HostedMacImage)
timeoutInMinutes: 120
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
EnableRegressionTest: true
steps:
- checkout: uitools
clean: true
submodules: recursive
path: s/UITools
persistCredentials: true

- powershell: |
# Use the branch name of the source being built or the PR target branch name. Fall back to 'main' if the branch is unknown.
$branchName = "main"
Set-Location -Path $(System.DefaultWorkingDirectory)/UITools
git checkout $branchName
git submodule update -q --init --recursive
displayName: Clone and update UITools
- task: NuGetAuthenticate@0
displayName: authenticate with azure artifacts
inputs:
forceReinstallCredentialProvider: true

- task: provisionator@2
displayName: provision designer dependencies
inputs:
github_token: $(GitHub.Token)
provisioning_script: $(System.DefaultWorkingDirectory)/UITools/src/bot-provisioning/dependencies.csx
provisioning_extra_args: -remove Xamarin.Android -vv DEVDIV_PKGS_NUGET_TOKEN=$(DevDiv.NuGet.Token) SECTOOLS_PKGS_NUGET_TOKEN=$(SecTools.NuGet.Token)
env:
PROVISIONATOR_CHANNEL: ${{ parameters.provisionatorChannel }}

- template: yaml-templates/setup-test-environment.yaml
parameters:
xaSourcePath: $(System.DefaultWorkingDirectory)/xamarin-android
jdkTestFolder: $(JAVA_HOME_8_X64)
provisionatorChannel: ${{ parameters.provisionatorChannel }}

- template: designer/android-designer-build-mac.yaml@yaml-templates
parameters:
designerSourcePath: $(System.DefaultWorkingDirectory)/UITools/src

- template: designer/android-designer-tests.yaml@yaml-templates
parameters:
designerSourcePath: $(System.DefaultWorkingDirectory)/UITools/src
runAddinTests: false

- task: CopyFiles@2
displayName: 'Copy binlogs'
inputs:
sourceFolder: $(System.DefaultWorkingDirectory)/UITools/src/Xamarin.Designer.Android
contents: '**/*.binlog'
targetFolder: $(Build.ArtifactStagingDirectory)/designer-binlogs
overWrite: true
flattenFolders: true
condition: ne(variables['Agent.JobStatus'], 'Succeeded')

- template: yaml-templates/publish-artifact.yaml
parameters:
displayName: upload designer binlogs
artifactName: Test Results - Designer - macOS
targetPath: $(Build.ArtifactStagingDirectory)/designer-binlogs
condition: ne(variables['Agent.JobStatus'], 'Succeeded')

# Check - "Xamarin.Android (Windows > Tests > Designer Integration)"
- job: designer_integration_win
displayName: Windows > Tests > Designer Integration
pool:
vmImage: $(HostedWinImage)
timeoutInMinutes: 120
cancelTimeoutInMinutes: 5
workspace:
clean: all
variables:
EnableRegressionTest: true
RegressionTestSuiteOutputDir: C:\Git\ADesRegTestSuite
VisualStudioInstallationPath: C:\Program Files\Microsoft Visual Studio\2022\Enterprise
steps:
- checkout: uitools
clean: true
submodules: recursive
path: s\UITools
persistCredentials: true

- powershell: |
# Use the branch name of the source being built or the PR target branch name. Fall back to 'main' if the branch is unknown.
$branchName = "main"
Set-Location -Path $(System.DefaultWorkingDirectory)\UITools
git checkout $branchName
git submodule update -q --init --recursive
displayName: Clone and update UITools
- task: NuGetAuthenticate@0
displayName: authenticate with azure artifacts
inputs:
forceReinstallCredentialProvider: true

- task: provisionator@2
displayName: provision designer dependencies
inputs:
github_token: $(GitHub.Token)
provisioning_script: $(System.DefaultWorkingDirectory)\UITools\src\bot-provisioning\dependencies.csx
provisioning_extra_args: -vv DEVDIV_PKGS_NUGET_TOKEN=$(DevDiv.NuGet.Token) SECTOOLS_PKGS_NUGET_TOKEN=$(SecTools.NuGet.Token)
env:
PROVISIONATOR_CHANNEL: ${{ parameters.provisionatorChannel }}

- template: yaml-templates\setup-test-environment.yaml
parameters:
xaSourcePath: $(System.DefaultWorkingDirectory)\xamarin-android
jdkTestFolder: $(JAVA_HOME_8_X64)
provisionatorChannel: ${{ parameters.provisionatorChannel }}

- task: VSBuild@1
displayName: Restore Xamarin.AndroidDesigner
inputs:
solution: $(System.DefaultWorkingDirectory)\UITools\src\Xamarin.Designer.Android\Xamarin.AndroidDesigner.sln
vsVersion: 17.0
msbuildArgs: >-
/t:Restore /p:RestoreDisableParallel=true
/p:RestoreConfigFile="$(System.DefaultWorkingDirectory)\UITools\NuGet.Config"
/p:JavaSdkDirectory="$(JAVA_HOME_8_X64)"
platform: Any CPU
configuration: DebugWin32

- task: VSBuild@1
displayName: Build Xamarin.AndroidDesigner
inputs:
solution: $(System.DefaultWorkingDirectory)\UITools\src\Xamarin.Designer.Android\Xamarin.AndroidDesigner.sln
vsVersion: 17.0
msbuildArgs: /t:Build /p:GitHubToken=$(GitHub.Token)
platform: Any CPU
configuration: DebugWin32

- template: yaml-templates/run-designer-tests.yml
parameters:
designerSourcePath: $(System.DefaultWorkingDirectory)\UITools\src

- stage: test
displayName: Test
dependsOn: mac_build
Expand Down

0 comments on commit 1fae01a

Please sign in to comment.