Skip to content

Commit

Permalink
Merge pull request #201: Port to .NET Core 3.0 and revamp engineering…
Browse files Browse the repository at this point in the history
…/build/test

**Non-goals**
- [ ] Signing

**New commands**
`dotnet build`: Build all managed code
`dotnet test`: Run all managed unit tests
`dotnet publish -r [osx-x64|win-x64]`: Build all code including Scalar installer packages and scripts

**Scripts**
`BuildScalarForMac.sh`/`BuildScalarForWindows.bat`: Build all code and generate installers and install scripts
`RunUnitTests.(sh|bat)`: Run the managed unit tests
`RunFunctionalTests.(sh|bat)`: Run the functional tests
`RestorePackages.bat`: No-op

**Directories**
```
.
├── out                          (build output)
│   ├── <Project1>
│   ├──    ...
│   ├── <ProjectN>
│   │
│   ├── Scalar.Distribution.Mac  (distribution: packages, scripts)
│   │  └── dist
│   │      └── <Configuration>
│   │          ├── InstallScalar.sh
│   │          │
│   │          ├── Scalar
│   │          │   └── Scalar.<version>.pkg
│   │          ├── GCM
│   │          │   └── gcmcore-osx-<version>.pkg
│   │          └── Git
│   │              └── git-<version>.pkg
│   │
│   └── Scalar.Distribution.Windows
│      └── dist
│          └── <Configuration>
│              ├── InstallScalar.bat
│              │
│              ├── Scalar
│              │   └── Scalar.<version>.exe
│              └── Git
│                  └── Scalar.<version>.exe
│
└── src                          (source)
    ├── <Project1>
    ├──    ...
    ├── <ProjectN>
    │
    ├── Dependencies.props       (Unified NuGet package versions)
    └── Directory.Build.props    (Common build props, including minimum Git version)
    └── Directory.Build.targets  (Common build target)
```

**Changes**
- Functional tests drops in CI do not include copies of the product; the installer drop must be consumed and installed.
- The metapackage on macOS has been dropped.. we only build the Scalar package and leave installation of the correct version of Git to the outside.
- The Platform projects have been folded into the Common project, under the `Platforms/` directory. There is now one platform loader that instantiates the correct type based on the current OS at runtime.
- The Scalar.Build project has been slimmed down massively; we only need:
   - Generate Scalar constants (version)
   - Compile templated file (x-plat replacement for `sed` to replace placeholder tokens)
   - Generate Windows application manifest (set supported Windows versions)
- The Scalar.Mac and Scalar.Windows projects have been folded into one, with a runtime fork to create the correct platform-specific types. The same applies for the other .Mac/.Windows projects.
- Scalar.Tests has been renamed to Scalar.TestInfratructure to better reflect its purpose.
- The Scalar.Notifications/Scalar.Mac directory has been flattened, and an MSBuild project been added that wraps the xcodeproj project. This allows us to chain in an xcodebuild into the MSBuild build with the `dotnet` CLI (see `dotnet build`/`publish`).
- Introduce shims for some missing APIs in .NET Core from .NET Framework, mainly things around ACLs which have been implemented in the new type `DirectoryEx` in the Common project.
- Include a copy of internal `NamedPipeServerStream` methods from the CoreFX to create pipes with the appropriate ACLs on Windows.
  • Loading branch information
derrickstolee authored Nov 7, 2019
2 parents f32f2f7 + ad94ae4 commit 7b6f1cb
Show file tree
Hide file tree
Showing 231 changed files with 2,122 additions and 4,041 deletions.
176 changes: 90 additions & 86 deletions .azure-pipelines/pull-request.yml
Original file line number Diff line number Diff line change
@@ -1,86 +1,90 @@
trigger:
- master
- releases/*

pr:
- master
- milestones/*
- releases/*
- servicing/*

variables:
branchCounter: $[counter(variables['Build.SourceBranch'], 0)]

jobs:
- job: Windows_Build_and_UnitTests
variables:
platformFriendlyName: Windows
strategy:
maxParallel: 2
matrix:
debug:
configuration: Debug
release:
configuration: Release
pool:
vmImage: vs2017-win2016
steps:
- powershell: $(Build.Repository.LocalPath)/Scripts/CI/Set-Version.ps1 -SourceBranchCounter $(branchCounter)
displayName: "Compute product version"
- template: templates/windows-build-and-unit-test.yml

- job: macOS_Build_and_UnitTests
variables:
platformFriendlyName: macOS
strategy:
maxParallel: 2
matrix:
debug:
configuration: Debug
release:
configuration: Release
pool:
name: 'Hosted macOS'
steps:
- powershell: $(Build.Repository.LocalPath)/Scripts/CI/Set-Version.ps1 -SourceBranchCounter $(branchCounter)
displayName: "Compute product version"
- template: templates/macos-build-and-unit-test.yml

- job: macOS_FunctionalTests
variables:
platformFriendlyName: macOS
configuration: Release
timeoutInMinutes: 30
pool:
name: 'Hosted macOS'
dependsOn: macOS_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # Use the drop from macOS_Build_and_UnitTests job
- template: templates/macos-functional-test.yml

- job: macOS_FunctionalTests_watchman
variables:
platformFriendlyName: macOS
configuration: Release
timeoutInMinutes: 30
pool:
name: 'Hosted macOS'
dependsOn: macOS_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # Use the drop from macOS_Build_and_UnitTests job
- template: templates/macos-functional-test-watchman.yml

- job: Windows_FunctionalTests
timeoutInMinutes: 30
variables:
platformFriendlyName: Windows
configuration: Release
pool:
vmImage: vs2017-win2016
dependsOn: Windows_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # We'll get the build drop from Windows_Build_and_UnitTests job
- template: templates/windows-functional-test.yml
trigger:
- master
- releases/*

pr:
- master
- milestones/*
- releases/*
- servicing/*

variables:
branchCounter: $[counter(variables['Build.SourceBranch'], 0)]

jobs:
- job: Windows_Build_and_UnitTests
variables:
platformFriendlyName: Windows
strategy:
maxParallel: 2
matrix:
debug:
configuration: Debug
release:
configuration: Release
pool:
vmImage: windows-2019
steps:
- powershell: $(Build.Repository.LocalPath)/Scripts/CI/Set-Version.ps1 -SourceBranchCounter $(branchCounter)
displayName: "Compute product version"
- template: templates/windows-build-and-unit-test.yml

- job: macOS_Build_and_UnitTests
variables:
platformFriendlyName: macOS
strategy:
maxParallel: 2
matrix:
debug:
configuration: Debug
release:
configuration: Release
pool:
name: 'Hosted macOS'
steps:
- powershell: $(Build.Repository.LocalPath)/Scripts/CI/Set-Version.ps1 -SourceBranchCounter $(branchCounter)
displayName: "Compute product version"
- template: templates/macos-build-and-unit-test.yml

- job: macOS_FunctionalTests
variables:
platformFriendlyName: macOS
configuration: Release
timeoutInMinutes: 30
pool:
name: 'Hosted macOS'
dependsOn: macOS_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # Use the drop from macOS_Build_and_UnitTests job
- template: templates/macos-functional-test.yml
parameters:
useWatchman: false

- job: macOS_FunctionalTests_watchman
variables:
platformFriendlyName: macOS
configuration: Release
timeoutInMinutes: 30
pool:
name: 'Hosted macOS'
dependsOn: macOS_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # Use the drop from macOS_Build_and_UnitTests job
- template: templates/macos-functional-test.yml
parameters:
useWatchman: true

- job: Windows_FunctionalTests
timeoutInMinutes: 30
variables:
platformFriendlyName: Windows
configuration: Release
pool:
vmImage: windows-2019
dependsOn: Windows_Build_and_UnitTests
condition: succeeded()
steps:
- checkout: none # We'll get the build drop from Windows_Build_and_UnitTests job
- template: templates/windows-functional-test.yml
37 changes: 22 additions & 15 deletions .azure-pipelines/templates/macos-build-and-unit-test.yml
Original file line number Diff line number Diff line change
@@ -1,35 +1,42 @@
steps:

- task: DotNetCoreInstaller@0
displayName: Use .NET Core SDK 2.1.301
- task: UseDotNet@2
displayName: Use .NET Core SDK 3.0.100
inputs:
packageType: sdk
version: '2.1.301'
version: 3.0.100

- script: $(Build.Repository.LocalPath)/Scripts/Mac/NukeBuildOutputs.sh
displayName: Delete previous build outputs
continueOnError: true

- task: InstallAppleCertificate@2
displayName: 'Install Apple certificate'
- task: InstallAppleCertificate@2
displayName: 'Install Apple certificate'
inputs:
certSecureFile: 'PrjFSKextCertExpiresJun6_2020_v2.p12'
certPwd: '$(kext.certificate.password)'
certSecureFile: 'PrjFSKextCertExpiresJun6_2020_v2.p12'
certPwd: '$(kext.certificate.password)'

- script: Scripts/Mac/BuildScalarForMac.sh $(configuration) $(majorAndMinorVersion).$(revision)
displayName: Build Scalar ($(configuration))

- script: Scripts/Mac/RunUnitTests.sh $(configuration) $(Common.TestResultsDirectory)
displayName: Run unit tests ($(configuration))

- task: PublishTestResults@2
displayName: Publish test results
inputs:
testRunner: NUnit
testResultsFiles: '**/TestResult.xml'
searchFolder: $(System.DefaultWorkingDirectory)
testRunner: VSTest
testResultsFiles: '**/*.trx'
searchFolder: $(Common.TestResultsDirectory)
testRunTitle: Mac $(configuration) Unit Tests
publishRunAttachments: true
condition: succeededOrFailed()

- script: Scripts/Mac/CI/CreateBuildDrop.sh $(configuration) $(Build.ArtifactStagingDirectory)/Tests
displayName: Create functional test drop
- script: Scripts/Mac/CI/CreateFTDrop.sh $(configuration) $(Build.ArtifactStagingDirectory)/Tests
displayName: Create functional tests drop

- task: PublishBuildArtifacts@1
displayName: Publish functional test drop artifact.
displayName: Publish functional tests drop
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)/Tests
artifactName: "FunctionalTests_$(platformFriendlyName)_$(configuration)"
Expand All @@ -38,10 +45,10 @@ steps:
condition: and(succeeded(), eq(variables['configuration'], 'Release'))

- script: Scripts/Mac/CI/CreateInstallerDrop.sh $(configuration) $(Build.ArtifactStagingDirectory)/Installers
displayName: Create installer drop
displayName: Create installers drop

- task: PublishBuildArtifacts@1
displayName: Publish installers
displayName: Publish installers drop
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)/Installers
artifactName: "Installers_$(platformFriendlyName)_$(configuration)"
Expand Down
60 changes: 0 additions & 60 deletions .azure-pipelines/templates/macos-functional-test-watchman.yml

This file was deleted.

68 changes: 45 additions & 23 deletions .azure-pipelines/templates/macos-functional-test.yml
Original file line number Diff line number Diff line change
@@ -1,45 +1,67 @@
parameters:
useWatchman: true

steps:

- task: DotNetCoreInstaller@0
displayName: Use .NET Core SDK 2.1.301
- task: UseDotNet@2
displayName: Use .NET Core SDK 3.0.100
inputs:
packageType: sdk
version: '2.1.301'

- bash: rm -rf $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/BuildOutput/Git/*
displayName: Clean previous Git installations
version: 3.0.100

- task: DownloadBuildArtifacts@0
displayName: Download functional test drop
displayName: Download functional tests drop
inputs:
buildType: current
downloadType: specific
artifactName: FunctionalTests_$(platformFriendlyName)_$(configuration)
downloadPath: $(Build.BinariesDirectory)

- task: DownloadBuildArtifacts@0
displayName: Download installers drop
inputs:
buildType: current
downloadType: specific
artifactName: Installers_$(platformFriendlyName)_$(configuration)
downloadPath: $(Build.BinariesDirectory)

- bash: |
chmod +x $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/src/Scripts/Mac/*.sh
chmod +x $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/Publish/*
displayName: Ensure tests assets are executable
chmod +x $(Build.BinariesDirectory)/Installers_$(platformFriendlyName)_$(configuration)/*.sh
displayName: Ensure all scripts are executable
- bash: $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/src/Scripts/Mac/CleanupFunctionalTests.sh
displayName: Clean environment
- ${{ if eq(parameters.useWatchman, 'true') }}:
- bash: $(Build.BinariesDirectory)/Installers_$(platformFriendlyName)_$(configuration)/InstallScalar.sh
displayName: Install product (with watchman)

- bash: $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/src/Scripts/Mac/PrepFunctionalTests.sh
displayName: Prep functional tests
- ${{ if ne(parameters.useWatchman, 'true') }}:
- bash: $(Build.BinariesDirectory)/Installers_$(platformFriendlyName)_$(configuration)/InstallScalar.sh --no-watchman
displayName: Install product (without watchman)

- bash: $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/src/Scripts/Mac/RunFunctionalTests.sh $(configuration) --trace2-output=$(Build.ArtifactStagingDirectory)/trace2-event-mac.txt
- bash: $(Build.BinariesDirectory)/FunctionalTests_$(platformFriendlyName)_$(configuration)/src/Scripts/Mac/RunFunctionalTests.sh $(configuration) --test-scalar-on-path --trace2-output=$(Build.ArtifactStagingDirectory)/trace2-event-mac.txt
displayName: Run functional tests

- task: PublishTestResults@2
displayName: Publish functional test results
inputs:
testRunner: NUnit
testResultsFiles: "**\\TestResult*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
testRunTitle: macOS $(configuration) Functional Tests
publishRunAttachments: true
condition: succeededOrFailed()
- ${{ if eq(parameters.useWatchman, 'true') }}:
- task: PublishTestResults@2
displayName: Publish functional tests results
inputs:
testRunner: NUnit
testResultsFiles: "**\\TestResult*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
testRunTitle: macOS $(configuration) Functional Tests (with watchman)
publishRunAttachments: true
condition: succeededOrFailed()

- ${{ if ne(parameters.useWatchman, 'true') }}:
- task: PublishTestResults@2
displayName: Publish functional tests results
inputs:
testRunner: NUnit
testResultsFiles: "**\\TestResult*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
testRunTitle: macOS $(configuration) Functional Tests (without watchman)
publishRunAttachments: true
condition: succeededOrFailed()

- task: PublishBuildArtifacts@1
displayName: Publish Git trace2 log
Expand Down
Loading

0 comments on commit 7b6f1cb

Please sign in to comment.