Aspire cli: Add AOT builds for Linux, and macOS #10275
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -112,11 +112,43 @@ extends: | |
|
|
||
| stages: | ||
|
|
||
| - stage: build_sign_native | ||
| displayName: Build+Sign native packages | ||
|
|
||
| jobs: | ||
| - template: /eng/pipelines/templates/build_sign_native.yml@self | ||
| parameters: | ||
| agentOs: macos | ||
| targetRidsForSameOS: | ||
| - osx-arm64 | ||
| - osx-x64 | ||
| codeSign: true | ||
| teamName: $(_TeamName) | ||
| extraBuildArgs: >- | ||
| /p:Configuration=$(_BuildConfig) | ||
| $(_SignArgs) | ||
| $(_OfficialBuildIdArgs) | ||
|
|
||
| - template: /eng/pipelines/templates/build_sign_native.yml@self | ||
| parameters: | ||
| agentOs: linux | ||
| targetRidsForSameOS: | ||
| - linux-x64 | ||
|
There was a problem hiding this comment. Isn't this missing arm64 and musl? There was a problem hiding this comment. We don't AOT those yet, so they are being built in the main Windows job instead of having separate agents to produce self-contained executables. There was a problem hiding this comment. Can you help me understand why? Why don't we aot those yet? I thought all we needed was to match the OS, so I would have expected that this same queue (the linux-x64 one) to be able to aot for those other two. There was a problem hiding this comment. We need to set the machine up for cross compliation. See Publish NativeAOT on Ubuntu doesn't work with -r linux-musl-x64 (dotnet/runtime#92294) and https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/cross-compile#linux There was a problem hiding this comment. I see, we should probably link those here where we say that a few of these are non aot'ed yet, but I don't think we should hold off this PR for that. There was a problem hiding this comment. The scripts PR has a README where we mention this - https://github.com/dotnet/aspire/pull/10254/files#diff-e2e2432bd7a67b477d58cdba86311ddd7d77d117ba1805bf70545131e380e0dd . There was a problem hiding this comment. .. and in the PR description. |
||
| # no need to sign ELF binaries on linux | ||
| codeSign: false | ||
| teamName: $(_TeamName) | ||
| extraBuildArgs: >- | ||
| /p:Configuration=$(_BuildConfig) | ||
| $(_SignArgs) | ||
| $(_OfficialBuildIdArgs) | ||
|
|
||
| # ---------------------------------------------------------------- | ||
| # This stage performs build, test, packaging | ||
| # ---------------------------------------------------------------- | ||
| - stage: build | ||
| displayName: Build | ||
| dependsOn: | ||
| - build_sign_native | ||
| jobs: | ||
| - template: /eng/common/templates-official/jobs/jobs.yml@self | ||
| parameters: | ||
|
|
@@ -160,6 +192,21 @@ extends: | |
| clean: true | ||
|
|
||
| steps: | ||
| - task: DownloadPipelineArtifact@2 | ||
| displayName: 🟣Download All Native Archives | ||
| inputs: | ||
| itemPattern: | | ||
| **/aspire-cli-*.zip | ||
| **/aspire-cli-*.tar.gz | ||
| targetPath: '$(Build.SourcesDirectory)/artifacts/packages/$(_BuildConfig)' | ||
|
|
||
| - task: PowerShell@2 | ||
| displayName: 🟣List artifacts packages contents | ||
| inputs: | ||
| targetType: 'inline' | ||
| script: | | ||
| Get-ChildItem -Path "$(Build.SourcesDirectory)\artifacts\packages" -File -Recurse | Select-Object FullName, @{Name="Size(MB)";Expression={[math]::Round($_.Length/1MB,2)}} | Format-Table -AutoSize | ||
|
|
||
| - template: /eng/pipelines/templates/BuildAndTest.yml | ||
| parameters: | ||
| dotnetScript: $(Build.SourcesDirectory)/dotnet.cmd | ||
|
|
@@ -169,40 +216,14 @@ extends: | |
| repoLogPath: $(Build.Arcade.LogsPath) | ||
| repoTestResultsPath: $(Build.Arcade.TestResultsPath) | ||
| isWindows: true | ||
| targetRids: | ||
|
There was a problem hiding this comment. Can you help me understand this? Why do we need to run build in all platforms when build native is happening in a previous step? Also, why are we removing the things like pool, etc? Finally, shouldn't now the There was a problem hiding this comment.
re:pools, that is dropping the old un-used job that ran on Linux. -
- - ${{ if eq(variables._RunAsPublic, True) }}:
- - job: Linux
- ${{ if or(startswith(variables['Build.SourceBranch'], 'refs/heads/release/'), startswith(variables['Bui>
- # If the build is getting signed, then the timeout should be increased.
- timeoutInMinutes: 120
- ${{ else }}:
- # timeout accounts for wait times for helix agents up to 30mins
- timeoutInMinutes: 90
-
- pool:
- name: NetCore1ESPool-Internal
- image: 1es-mariner-2
- os: linux
-
- variables:
- - name: _buildScript
- value: $(Build.SourcesDirectory)/build.sh --ci
-
- preSteps:
- - checkout: self
- fetchDepth: 1
- clean: true
-
- steps:
- - template: /eng/pipelines/templates/BuildAndTest.yml
- parameters:
- dotnetScript: $(Build.SourcesDirectory)/dotnet.sh
- buildScript: $(_buildScript)
- buildConfig: $(_BuildConfig)
- repoArtifactsPath: $(Build.Arcade.ArtifactsPath)
- repoLogPath: $(Build.Arcade.LogsPath)
- repoTestResultsPath: $(Build.Arcade.TestResultsPath)
- isWindows: falseThere was a problem hiding this comment. can you expand why win-x86, linux-arm64 and linux-musl are not aot? We of course would care for those to be aot as well. There was a problem hiding this comment. NativeAOT is not supported for win-x86. |
||
| # aot | ||
| - win-x64 | ||
| - win-arm64 | ||
| # non-aot - single file builds | ||
| - win-x86 | ||
| - linux-arm64 | ||
| - linux-musl-x64 | ||
|
|
||
| - ${{ if and(notin(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main')) }}: | ||
| - template: /eng/common/templates-official/job/onelocbuild.yml@self | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| parameters: | ||
| # values: windows/mac/linux | ||
| agentOs: 'windows' | ||
| targetRidsForSameOS: | ||
| - linux-x64 | ||
| extraBuildArgs: '' | ||
| codeSign: false | ||
| teamName: '' | ||
|
|
||
| jobs: | ||
|
|
||
| - ${{ each targetRid in parameters.targetRidsForSameOS }}: | ||
| - template: /eng/common/templates-official/jobs/jobs.yml@self | ||
| parameters: | ||
| enableMicrobuild: ${{ eq(parameters.codeSign, true) }} | ||
| enableMicrobuildForMacAndLinux: ${{ and(eq(parameters.codeSign, true), ne(parameters.agentOs, 'windows')) }} | ||
| enableTelemetry: true | ||
| # Publish build logs | ||
| enablePublishBuildArtifacts: true | ||
|
|
||
| jobs: | ||
| - job: BuildNative_${{ replace(targetRid, '-', '_') }} | ||
| displayName: ${{ replace(targetRid, '-', '_') }} | ||
| timeoutInMinutes: 40 | ||
|
|
||
| variables: | ||
| - TeamName: ${{ parameters.teamName }} | ||
| - ${{ if eq(parameters.codeSign, true) }}: | ||
| - _buildArgs: '--sign' | ||
| - ${{ else }}: | ||
| - _buildArgs: '' | ||
|
|
||
| - ${{ if eq(parameters.agentOs, 'windows') }}: | ||
| - scriptName: build.cmd | ||
| - ${{ else }}: | ||
| - scriptName: build.sh | ||
|
|
||
| pool: | ||
| ${{ if eq(parameters.agentOs, 'windows') }}: | ||
| name: NetCore1ESPool-Internal | ||
| image: windows.vs2022preview.amd64 | ||
| os: windows | ||
| ${{ if eq(parameters.agentOs, 'linux') }}: | ||
| name: NetCore1ESPool-Internal | ||
| image: 1es-mariner-2 | ||
| os: linux | ||
| ${{ if eq(parameters.agentOs, 'macos') }}: | ||
| name: Azure Pipelines | ||
| vmImage: macOS-latest-internal | ||
| os: macOS | ||
|
|
||
| preSteps: | ||
| - checkout: self | ||
| fetchDepth: 1 | ||
| clean: true | ||
|
|
||
| # Installing Microbuild plugin fails due to https://github.com/dotnet/arcade/issues/15946#issuecomment-3045780552 | ||
| # because of the preview sdk. To fix that `restore` from `global.json` so the above step | ||
| # does not have to install anything. | ||
| - script: $(Build.SourcesDirectory)/$(scriptName) -restore /p:Configuration=$(_BuildConfig) | ||
| displayName: 🟣Restore | ||
|
|
||
| steps: | ||
| - script: >- | ||
| $(Build.SourcesDirectory)/$(scriptName) | ||
| --ci | ||
| --build | ||
| --restore | ||
| /p:SkipManagedBuild=true | ||
| /p:TargetRids=${{ targetRid }} | ||
| $(_buildArgs) | ||
| ${{ parameters.extraBuildArgs }} | ||
| /bl:$(Build.Arcade.LogsPath)Build.binlog | ||
| displayName: 🟣Build native packages | ||
|
|
||
| - task: 1ES.PublishBuildArtifacts@1 | ||
| displayName: 🟣Publish Artifacts | ||
| condition: always() | ||
| inputs: | ||
| PathtoPublish: '$(Build.Arcade.ArtifactsPath)packages/' | ||
| ArtifactName: native_archives_${{ replace(targetRid, '-', '_') }} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this always be correct? I think there are more RIDs than the ones we currently support
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For instance, take a look at this issue: #5486.
I think if someone is building in those similar environments then they would now fail to find the .csproj and therefore fail to build. One thing we could do is to condition the ProjectToBuild addition to only in the case the project exists. Another alternative is to use the logic we have here: https://github.com/dotnet/aspire/blob/main/src/Aspire.AppHost.Sdk/Aspire.RuntimeIdentifier.Tool/Program.cs to pick the best rid (we support) based on the rid you are currently on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm this needs
NuGet.ProjectModel, and will need a custom task too. I can explore this in a follow up PR.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with waiting for a follow up PR, but I believe that the way you have it now it would actually break the build in some of those distros, so at the very least we should add some defensive code here for those cases. For instance, if you are in one of those distros like rhel, I suspect the build would fail trying to find a project called Aspire.Cli.rhel.....csproj which obviously won't exist.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While we personally might not build in those distros, we don't want to prevent folks working in those distros from building.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For this PR I added a check so we skip if the correct Cli project cannot be found.