Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-build: support building runtime using non-portable runtime packages. #75597

Merged
merged 8 commits into from
Oct 12, 2022

Conversation

tmds
Copy link
Member

@tmds tmds commented Sep 14, 2022

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

@ghost ghost added the community-contribution Indicates that the PR has been added by a community member label Sep 14, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

@tmds
Copy link
Member Author

tmds commented Sep 14, 2022

This isn't working fully.

For some reason ILCompiler.csproj and crossgen2.csproj go look for Microsoft.AspNetCore.App.Runtime.<rid>.

  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-x64 with version (= 7.0.0-rc.2.22452.11) [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102:   - Found 1 version(s) in previously-source-built [ Nearest version: 7.0.0-rc.1.22404.6 ] [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102:   - Found 0 version(s) in source-built [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102:   - Found 0 version(s) in reference-packages [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-x64 with version (= 7.0.0-rc.2.22452.11) [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102:   - Found 1 version(s) in previously-source-built [ Nearest version: 7.0.0-rc.1.22404.6 ] [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102:   - Found 0 version(s) in source-built [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102:   - Found 0 version(s) in prebuilt [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
  /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102:   - Found 0 version(s) in reference-packages [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]

I don't understand where this reference comes from, but probably we don't actually need this?
@ericstj @ViktorHofer how can I handle this?

@tmds
Copy link
Member Author

tmds commented Sep 15, 2022

For some reason ILCompiler.csproj and crossgen2.csproj go look for Microsoft.AspNetCore.App.Runtime..

Apparently the reference happens as soon as you add <RuntimeIdentifier> based on the KnownFrameworkReferences.
The fix is to remove the reference for Microsoft.AspNetCore.App.

@ghost
Copy link

ghost commented Sep 15, 2022

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author: tmds
Assignees: -
Labels:

area-Infrastructure, community-contribution

Milestone: -

@ghost
Copy link

ghost commented Sep 15, 2022

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author: tmds
Assignees: -
Labels:

area-Infrastructure-coreclr, community-contribution

Milestone: -

@ghost
Copy link

ghost commented Sep 15, 2022

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author: tmds
Assignees: -
Labels:

area-Infrastructure, community-contribution

Milestone: -

@tmds
Copy link
Member Author

tmds commented Sep 20, 2022

@ViktorHofer @jkotas I took a step back and wrote up an issue that describes the goal: dotnet/source-build#3027. And made a PR that includes these changes as a patch to help validate it: dotnet/installer#14549.

@ViktorHofer I will address your feedback this week.

Besides this, we also need #74721 (comment).

@ViktorHofer
Copy link
Member

@jkotas who would be the right person to review the ILCompiler and Crossgen2 changes?

@jkotas
Copy link
Member

jkotas commented Sep 20, 2022

@jkotas who would be the right person to review the ILCompiler and Crossgen2 changes?

I can take a look or route as necessary once the feedback is addressed.

Directory.Build.targets Outdated Show resolved Hide resolved
…kages.

Currently source-build performs a 'runtime-portable' build that produces
'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by
a previous (non-portable) 'runtime' build. This helps eliminate the
'runtime-portable' build.
@tmds
Copy link
Member Author

tmds commented Oct 11, 2022

CI failed because of an issue with the dnceng nuget feed.

@ViktorHofer can you trigger CI, and also take another look at this PR?

@build-analysis build-analysis bot mentioned this pull request Oct 11, 2022
2 tasks
@ViktorHofer ViktorHofer reopened this Oct 11, 2022
@ViktorHofer
Copy link
Member

Closing and reopening usually does the trick and retriggers all the legs.

@tmds
Copy link
Member Author

tmds commented Oct 12, 2022

@ViktorHofer since this has Condition=" '$(DotNetBuildFromSource)' == 'true' " and the integration test in dotnet/installer#14549 is passing, I think we can consider CI green on this.

@ViktorHofer
Copy link
Member

Added a style nit commit. Also, to retrigger CI as there were more than 20 failing legs.

@ViktorHofer ViktorHofer merged commit e2bee8a into dotnet:main Oct 12, 2022
@ViktorHofer
Copy link
Member

Thanks a lot, Tom!

@tmds
Copy link
Member Author

tmds commented Nov 9, 2022

@ViktorHofer can you initiate backports to 7.0 for this and the other 2 related PRs: #74504, #76068.
Can the bot handle these?

@ViktorHofer
Copy link
Member

We just merged some of these into release/6.0. Ideally, we would have backported these changes at the same into release/6.0 and release/7.0 as we now need another set of approvals.

That said, yes, I will cherry-pick them together into a single PR and add the servicing template and ask for an approval.

carlossanlop pushed a commit that referenced this pull request Nov 9, 2022
@tmds
Copy link
Member Author

tmds commented Nov 14, 2022

Sorry for the overhead, and thank you for taking care of it!

ViktorHofer added a commit that referenced this pull request Nov 15, 2022
…kages. (#75597)

* source-build: support building runtime using non-portable runtime packages.

Currently source-build performs a 'runtime-portable' build that produces
'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by
a previous (non-portable) 'runtime' build. This helps eliminate the
'runtime-portable' build.

* Avoid references to Microsoft.AspNetCore.App.<rid>.

* crossgen2: set RuntimeIdentifier explicitly.

* PR feedback

* Revert "crossgen2: set RuntimeIdentifier explicitly."

This reverts commit 10be274.

* PR feedback.

* Revert changes to ILCompiler.csproj.

* Update Directory.Build.targets

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
carlossanlop pushed a commit that referenced this pull request Nov 18, 2022
* Use generated runtime.json when building shared framework packages. (#76068)

* Use generated runtime.json when building shared framework packages.

* Don't UpdateRuntimeJson on Build.

* PR feedback.

* Update Microsoft.NETCore.Platforms.csproj

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

* source-build: support building runtime using non-portable runtime packages. (#75597)

* source-build: support building runtime using non-portable runtime packages.

Currently source-build performs a 'runtime-portable' build that produces
'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by
a previous (non-portable) 'runtime' build. This helps eliminate the
'runtime-portable' build.

* Avoid references to Microsoft.AspNetCore.App.<rid>.

* crossgen2: set RuntimeIdentifier explicitly.

* PR feedback

* Revert "crossgen2: set RuntimeIdentifier explicitly."

This reverts commit 10be274.

* PR feedback.

* Revert changes to ILCompiler.csproj.

* Update Directory.Build.targets

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

* source-build: set the properties to build for mono runtime. (#77786)

* Pass TargetRid and SourceBuildNonPortable to the native scripts (#74504)

* Pass SourceBuild TargetRid and SourceBuildPortable args through the native script.

* Rename -packagerid to -distrorid.

* Fix init-distro-rid.sh.

* Use OutputRid instead of PackageRid.

* Drop unnecessary Condition.

* Try passing --distrorid as OutputRid.

* NativeExport: use PackageRID for AppHostRuntimeIdentifier.

* Rename --distrorid to --outputrid.

* Fix SourceBuild.props.

* Undo changes to init-distro-rid.sh.

* Let source-build leg build 'banana-x64' rid.

* SourceBuild.props: don't force a RuntimeOS.

* SourceBuild.props: set AdditionalRuntimeIdentifierParent.

* Build banana.24-x64 instead.

* Fix SourceBuild.props.

* SourceBuild.props: try fix AdditionalRuntimeIdentifier* usage.

* source-build.yml: add runtimeOS parameter.

* SourceBuild.props: derive RuntimeOS from NETCoreSdkRuntimeIdentifier.

* Undo using NETCoreSdkRuntimeIdentifier.

* Update eng/pipelines/common/global-build-job.yml

Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com>

Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com>

Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com>
Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com>
@ghost ghost locked as resolved and limited conversation to collaborators Dec 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure community-contribution Indicates that the PR has been added by a community member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants