Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dogfood .NET6 SDK & use native SDK for Apple Silicon native builds (WIP) #47245

Closed
wants to merge 3 commits into from

Conversation

sdmaclea
Copy link
Contributor

@sdmaclea sdmaclea commented Jan 20, 2021

This doesn't work yet.... Looking for some help...

I see lots of errors like this

/Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj : 
  error NU1012: 
  Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-ios, net6.0-tvos [/Users/stmaclea/git/runtime/Build.proj]
  Failed to restore /Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj (in 20 ms).

I am not sure about the osx-x64 error, but it might also be a SDK issue.

/cc @vitek-karas @ViktorHofer @janvorli @mangod9 @jkoritzinsky

Experiment related to #46960

@sdmaclea sdmaclea added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) area-Infrastructure labels Jan 20, 2021
@sdmaclea sdmaclea added this to the 6.0.0 milestone Jan 20, 2021
@sdmaclea sdmaclea self-assigned this Jan 20, 2021
@ghost
Copy link

ghost commented Jan 20, 2021

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

This doesn't work yet.... Looking for some help...

On osx-x64, I see lots of errors like this

/Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj : 
  error NU1012: 
  Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-ios, net6.0-tvos [/Users/stmaclea/git/runtime/Build.proj]
  Failed to restore /Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj (in 20 ms).

On osx-arm64 native M1 build

/usr/local/share/dotnet/sdk/6.0.100-alpha.1.21070.6/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(99,5): 
  error NETSDK1084: 
  There is no application host available for the specified RuntimeIdentifier 'osx-arm64'. 

The missing app-host looks like something is missing from the SDK and/or runtime to support osx-arm64 completeley.

I am not sure about the osx-x64 error, but it might also be a SDK issue.

/cc @vitek-karas @ViktorHofer @janvorli @mangod9 @jkoritzinsky

Author: sdmaclea
Assignees: sdmaclea
Labels:

* NO MERGE *, area-Infrastructure

Milestone: 6.0.0

@sdmaclea sdmaclea marked this pull request as draft January 20, 2021 20:31
@vitek-karas
Copy link
Member

The package Microsoft.NetCore.DotnetAppHost is missing the osx-arm64 variant (the actual package should be called runtime.osx-arm64.Microsoft.NetCore.DotnetAppHost).
I think these packaged are creates in the runtime repo (I would guess in the installer leg), but I don't know the details on that.

@sdmaclea
Copy link
Contributor Author

The apphost appears to be present

/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-arm64/6.0.0-alpha.1.21069.7/runtimes/osx-arm64/native/apphost
/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-arm64/6.0.0-alpha.1.21069.7/runtimes/osx-arm64/native/singlefilehost
/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-arm64/6.0.0-alpha.1.21069.7/runtimes/osx-arm64/native/libnethost.dylib
/usr/local/share/dotnet/sdk/6.0.100-alpha.1.21070.6/AppHostTemplate/apphost

@sdmaclea
Copy link
Contributor Author

Thanks @vitek-karas I'll looks for that code.

@sdmaclea
Copy link
Contributor Author

/cc @Anipik

@jkoritzinsky
Copy link
Member

Is it possible that the KnownFrameworkReference for the apphost pack hasn’t had its RuntimeIdentifiers list updated to include osx-arm64 in the SDK.

@sdmaclea
Copy link
Contributor Author

@jkoritzinsky There are a lot of places where osx-arm64 is missing in src/installers. I am working on a PR.

@sdmaclea
Copy link
Contributor Author

In case it is not clear the osx-x64 error above is for building osx-x64 on osx-x64.

@jkoritzinsky
Copy link
Member

The update that I mentioned might be needed would be in dotnet/sdk.

@sdmaclea
Copy link
Contributor Author

@jkoritzinsky I don't see any relevant osx references which need fixing for osx-arm64 in dotnet\sdk except in tests. And I don't think even the tests need fixes.

@jkoritzinsky
Copy link
Member

It looks like what I thought might be the issue has already been handled.

@@ -57,7 +57,7 @@ if [[ "$cpuname" == "unknown" ]]; then
fi

case $cpuname in
aarch64)
aarch64|arm64)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

global.json Outdated
"allowPrerelease": true,
"rollForward": "major"
},
"tools": {
"dotnet": "5.0.100"
"dotnet": "6.0.100-alpha.1.21070.6"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't intent do commit the SDK update, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't intend to commit this PR. It was more of an experiment to see if the .NET 6 SDK would work (especially on native Apple Silicon because NET 5 doesn't support Apple Silicon native.)

@jkoritzinsky has suggested we might commit something like this to solve the crossgen2 issues @trylek is working on.

@sdmaclea
Copy link
Contributor Author

Thanks to @jkoritzinsky, this version of the PR, the osx-arm64 build has similar symptoms to other builds.

For some reason, the element:

<TargetFramework>net6.0-Unix</TargetFramework>

is causing the libs.src & libs.tests build to fail with the message (same as above):

/Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj : 
  error NU1012: 
  Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-ios, net6.0-tvos [/Users/stmaclea/git/runtime/Build.proj]
  Failed to restore /Users/stmaclea/git/runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System.Security.Cryptography.X509Certificates.csproj (in 20 ms).

I guess I don't know how it works. I tried targeting net5.0-Unix with the 5.0.100 SDK helloWorld and it failed too.

@ViktorHofer
Copy link
Member

@Anipik can you help @sdmaclea with the error that he is seeing when upgrading to a 6.0 SDK? Sounds like it's related to the TargetFramework.sdk.

@mangod9
Copy link
Member

mangod9 commented Jan 27, 2021

Assume there are no concerns with upgrading the sdk? Given that preview1 is now forked would be ideal to start consuming those bits in main.

@trylek
Copy link
Member

trylek commented Jan 27, 2021

Please note that upgrading the SDK to its current main won't solve the CG2 framework compilation in its entirety. As Viktor pointed out on the framework switch-over PR thread, limitations of the current implementation in the SDK repo forced me to use hacky constructs like globally exposing the DOTNET_ROOT environment variable across the entire build and we shouldn't be doing that as that's exactly the kind of thing that adds debt and messiness to the build infra. I'm currently working on a change against the SDK repo to fix this cleanly and I hope to be able to send it out for PR today or tomorrow; for Crossgen2 purposes it would be ideal to then roll forward to the SDK drop containing my change currently in progress and make the framework CG2 switch-over on top of that.

@mangod9
Copy link
Member

mangod9 commented Jan 27, 2021

ok thats fair. So please update when the sdk changes are done. Would we want to port the changes to preview1 to pick that sdk build?

@ViktorHofer
Copy link
Member

We definitely would not want to use a random nightly SDK build. I assume porting the change to preview1 is already too late so I would assume the next applicable one would be preview2 which will take at least another month. That said, maybe we can still get Tomas's change into P1, that would be fantastic.

@mangod9
Copy link
Member

mangod9 commented Jan 28, 2021

if the sdk fix is straight forward might be good to get that into p1 so we can get that integrated.

@steveisok steveisok mentioned this pull request Feb 4, 2021
3 tasks
@sdmaclea
Copy link
Contributor Author

sdmaclea commented Feb 4, 2021

@Anipik can you help @sdmaclea with the error that he is seeing when upgrading to a 6.0 SDK? Sounds like it's related to the TargetFramework.sdk.

Updated the runtime reference to use the latest preview1 candidate.

@Anipik If you have a chance it would be nice to see this working correctly with the preview1 candidate.

@Anipik
Copy link
Contributor

Anipik commented Feb 5, 2021

@Anipik If you have a chance it would be nice to see this working correctly with the preview1 candidate.

sure i will take a look

@ViktorHofer
Copy link
Member

ping @Anipik

C:\git\runtime\src\libraries\System.Net.Http\src\System.Net.Http.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-linux, net6.0-osx, net6.0-freebsd, net6.0-maccatalyst, net6.0-ios, net6.0-tvos, net6.0-browser, net6.0-illumos, net6.0-solaris [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.HttpListener\src\System.Net.HttpListener.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Linq\src\System.Linq.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-browser, net6.0-android, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.Pipes\src\System.IO.Pipes.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.MemoryMappedFiles\src\System.IO.MemoryMappedFiles.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.IsolatedStorage\src\System.IO.IsolatedStorage.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.FileSystem\src\System.IO.FileSystem.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.FileSystem.Watcher\src\System.IO.FileSystem.Watcher.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-linux, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos, net6.0-freebsd [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Cryptography.OpenSsl\src\System.Security.Cryptography.OpenSsl.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Cryptography.Encoding\src\System.Security.Cryptography.Encoding.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Cryptography.Csp\src\System.Security.Cryptography.Csp.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Cryptography.Algorithms\src\System.Security.Cryptography.Algorithms.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-android, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Runtime.Serialization.Formatters\src\System.Runtime.Serialization.Formatters.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Runtime.InteropServices.RuntimeInformation\src\System.Runtime.InteropServices.RuntimeInformation.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\Microsoft.Win32.Primitives\src\Microsoft.Win32.Primitives.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Drawing.Common\src\System.Drawing.Common.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.DirectoryServices.Protocols\src\System.DirectoryServices.Protocols.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-osx, net6.0-linux [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Diagnostics.Process\src\System.Diagnostics.Process.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-freebsd, net6.0-linux, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Diagnostics.FileVersionInfo\src\System.Diagnostics.FileVersionInfo.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Principal.Windows\src\System.Security.Principal.Windows.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Security.Cryptography.X509Certificates\src\System.Security.Cryptography.X509Certificates.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Private.Runtime.InteropServices.JavaScript\src\System.Private.Runtime.InteropServices.JavaScript.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.WebSockets.Client\src\System.Net.WebSockets.Client.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.WebClient\src\System.Net.WebClient.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Sockets\src\System.Net.Sockets.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Security\src\System.Net.Security.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Requests\src\System.Net.Requests.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.FileSystem.DriveInfo\src\System.IO.FileSystem.DriveInfo.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.Compression\src\System.IO.Compression.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.IO.Compression.Brotli\src\System.IO.Compression.Brotli.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Data.Odbc\src\System.Data.Odbc.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-freebsd, net6.0-illumos, net6.0-solaris, net6.0-linux, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Console\src\System.Console.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-android, net6.0-maccatalyst, net6.0-ios, net6.0-tvos, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Quic\src\System.Net.Quic.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-linux, net6.0-osx, net6.0-freebsd, net6.0-maccatalyst, net6.0-ios, net6.0-tvos [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Primitives\src\System.Net.Primitives.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Ping\src\System.Net.Ping.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.NetworkInformation\src\System.Net.NetworkInformation.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-linux, net6.0-osx, net6.0-maccatalyst, net6.0-ios, net6.0-tvos, net6.0-freebsd, net6.0-illumos, net6.0-solaris [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.Mail\src\System.Net.Mail.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]
C:\git\runtime\src\libraries\System.Net.NameResolution\src\System.Net.NameResolution.csproj : error NU1012: Platform version is not present for one or more target frameworks, even though they have specified a platform: net6.0-unix, net6.0-browser [C:\git\runtime\Build.proj]

@Anipik
Copy link
Contributor

Anipik commented Feb 11, 2021

@ViktorHofer @sdmaclea i fixed the failure here.
Apparently nuget changed how the treat the empty version. targetplatformVersion is redundant string for us and can be set to anything. I tested the change locally, i will keep a look at the build as well

@sdmaclea
Copy link
Contributor Author

I never would have found that, thanks @Anipik

@Anipik
Copy link
Contributor

Anipik commented Feb 11, 2021

I never would have found that, thanks @Anipik

welcome :)

@ViktorHofer
Copy link
Member

@Anipik would you mind submitting the fix as a separate PR so that we unblock devs who already use a newer SDK?

@sdmaclea I just created the March rollout issue which lists updating the minimum required SDK to 6.0 P1: #48168. The planned roll out date for that is March 1st.

@sdmaclea
Copy link
Contributor Author

Looks like

  • Some of the build legs failed uploading to blob storage (I reran failed job to see if these were transient)
  • all the helix runs failed.

@mangod9
Copy link
Member

mangod9 commented Feb 11, 2021

@ViktorHofer the 6.0 sdk is required for crossgen2 work as well, since 3/1 is still a few weeks away, is there concern about getting it integrated before then (assuming all issues are resolved)?

@ViktorHofer
Copy link
Member

Updating the minimum required SDK is defined as a breaking change and needs to go through a monthly rollout. Unfortunately there is no way around that. Another thing to be aware of is that we don't want to depend on a non publicly released SDK as the minimum supported version. There is a document about this in Arcade which I can't find right now.

As 6.0 Preview 1 isn't released yet, it will probably just be a few days between that SDK being released and March 1st. Hope that is helpful...

@mangod9
Copy link
Member

mangod9 commented Feb 11, 2021

Ok thanks for the context, will check with @trylek and if this is going to cause too much of a delay we might have to ask for an exception.

@trylek
Copy link
Member

trylek commented Feb 11, 2021

Hmm, this is complicated. What I'm getting out of this discussion is as follows:

  • I'm currently testing my CG2 SDK changes after privately applying Steve's and Anirudh's changes. Let's hypothetically assume the change against the SDK repo can go in before the end of next week (I'm very hopeful, it seems to be almost working already.)
  • Next Friday i.e. 2/19 we'll snap off Preview 2.
  • On 3/1, we'll roll forward SDK / tools drop to use 6.0 preview 1. So that change will go in Preview 3.

Based on similar arguments, we'll only be able to further roll forward the SDK drop to Preview2 (containing my change currently in progress) only after Preview2 has shipped. Perhaps we'll again need to sync up with the monthly infra rollout cadence. Long story short, not ideal, we would to wait for SDK CG2 switchover until something like Preview 4 or so which is way too long.

In light of this I believe we need a stopgap solution. I'll go over my current CG2 SDK PR and I'll try to figure out whether there's an easy way to address Viktor's concerns to let us use the existing SDK for the time being and move forward with testing, perf / size measurements and ASP.NET switchover. Once the newer SDK lights up, all that will remain is a small mop-up change to clean up some of the scripts using the new SDK functionality.

@sdmaclea sdmaclea closed this Feb 26, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Mar 28, 2021
@sdmaclea sdmaclea deleted the build_with_arm64 branch June 10, 2021 00:31
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants