Use and support OpenSSL 3 on macOS

[vcsjones]

This changes macOS builds and runs to use OpenSSL 3, as well as support it for pkg-config.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This changes macOS builds and runs to use OpenSSL 3, as well as support it for pkg-config.

Author:	vcsjones
Assignees:	-
Labels:	area-Infrastructure
Milestone:	-

[EgorBo]

should it update https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/macos-requirements.md doc too?

[vcsjones]

@EgorBo Good thinking. Thank you!

[danmoseley]

Somehow libraries_test_assets_OSX_x64_Debug.tar.gz is corrupt. I verified this locally as well. The issue is either in the tar step or the upload/download. There is no error in the build where it is created and uploaded though.

2022-04-14T20:40:22.8805630Z Found 1 files
2022-04-14T20:40:22.8807720Z Archiving file: helix
2022-04-14T20:40:22.8850840Z [command]/usr/bin/tar -c --gz -f /Users/runner/work/1/a/libraries_test_assets_OSX_x64_Debug.tar.gz helix
2022-04-14T20:40:39.6572190Z ##[section]Finishing: Zip Test Assets

Uploading 'libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz' (98%)
File upload succeed.
Upload '/Users/runner/work/1/a/libraries_test_assets_OSX_x64_Debug.tar.gz' to file container: '#/10179179/libraries_test_assets_OSX_x64_Debug'
Associated artifact 33080478 with build 1718532

nor on download, but there's 3 places its downloaded and all are failing so that's presumably fine.

Downloading items from container resource #/10179179/libraries_test_assets_OSX_x64_Debug
Downloading artifact libraries_test_assets_OSX_x64_Debug from: https://dev.azure.com/dnceng//_apis/resources/Containers/10179179?itemPath=libraries_test_assets_OSX_x64_Debug&isShallow=true&api-version=4.1-preview.4
Downloading libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz to /Users/runner/work/1/s/__download__/libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz
Downloaded libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz to /Users/runner/work/1/s/__download__/libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz
Total Files: 1, Processed: 1, Skipped: 0, Failed: 0, Download time: 50.762 secs, Download size: 500.621MB
Starting artifact items integrity check
Artifact items integrity check successfully finished
Successfully downloaded artifacts to /Users/runner/work/1/s/__download__

then they all fail with

2022-04-14T20:46:00.2919370Z error: undefined;
2022-04-14T20:46:00.3028700Z ##[error]Extraction failed for file: /Users/runner/work/1/s/__download__/libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz 

@MattGal have you ever seen something like this?
Any idea about this?

[danmoseley]

Maybe after zipping we should have something like tar -tzf foo.gz >/dev/null which I guess would error if it was on that end.

doing that manually I get this -- suggesting the file just wasn't completely uploaded.

gzip: stdin: invalid compressed data--format violated
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

[vcsjones]

This has been seen before in #67159 so at least there appears to be a tracking issue for it.

[danmoseley]

Well I clicked rerun failed jobs, in case you can't @vcsjones

[vcsjones]

Thanks @danmoseley. Looks like it failed again, I wonder if those re-runs just re-used whatever bad asset was produced. I'll push up an empty commit.

[danmoseley]

Starting:    System.Text.Json.SourceGeneration.Roslyn3.11.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.
   at System.Text.Json.JsonSerializer.LookupProperty(Object obj, ReadOnlySpan`1 unescapedPropertyName, ReadStack& state, JsonSerializerOptions options, Boolean& useExtensionProperty, Boolean createExtensionProperty) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.HandlePropertyName.cs:line 36
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) in /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectDefaultConverter.cs:line 63

I think this is the one that @krwq is currently root causing.

[vcsjones]

This looks good to me, the System.Text.Json failure passed on re-run and is being tracked by #67816. I will leave it to the area owners to merge in case they want to consider any optional pipeline runs before merging (though I myself cannot think of any that would be useful).

[danmoseley]

I'm not sure who has context here to review. @wfurt?

[wfurt]

Is OpenSSL3 available on all supported versions @vcsjones. And what happen to uses who upgrade .NET but not existing OpenSSL?

[vcsjones]

Is OpenSSL3 available on all supported versions @vcsjones.

No version of macOS comes with a useable version of OpenSSL. We still need it for building locally though. So this PR allows using OpenSSL 3, in addition to 1.1, for local builds. At run time, it's an optional dependency. If present, it lights up some functionality like RSAOpenSsl, AesGcm, etc.

And what happen to uses who upgrade .NET but not existing OpenSSL?

Let me double check that this works. I think we use a portable build for macOS, so it should just "work". In the mean time, I'll mark this as draft to verify that scenario.

[wfurt]

Is OpenSSL3 available on all supported versions @vcsjones.

No version of macOS comes with a useable version of OpenSSL. We still need it for building locally though. So this PR allows using OpenSSL 3, in addition to 1.1, for local builds. At run time, it's an optional dependency. If present, it lights up some functionality like RSAOpenSsl, AesGcm, etc.

And what happen to uses who upgrade .NET but not existing OpenSSL?

Let me double check that this works. I think we use a portable build for macOS, so it should just "work". In the mean time, I'll mark this as draft to verify that scenario.

Thanks. If we can verify that that it does not break existing uses I'm fine with moving forward.

[vcsjones]

And what happen to uses who upgrade .NET but not existing OpenSSL?

Developers today have to tell us which version of OpenSSL they want to use. If they wanted to use .NET 6 today with OpenSSL 3 on macOS, they could:

DYLD_LIBRARY_PATH=/opt/homebrew/opt/openssl@3/lib dotnet run  

So all this change is doing is changing what we install with CI and what headers are available for compiling.

If we want to wait for @bartonjs to confirm I'm not overlooking anything, I'm okay with that.

[wfurt]

LGTM

[bartonjs]

Belatedly, this seems OK to me 😄.