HTTPS requests fail on Ubuntu 24.04 Noble ARM32 due to bundled certs "NotTimeValid" error

[lbussell]

Update: A final resolution to this issue has been posted at dotnet/core#9285.

Description

HTTPS requests from .NET are failing on Ubuntu 24.04 Noble on ARM32 due to bundled certs "NotTimeValid" errors.

I believe this may be because 24.04 has migrated to 64-bit time. The OpenSSL package has changed from libssl3 to libssl3t64.

Reproduction Steps

1. dotnet new console
2. dotnet add package System.Text.Json
3. .NET will try to restore the package from NuGet.org and fail with the error under "Actual behavior" below.

I created this repro Dockerfile. It may or may not work since I had issues with it on my AMD64 dev machine. No easy way for me to test this currently outside of our official .NET Container image builds. With some trial and error you can probably get the issue to repro on a real arm32 machine using Docker. https://gist.github.com/lbussell/52e0ac904108d238d0e511f8b6ec89e1

Expected behavior

The .NET CLI should successfully access NuGet.org.

Actual behavior

From our PR validation in .NET Docker: https://dev.azure.com/dnceng-public/public/_build/results?buildId=652911&view=logs&j=7bc65791-3246-5ca2-874f-59d2e579cf6b&t=08651ed6-ba3f-5f8a-52df-50083ac157c2&l=942

 #10 2.407 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
 #10 2.491 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
 #10 2.494 info : Adding PackageReference for package 'foo' into project '/source/app/app.csproj'.
 #10 8.602 error: Unable to load the service index for source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json.
 #10 8.602 error:   The SSL connection could not be established, see inner exception.
 #10 8.602 error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Regression?

No response

Known Workarounds

No response

Configuration

• .NET Runtime: 8.0.4 and 9.0.0-preview.4.24218.7
• OS: Ubuntu 24.04 "Noble"
• Arch: ARM
• Do you know whether it is specific to that configuration? Yes it is specific to this config. Likely affects older .NET versions as well.

Other information

Discovered in dotnet/dotnet-docker#5241

Known build error template

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=674655&view=results
Build error leg or test failing:
Pull request:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["arm32v7", "The remote certificate is invalid because of errors in the certificate chain: NotTimeValid"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=674655
Error message validated: [arm32v7 The remote certificate is invalid because of errors in the certificate chain: NotTimeValid]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/16/2024 3:23:33 PM UTC

Report

Build	Definition	Test	Pull Request
674655	dotnet/runtime	System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.ReceiveAsync_MultipleOutstandingReceiveOperations_Throws

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

[vcsjones]

I can take a look (and finally have an excuse to take that Raspberry Pi out of the box!)

[vcsjones]

Firstly

1. I can reproduce it

   Distributor ID:	Ubuntu
   Description:	Ubuntu Noble Numbat (development branch)
   Release:	24.04
   Codename:	noble
   root@1dc47c31f061:/project# dotnet add package System.Text.Json
     Determining projects to restore...
     Writing /tmp/tmpiHt9zR.tmp
   info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
   info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
   info : Adding PackageReference for package 'System.Text.Json' into project '/project/project.csproj'.
   error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
   error:   The SSL connection could not be established, see inner exception.
   error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid
   

2. .NET "knows" it is the right time. So it's not like the clock is wrong, or that the source of time is wrong.

   root@1dc47c31f061:/project# date
   Tue Apr 23 21:40:27 EDT 2024
   root@1dc47c31f061:/project# dotnet run
   Hello, World!04/23/2024 21:40:37
   root@1dc47c31f061:/project#
   

3. OpenSSL CLI is okay with the host. So, it's not like the certificate really is expired or there is some underlying problem in OpenSSL.

   No client certificate CA names sent
   Peer signing digest: SHA256
   Peer signature type: RSA-PSS
   Server Temp Key: ECDH, prime256v1, 256 bits
   ---
   SSL handshake has read 4273 bytes and written 441 bytes
   Verification: OK
   ---
   

4. The X509Certificate2 is able to load the notBefore and notAfter correctly. (.NET and OpenSSL differ by my timezone offset since .NET is showing it in local time and OpenSSL is showing it in UTC time)

   root@d2b82c600199:/project# dotnet run
   10/18/2023 16:00:08
   10/12/2024 16:00:08
   Hello, World!
   root@d2b82c600199:/project# openssl x509 -in cert.pem -text -noout
   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               33:00:0d:53:71:cb:65:3d:f3:ee:7c:dc:5c:00:00:00:0d:53:71
           Signature Algorithm: sha384WithRSAEncryption
           Issuer: C = US, O = Microsoft Corporation, CN = Microsoft Azure RSA TLS Issuing CA 07
           Validity
               Not Before: Oct 18 20:00:08 2023 GMT
               Not After : Oct 12 20:00:08 2024 GMT
   

This will take a bit of additional time to troubleshoot, whatever is going on is not immediately obvious (to me)

[dviererbe]

@vcsjones libssl3t64 changed the ABI compared to libssl3. This may cause the error.

[vcsjones]

Yep. I am trying to narrow it down to the smallest possible place to start looking for ABI mismatches.

It does reproduce with X509Chain.Build. So it is something specifically with chain building. Reading the dates off the individual certificates seems okay.

Right now I believe this code to be problematic under the ABI change.

runtime/src/native/libs/System.Security.Cryptography.Native/openssl.c

Lines 953 to 968 in 5661009

	time_t verifyTime = MakeTimeT(year, month, day, hour, minute, second, isDst);
	if (verifyTime == (time_t)-1)
	{
	return 0;
	}
	X509_VERIFY_PARAM* verifyParams = X509_STORE_get0_param(ctx);
	if (!verifyParams)
	{
	return 0;
	}
	X509_VERIFY_PARAM_set_time(verifyParams, verifyTime);
	return 1;

We always set the verification time when building the chain:

runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs

Line 120 in 5661009

Interop.Crypto.X509StoreSetVerifyTime(store, verificationTime);

Next steps is I need to figure out how to get a debugger going here.

[bartonjs]

@AndyAyersMS On ARM32, if time_t is a 64-bit value then it gets passed using the stack pointer offset convention rather than as a register, right? That suggests to me that we need to build/distribute an entirely different runtime for arm32+time_t64 and arm32+time_t32.

Unless our call to X509_VERIFY_PARAM_set_time is literally the only place in all of .NET that a time_t is passed by value, in which case I think we can use the time(2) function to detect the size of time_t at runtime:

uint64_t timeOrBigger = 0xFFFFFFFFFFFFFFFF;
time((time_t*)&timeOrBigger);

if (timeOrBigger >> 32 == 0xFFFFFFFF)
{
    // Either a: the top 32 bits didn't get written, or
    // b: it's approximately the year 128,000,002,000 and all times start with that pattern.
    //
    // Assume a, and be happy
    // normal mktime, normal call.
}
else
{
    uint64_t (*mktime_ptr)(struct tm*) = (whatever the cast looks like)(void*)mktime;
    uint64_t timeval = mktime_ptr(&tm);
    
    void (*hacky)(X509_VERIFY_PARAM*, uint64_t) = (cast)(void*)X509_VERIFY_PARAM_set_time;
    hacky(verifyParams, timeval);

}

But I highly suspect that this isn't the only thing broken right now... just the one people see first. (And I'm concerned that it means any function called that wants a time_t is going to be reading arbitrary stack state from the caller, which seems... bad).

[bartonjs]

runtime/src/coreclr/pal/src/file/filetime.cpp

Lines 229 to 265 in ce0ae49

	BOOL PALAPI FileTimeToSystemTime( CONST FILETIME * lpFileTime,
	LPSYSTEMTIME lpSystemTime )
	{
	UINT64 FileTime = 0;
	time_t UnixFileTime = 0;
	struct tm * UnixSystemTime = 0;
	/* Combine the file time. */
	FileTime = lpFileTime->dwHighDateTime;
	FileTime <<= 32;
	FileTime |= (UINT)lpFileTime->dwLowDateTime;
	bool isSafe = ClrSafeInt<UINT64>::subtraction(
	FileTime,
	SECS_BETWEEN_1601_AND_1970_EPOCHS * SECS_TO_100NS,
	FileTime);
	if (isSafe == true)
	{
	#if HAVE_GMTIME_R
	struct tm timeBuf;
	#endif /* HAVE_GMTIME_R */
	/* Convert file time to unix time. */
	if (((INT64)FileTime) < 0)
	{
	UnixFileTime = -1 - ( ( -FileTime - 1 ) / 10000000 );
	}
	else
	{
	UnixFileTime = FileTime / 10000000;
	}
	/* Convert unix file time to Unix System time. */
	#if HAVE_GMTIME_R
	UnixSystemTime = gmtime_r( &UnixFileTime, &timeBuf );
	#else /* HAVE_GMTIME_R */
	UnixSystemTime = gmtime( &UnixFileTime );
	#endif /* HAVE_GMTIME_R */

sends a time_t* into gmtime/gmtime_r, which means (assuming this file is used on arm32) it's getting the 32 bits we assigned, and the 32 bits declared after (before?) it on the stack. Which might be our conveniently zero value for UnixSystemTime, or might be part of FileTime, or might just be garbage.

[vcsjones]

Next steps is I need to figure out how to get a debugger going here.

Basically we are sending un-initialized data to X509_VERIFY_PARAM_set_time. The problem actually starts further up here:

runtime/src/native/libs/System.Security.Cryptography.Native/openssl.c

Line 64 in 5661009

return mktime(&currentTm);

We return a time_t just like mktime, but because the convention of how the value is returned, the value is incorrect. During my debugging session I saw various "random" values passed to X509_VERIFY_PARAM_set_time, but all of them in the future. That results in the chain build failing with a status of X509_V_ERR_CERT_HAS_EXPIRED.

[AndyAyersMS]

@AndyAyersMS On ARM32, if time_t is a 64-bit value then it gets passed using the stack pointer offset convention rather than as a register, right? That suggests to me that we need to build/distribute an entirely different runtime for arm32+time_t64 and arm32+time_t32.

Seems likely, but I would have to read up on the arm32 ABI. Perhaps @jakobbotsch knows as he's been working on this area recently...

[jakobbotsch]

ARM32 never passes parameters by reference, they are always passed by value. What is the definition of time_t in this case? Depending on whether this type has float/double fields it has special ABI rules on arm32.

[filipnavara]

I don't think the problem is the (ARM32) ABI per-se but the fact whether time_t is defined as 64-bit (__TIMESIZE == 64 / _TIME_BITS == 64) when compiling the OpenSSL or not, and whether that matches the time_t as defined when compiling the native shim. Even if it's passed by value, the non-matching definition would occupy a different number of registers.

Effectively, if the system libraries start to use a different time_t definition on various platforms we are screwed. That said, I don't think it's the case for most system APIs. IIRC for glibc there's an exported localtime and localtime64 and _TIME_BITS merely controls which one is used through C defines (similar for other APIs). If OpenSSL exports a single symbol that behaves differently then it's most unfortunate but I don't think the same problem translates to every time_t API.

[bartonjs]

What is the definition of time_t in this case?

When we built the arm32 portable build:

typedef int32_t time_t;

The particular flavors of Linux:

typedef int64_t time_t;

When thinking this might just be limited to us calling into OpenSSL, I was trying to figure out whether we could just force the data to work out; so I was trying to figure out the calling convention for a 64-bit integer being passed around; and I waffled a bit between whether it was "adjacent registers" or the magical thing with relative stack (which might be the same as pass by reference); particularly when the long long/int64_t is the second parameter, after a pointer.

If I understand the Godbolt output correctly, it does look like we could get away with always treating time_t as int64_t on arm32 for calling X509_VERIFY_PARAM_set_time, and it would guarantee that r3 gets set to 0 rather than is just left floating. (When compiling without -O (to prevent inlining) I guess I got confused by the preamble... optimize except no inlining seems to be the way to go)

It looks like an int64_t return might be spread across r0 and r1, so we could also reshape the call to mktime and probably be fine...

But, since this isn't the only place that we use time_t, I think I'm returning to "we probably just need a separate portable build for this configuration", otherwise we're susceptible to OOB reads/writes when using pointers, and mis-offsetted reads/writes when time_t is in a struct (like fstat)

[bartonjs]

Another answer, of course, could be that we don't support the portable build on Linux arm32-with-64-bit-time... but that's a business decision vs a technical one, so less interesting to discuss (and not our decision to make).

[filipnavara]

Not saying you should do that... but setting _TIME_BITS=64 in this particular case would likely also get the job done. mktime would become mktime64, time_t would be 64-bit, and the last parameter to X509_VERIFY_PARAM_set_time would be spread into two registers (and OpenSSL builds with 32-bit time_t would ignore the extra parameter passed in r3 register).

(I'd much rather see OpenSSL fixed to follow the same convention as glibc, ie. export two versions of X509_VERIFY_PARAM_set_time under different names, but it may be too late for that.)

[bartonjs]

@richlander We might be looking at a new RID right now, and/or a hole in our OS support story.

https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md says that .NET 8 is supported on Ubunutu 20.04+ on ARM32; but the 32-vs-64-bit time thing breaks it (X509Chain is the most visible break since it happens during dotnet restore, but reading the last modified time from fstat is almost certainly broken (so FileSystemEntry.LastWriteTime(Utc) is misbehaving), and there are probably a lot of other subtle bugs. (Think back to early win64... but with fewer BSODs)

Because this has created another place where binaries are incompatible, I think that we a) need a new portable build, and b) need to define a RID for it (e.g. linux-arm32-time64). The main alternative to doing both of those is to say that arm32 is no longer supported on Ubuntu 24.04 (or any other Linux-arm32 that has moved to 64-bit time). The only other alternative I can think of is to try and do hacky runtime detection anywhere that uses time_t, which will probably take a long time to work out.

With less rambling:

• Plan A: new rid, new portable build. (linux-arm32-time64)
• Plan B: Declare that our portable builds are not supported in that configuration
  • A distro-specific build would work fine, of course.
• Plan C: hack and slash at things for years trying to corner down every time_t size mismatch problem.

I picked you because you've done the most recent work on RIDs (to my knowledge). Not sure who gets to (has to) decide between A and B, though. (But maybe you know who 😄)

[filipnavara]

but reading the last modified time from fstat is almost certainly broken

I don't think that's the case. As I previously mentioned, glibc normally exports different versions of the functions and only headers switch the #defines to import one or another, and the use the appropriate time_t definition. fstat is a fun example since it's affected both by _TIME_BITS and _FILE_OFFSET_BITS, so there's at least fstat, fstat64, and __fstat64_time64 (the decorated names may not be exactly accurate).

(Essentially, it's a similar model to A/W/TCHAR in Windows API where the consumer decides what they want; the problem only affects shared libraries other than glibc that don't use the glibc model, and that use time_t as part of public API.)

[bartonjs]

I was writing up an explanation of why I still think fstat is broken, but I think it ends up working out because we don't call fstat, we call fstat64 (assuming it is defined already for our Linux/arm32 build).

The problem with fstat is that the caller allocates the struct stat destination, so it depends on the caller interpretation of the layout (which depends on the size of time_t); and AFAICT, the implementation doesn't have a "uh, maybe you got here from a 32-bit time_t perspective", so it'll just OOB-write.

fstat64 doesn't have that problem since it pre-promoted to time64_t.

As for other (actual) time_t-based calls... unless time_t in glibc is always 32-bit, things like gmtime's implementation of "copy the time_t value to a time64_t and call gmtime64" are going to have a problem where the caller (us) expects a 32-bit read, but the target (glibc!gmtime) now does a 64-bit read. https://sourceware.org/git/?p=glibc.git;a=blob;f=time/gmtime.c;h=9f39c36bd85d33d0e17f95d719a1f01b003f40ce;hb=HEAD. And, actually, assuming that the build of glibc on Ubuntu 24.04 was itself built with 64-bit time, that body doesn't even exist, it just assumes the headers lined up correctly for the caller to alias gmtime to gmtime64 (either through __REDIRECT_NTH, which (AFAICT) is just writes a jump-stub (struct tm* gmtime(time_t* timer) ASM { __gmtime }, or something like that); or just via #define gmtime __gmtime64). So, I don't think that the glibc headers trick works when glibc and the caller disagree on the time_t size.

I'm happy to be shown what I'm missing, though. Like, maybe Ubuntu built glibc with 32-bit time but then set the headers to 64-bit time afterwards... (though I think that'd still make struct stat misalign in implementation (32-bit offsetting) and consumption (64-bit offsetting); because I didn't see any size conversion code in it). (Or that glibc is still 32-bit time, and only OpenSSL is 64-bit time?)

[tmds]

Glibc builds support both 32-bit and 64-bit time_t users simultaneously. A user decides at compile time what time representation it has (_TIME_BITS=64). If that user exposes time_t in their API (like OpenSSLX509_VERIFY_PARAM_set_time) then its users (like .NET) must be using the same _TIME_BITS or there is an ABI mismatch.

[filipnavara]

I'm happy to be shown what I'm missing, though.

There's not a single stat_t structure definition. There are multiple. For example, there's __stat64_t64 and stat64. Likewise, there's __fstat64_time64 and __fstat64 (https://github.com/bminor/glibc/blob/41903cb6f460d62ba6dd2f4883116e2a624ee6f8/sysdeps/unix/sysv/linux/fstat64.c).

Then the header magic is driven by __TIMESIZE (platform "default" time_t size; which should presumably still be 32 on ARM32), _TIME_BITS (what the user wants to compile against, which could be time_t == __time64_t or time_t == __time_t), and __USE_TIME64_REDIRECTS (set automatically based on the first two defines).

[lbussell]

Because this has created another place where binaries are incompatible, I think that we a) need a new portable build, and b) need to define a RID for it (e.g. linux-arm32-time64). The main alternative to doing both of those is to say that arm32 is no longer supported on Ubuntu 24.04 (or any other Linux-arm32 that has moved to 64-bit time).
...

• Plan B: Declare that our portable builds are not supported in that configuration

It is worth noting that Debian is also making the same transition for the next release (anticipated sometime next year) so going with Plan B would mean that our forward-looking arm32 story is musl-only (already transitioned).

[tmds]

Plan A: new rid, new portable build. (linux-arm32-time64)
Plan B: Declare that our portable builds are not supported in that configuration

Based on the assumption that distros will be changing to 64-bit times on arm32 to address the year 2038 problem, a future version of .NET could choose for portable arm32 to swap to support only 64-bit time_t OpenSSL.

Plan C: hack and slash at things for years trying to corner down every time_t size mismatch problem.

Definitely a hack.

I don't know how .NET could determine what time_t OpenSSL was built with.

Besides the already mentioned X509_VERIFY_PARAM_set_time there is: X509_cmp_time. I didn't find others.

[richlander]

Stating the obvious ... the RID graph was never intended to describe this level of differentiation / ABI difference. We'll be very hesitant to go down that path and to produce builds of any of our packages for both Arm32 RIDs.

[jkotas]

Besides the already mentioned X509_VERIFY_PARAM_set_time there is: X509_cmp_time. I didn't find others.

It should be possible to fix both of these calls by backporting #100461 that @filipnavara suggested above. It is a hack that depends on Arm ABI details, but anything else would be a hack too - just more complicated one.

[tmds]

It should be possible to fix both of these calls by backporting #100461 that @filipnavara suggested above. It is a hack that depends on Arm ABI details, but anything else would be a hack too - just more complicated one.

You're referring to: #101444 (comment). This is for the call to X509_VERIFY_PARAM_set_time which takes time_t as an argument.

X509_cmp_time accepts time_t as a pointer, which is probably also ok because the arm32 builds are little endian.

[richlander]

dotnet/core#9285

[bartonjs]

@richlander If we're declaring that we don't support t64 on arm32 with .NET 8, should we just close this issue now?

[richlander]

The OP was about .NET 8 and 9. It was effectively saying "we have nothing to ship for Ubuntu 24.04 Arm32". That's still relevant. We're also close to being able to resolve this.

[v-wenyuxu]

Failed in: runtime-coreclr libraries-jitstress 20240514.1

Failed tests:

net9.0-linux-Release-arm-jitminopts-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress2_tiered-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress1-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress2-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-no_tiered_compilation-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-tailcallstress-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-disabler2r-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress1_tiered-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate

Error message:

 System.Security.Authentication.AuthenticationException : The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Stack trace:

   at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 509
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 576
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 352
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 90
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSniTest.cs:line 267
--- End of stack trace from previous location ---

[v-wenyuxu]

Failed in: runtime-coreclr libraries-jitstress 20240515.1

Failed tests:

net9.0-linux-Release-arm-jitstress2_tiered-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-tailcallstress-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-no_tiered_compilation-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress1-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitminopts-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress2-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitstress1_tiered-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-disabler2r-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate

Error message:

 System.Security.Authentication.AuthenticationException : The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Stack trace:

   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 576
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 352
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 90
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSniTest.cs:line 267
--- End of stack trace from previous location ---

[rzikm]

This is now happening on Debian 12 images as well and causing some test failures.

[rzikm]

Reading further on this, looks like this was caused by #102059

Based on dotnet/core#9285, I assume we want to remove Debian 12 ARM32 from our CI and have only Debian 13 for main?

cc: @sbomer, @richlander

[sbomer]

I'm reverting that change while I investigate the failures. We are running tests with a product that has _TIME_BITS=64, with a Y2038-compatible glibc, but an openssl library with _TIME_BITS=32. Based on #101444 (comment) I would have expected this to be OK, but clearly I am missing something.

[richlander]

we want to remove Debian 12 ARM32 from our CI and have only Debian 13 for main

We believe that there is a path to supporting Debian 12 and 13 (but not 11). Theory tells us that this should work. Practice is turning out to be challenging. There is something we don't yet understanding. We'll be sure to document our findings since this little "3 hour tour" will certainly be useful for someone else in future.

[v-wenyuxu]

Failed in: runtime-coreclr libraries-pgo 20240516.1

Failed tests:

net9.0-linux-Release-arm-fullpgo_methodprofiling-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-fullpgo_random_gdv_methodprofiling_only-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-defaultpgo-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-fullpgo_random_gdv-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitosr_stress_random-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitrlcse-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitosr_stress-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-syntheticpgo-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-jitrpolayout-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-fullpgo-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-syntheticpgo_blend-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate
net9.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
    - System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate

Error message:

 System.Security.Authentication.AuthenticationException : The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Stack trace:

   at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 509
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 576
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 352
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 90
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSniTest.cs:line 267
--- End of stack trace from previous location ---

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2024-05-16T10:23:04.09Z	runtime-nativeaot-outerloop	Release	ubuntu.2004.armarch.open	arm	System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate	Failed	{"System.PhaseName":"build_linux_arm_Release_NativeAOT_Libs","BuildId":"677440","DefinitionId":"265","System.JobAttempt":"1","architecture":"arm","System.PhaseAttempt":"1","runtimeFlavor":"coreclr","System.StageName":"Build","System.JobName":"__default","System.JobId":"a8f24b3c-c71a-5a83-5031-ad8ed12efa6f","DefinitionName":"runtime-nativeaot-outerloop","BuildNumber":"20240516.2","configuration":"Release","Project":"public","Reason":"Schedule","operatingSystem":"(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7","AzurePipelinesTestRunId":"16792704","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1"}	Schedule
2024-05-16T10:23:04.09Z	runtime-nativeaot-outerloop	Release	ubuntu.2004.armarch.open	arm	System.Net.Security.Tests.SslStreamSniTest.UnencodedHostName_ValidatesCertificate	Failed	{"System.PhaseName":"build_linux_arm_Release_NativeAOT_Libs","BuildId":"677440","DefinitionId":"265","System.JobAttempt":"1","architecture":"arm","System.PhaseAttempt":"1","runtimeFlavor":"coreclr","System.StageName":"Build","System.JobName":"__default","System.JobId":"a8f24b3c-c71a-5a83-5031-ad8ed12efa6f","DefinitionName":"runtime-nativeaot-outerloop","BuildNumber":"20240516.2","configuration":"Release","Project":"public","Reason":"Schedule","operatingSystem":"(Debian.12.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7","AzurePipelinesTestRunId":"16792704","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1"}	Schedule