Some DateTimeTest format tests fail on the upcoming Fedora 39

[tmds]

System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"f\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"F\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"g\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"G\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"t\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"T\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"U\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"U\")
System.Tests.DateTimeOffsetTests.TryFormat_ToString_EqualResults
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")

It's likely there was a regression in #83589 which was not caught in CI.

The test failures are from our CI machine. I will reproduce the issue locally and share more information.

cc @tarekgh @omajid

[ghost]

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"f\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"F\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"g\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"G\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"t\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"T\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"U\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"U\")
System.Tests.DateTimeOffsetTests.TryFormat_ToString_EqualResults
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")

It's likely there was a regression in #83589 which was not caught in CI.

The test failures are from our CI machine. I will reproduce the issue locally and share more information.

cc @tarekgh @omajid

Author:	tmds
Assignees:	-
Labels:	area-System.Globalization, untriaged, needs-area-label
Milestone:	-

[tarekgh]

The problem is in DateTime.TryFormat with Utf-8. This assumes all date/time pattern characters are ascii which is not the case as CLDR started to use U+202F as the space character. The following line should convert the character to Utf-8 and store it to the buffer instead of just casting it.

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/DateTimeFormat.cs

Line 730 in 9aefa9d

result.Append(TChar.CastFrom(ch));

[tmds]

I had tried to reproduce the issue in a Fedora 38, and a Fedora 39 container yesterday and to my surprise it didn't reproduce.

The problem is in DateTime.TryFormat with Utf-8. This assumes all date/time pattern characters are ascii which is not the case as CLDR started to use U+202F as the space character. The following line should convert the character to Utf-8 and store it to the buffer instead of just casting it.

Thanks for pointing me to the problem.
This matches with the failures: they show / (0x2F) which is the lower byte of U+202F:

Expected: ···ril 18, 2023 5:10:51 AM
Actual:   ···ril 18, 2023 5:10:51/AM

[tarekgh]

I had tried to reproduce the issue in a Fedora 38, and a Fedora 39 container yesterday and to my surprise it didn't reproduce.

I am able to produce the issue on edora 39. I just used Fedora:39 docker image

[tmds]

I am able to produce the issue on Fedora 39. I just used Fedora:39 docker image

Let me try again.

I'm using the image from registry.fedoraproject.org/fedora , which has a sha ae8f1aaa5e2b for Fedora 39.
Are we using the same one?

[tarekgh]

I just did docker pull fedora:39 and then manually added all needed dependencies to build the runtime repo.

[tmds]

We're using a different image:

registry.fedoraproject.org/fedora            39     ae8f1aaa5e2b  33 hours ago  197 MB
docker.io/library/fedora                     39     9526ca27af97  5 weeks ago   196 MB

I'm going to try building in the Docker Hub image, and see if that reproduces the issue for me as well.

[tarekgh]

Maybe you only need to ensure using the latest ICU version in your container

[root@b6b1f50cc61b tests]# dnf list installed | grep -i icu
libicu.x86_64                              72.1-2.fc38                 @rawhide
libicu-devel.x86_64                        72.1-2.fc38                 @rawhide

[tmds]

Maybe you only need to ensure using the latest ICU version in your container

I have the same versions for icu.

And, the tests don't fail with the Docker Hub image for me either. 🤔

I'm executing the tests by running dotnet test in src/libraries/System.Runtime/tests.

My env looks like:

# env
HOSTNAME=a876272645f7
DISTTAG=f39container
PWD=/root/runtime/src/libraries/System.Runtime/tests
FBR=f39
container=podman
HOME=/root
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:*~=00;90:*#=00;90:*.bak=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.swp=00;90:*.tmp=00;90:*.dpkg-dist=00;90:*.dpkg-old=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:
FGC=f39
TERM=xterm
SHLVL=1
PATH=/root/runtime/.dotnet:/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
OLDPWD=/root/runtime
_=/usr/bin/env

My timezone is UTC:

# date +%Z
UTC

[tmds]

Aha, this does it:

export LANG=en_US.UTF-8

[tarekgh]

right, don't keep C locale as default. We use Invariant culture at that time which will not have U+202F in the time formats.

[tmds]

@tarekgh the parser issues occur when a non-breaking space occurs in the format string.

The tests fail for AllowWhiteSpaces because the parser expects skipped whitespace to match with regular spaces only.
We can allow it to match it also with the non-breaking spaces.

What is probably not covered by any tests: what should happen (without AllowWhiteSpaces) when the format string has a non-breaking space, and the string being parsed has a regular, or a different type of non-breaking space at that position?

(edit:) or is this fully covered by:

runtime/src/libraries/System.Runtime/tests/System/DateTimeTests.cs

Lines 2021 to 2023 in 33b1869

	[Theory]
	[MemberData(nameof(FormatAndParse_DifferentUnicodeSpaces_Succeeds_MemberData))]
	public void FormatAndParse_DifferentUnicodeSpaces_Succeeds(char formatSpaceChar, char parseSpaceChar)

[tarekgh]

@tmds are you willing to open a PR for it? If not, I'll take care with it.

[tmds]

I was working on this patch: main...tmds:runtime:parse_nbsp.

This fixes the FormatExceptions I had mentioned in #84963 (comment).

If this looks like the way to go, I'll open up a PR.

[tarekgh]

You may use Char.IsWhiteSpace instead of checking every character manually. And if you confirm this will fix the FormatException, then it should be good and go ahead open the PR. Thanks for the help with this issue.

[tmds]

Reopen to track fixing the FormatExceptions.