[Android][libraries] TimeZoneInfo Android imp #54845

mdh1418 · 2021-06-28T17:53:11Z

Android has removed zone.tab, so TimeZoneInfo.Unix.cs will no longer work properly on Android as GetTimeZoneIds directly depends on the zone.tab file. The chain of dependency is as follows

GetSystemTimeZones -> PopulateAllSystemTimeZones -> GetTimeZoneIds -> zone.tab
TZI.cs TZI.Unix.cs TZI.Unix.cs TZI.Unix.cs
Where TZI is TimeZoneInfo

zone.tab is a file that is found on the unix system under /usr/share/zoneinfo/
GetTimeZoneIds reads zone.tab to obtain the TimeZoneId in that file
PopulateAllSystemTimeZones caches all the TimeZone Ids in cachedData
GetSystemTimeZones returns a ReadOnlyCollection containing all valid TimeZone’s from the local machine, and the entries are sorted by their DisplayName. It relies on cachedData._systemTimeZones being populated.

The problem is that the time zone data for Android can be found in the file tzdata at the possible locations

/apex/com.android.tzdata/etc/tz/tzdata
/apex/com.android.runtime/etc/tz/tzdata
/data/misc/zoneinfo/tzdata
/system/usr/share/zoneinfo/tzdata

The rest of unix the time zone data can be found in the file zone.tab at

 /usr/share/zoneinfo/zone.tab

Android's TimeZoneInfo implementation should read time zone data from its locations instead of the general /usr/share/zoneinfo/zone.tab path. Moreover, tzdata contains all timezones byte data.

This PR achieves the following:

Splits TimeZoneInfo.Unix.cs into TimeZoneInfo.Unix.cs, TimeZoneInfo.Unix.NonAndroid.cs (non-Android), and TimeZoneInfo.Unix.Android.cs (Android specific)
Adds an interop to obtain the default time zone on Android based on persist.sys.timezone
Implements GetLocalTimeZoneCore TryGetTimeZoneFromLocalMachineCore and GetTimeZoneIds for Android based on mono/mono implementation https://github.com/mono/mono/blob/main/mcs/class/corlib/System/TimeZoneInfo.Android.cs
Adds new string resources to throw exceptions
Refactors the mono/mono implementation of parsing tzdata

Android tzdata files are found in the format of
Header <Beginning of Entry Index> Entry Entry Entry ... Entry <Beginning of Data Index> <TZDATA>

https://github.com/aosp-mirror/platform_bionic/blob/master/libc/tzcode/bionic.cpp

The header (24 bytes) contains the following information
signature - 12 bytes of the form "tzdata2012f\0" where 2012f is subject to change
index offset - 4 bytes that denotes the offset at which the index of the tzdata file starts
data offset - 4 bytes that denotes the offset at which the data of the tzdata file starts
final offset - 4 bytes that used to denote the final offset, which we don't use but will note.

Each Data Entry (52 bytes) can be used to generate a TimeZoneInfo and contain the following information
id - 40 bytes that contain the id of the time zone data entry timezone<id>
byte offset - 4 bytes that denote the offset from the data offset timezone<id> data can be found
length - 4 bytes that denote the length of the data for timezone<id>
unused - 4 bytes that used to be raw GMT offset, but now is always 0 since tzdata2014f (L).

When GetLocalTimeZoneCore TryGetTimeZoneFromLocalMachineCore or GetTimeZoneIds are called, an android timezone data instance is instantiated and loaded by attempting to load a tzdata file that can be found at four locations mentioned earlier. The file is parsed by first loading the header which contains information about where the data index and data begin. The data index is then parsed to obtain the timezone and the corresponding bytes location in the file to fill the three arrays _ids _byteOffsets _lengths. These arrays are referenced to obtain the corresponding byte data for a timezone, and functions from TimeZoneInfo.Unix.cs are leveraged to create a TimeZoneInfo from there.

…ones

dotnet-issue-labeler · 2021-06-28T17:53:15Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

mdh1418 · 2021-06-28T19:35:48Z

Unlike other Unix implementations where zone.tab contains meta-data and specific timezone files (Europe/London) contain the actual timezone data, tzdata on Android combines all timezone data together with the metadata into tzdata. As such, more mono implementation is needed to be ported over in order to get TryGetTimeZoneFromLocalMachine to work.

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.AnyUnix.cs

steveisok · 2021-06-29T03:35:34Z

@mdh1418 As I suspected, there's a little bit more to this. I have more local and we can talk tomorrow about the details.

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.AnyUnix.cs

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Android.cs

marek-safar · 2021-06-29T08:22:26Z

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Android.cs

+                }
+            }
+
+            return Environment.GetEnvironmentVariable("ANDROID_ROOT") + DefaultTimeZoneDirectory;


This look odd. Are you sure we should mix 2 rooted locations?

I think I'm missing something, what are the two rooted locations, ANDROID_ROOT and?
The code was adapted from https://github.com/mono/mono/blob/7791e4d94206dd63acff49268b56a01c0b69983c/mcs/class/corlib/System/TimeZoneInfo.Android.cs#L68-L73 and https://github.com/mono/mono/blob/7791e4d94206dd63acff49268b56a01c0b69983c/mcs/class/corlib/System/TimeZoneInfo.Android.cs#L488-L495 before AndroidTzData's own Path was introduced into the PR.

ghost · 2021-06-29T08:22:45Z

Tagging subscribers to this area: @tarekgh, @safern
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #41867

Android has removed zone.tab, so TimeZoneInfo.Unix.cs will no longer work properly on Android as GetTimeZoneIds directly depends on the zone.tab file. The chain of dependency is as follows

GetSystemTimeZones -> PopulateAllSystemTimeZones -> GetTimeZoneIds -> zone.tab
TZI.cs TZI.Unix.cs TZI.Unix.cs TZI.Unix.cs
Where TZI is TimeZoneInfo

zone.tab is a file that is found on the unix system under /usr/share/zoneinfo/
GetTimeZoneIds reads zone.tab to obtain the TimeZoneId in that file
PopulateAllSystemTimeZones caches all the TimeZone Ids in cachedData
GetSystemTimeZones returns a ReadOnlyCollection containing all valid TimeZone’s from the local machine, and the entries are sorted by their DisplayName. It relies on cachedData._systemTimeZones being populated.

The problem is that the time zone data for Android can be found in the file tzdata at the possible locations

/apex/com.android.tzdata/etc/tz/tzdata
/apex/com.android.runtime/etc/tz/tzdata
/data/misc/zoneinfo/tzdata
/system/usr/share/zoneinfo/tzdata

The rest of unix the time zone data can be found in the file zone.tab at

 /usr/share/zoneinfo/zone.tab

As zone.tab and tzdata have the same format and contents, Android's TimeZoneInfo implementation should read time zone data from its locations instead of the general /usr/share/zoneinfo/zone.tab path.

This PR achieves the following:

Splits TimeZoneInfo.Unix.cs into TimeZoneInfo.AnyUnix.cs, TimeZoneInfo.Unix.cs (non-Android), and TimeZoneInfo.Android.cs (Android specific)
Modifies which directory is used in PopulateAllSystemTimeZones GetTimeZoneDirectory() following the order of https://github.com/mono/mono/blob/7791e4d94206dd63acff49268b56a01c0b69983c/mcs/class/corlib/System/TimeZoneInfo.Android.cs#L68-L73
Renames ZoneTabFileName to TimeZoneFileName and split definitions for Android or non-Android (TimeZoneInfo.Unix.cs)

Author:	mdh1418
Assignees:	-
Labels:	`area-System.Globalization`
Milestone:	-

ghost · 2021-06-29T08:22:55Z

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #41867

Android has removed zone.tab, so TimeZoneInfo.Unix.cs will no longer work properly on Android as GetTimeZoneIds directly depends on the zone.tab file. The chain of dependency is as follows

GetSystemTimeZones -> PopulateAllSystemTimeZones -> GetTimeZoneIds -> zone.tab
TZI.cs TZI.Unix.cs TZI.Unix.cs TZI.Unix.cs
Where TZI is TimeZoneInfo

zone.tab is a file that is found on the unix system under /usr/share/zoneinfo/
GetTimeZoneIds reads zone.tab to obtain the TimeZoneId in that file
PopulateAllSystemTimeZones caches all the TimeZone Ids in cachedData
GetSystemTimeZones returns a ReadOnlyCollection containing all valid TimeZone’s from the local machine, and the entries are sorted by their DisplayName. It relies on cachedData._systemTimeZones being populated.

The problem is that the time zone data for Android can be found in the file tzdata at the possible locations

/apex/com.android.tzdata/etc/tz/tzdata
/apex/com.android.runtime/etc/tz/tzdata
/data/misc/zoneinfo/tzdata
/system/usr/share/zoneinfo/tzdata

The rest of unix the time zone data can be found in the file zone.tab at

 /usr/share/zoneinfo/zone.tab

As zone.tab and tzdata have the same format and contents, Android's TimeZoneInfo implementation should read time zone data from its locations instead of the general /usr/share/zoneinfo/zone.tab path.

This PR achieves the following:

Splits TimeZoneInfo.Unix.cs into TimeZoneInfo.AnyUnix.cs, TimeZoneInfo.Unix.cs (non-Android), and TimeZoneInfo.Android.cs (Android specific)
Modifies which directory is used in PopulateAllSystemTimeZones GetTimeZoneDirectory() following the order of https://github.com/mono/mono/blob/7791e4d94206dd63acff49268b56a01c0b69983c/mcs/class/corlib/System/TimeZoneInfo.Android.cs#L68-L73
Renames ZoneTabFileName to TimeZoneFileName and split definitions for Android or non-Android (TimeZoneInfo.Unix.cs)

Author:	mdh1418
Assignees:	-
Labels:	`os-android`
Milestone:	-

ghost · 2021-06-29T08:24:17Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #41867

Android has removed zone.tab, so TimeZoneInfo.Unix.cs will no longer work properly on Android as GetTimeZoneIds directly depends on the zone.tab file. The chain of dependency is as follows

GetSystemTimeZones -> PopulateAllSystemTimeZones -> GetTimeZoneIds -> zone.tab
TZI.cs TZI.Unix.cs TZI.Unix.cs TZI.Unix.cs
Where TZI is TimeZoneInfo

zone.tab is a file that is found on the unix system under /usr/share/zoneinfo/
GetTimeZoneIds reads zone.tab to obtain the TimeZoneId in that file
PopulateAllSystemTimeZones caches all the TimeZone Ids in cachedData
GetSystemTimeZones returns a ReadOnlyCollection containing all valid TimeZone’s from the local machine, and the entries are sorted by their DisplayName. It relies on cachedData._systemTimeZones being populated.

The problem is that the time zone data for Android can be found in the file tzdata at the possible locations

/apex/com.android.tzdata/etc/tz/tzdata
/apex/com.android.runtime/etc/tz/tzdata
/data/misc/zoneinfo/tzdata
/system/usr/share/zoneinfo/tzdata

The rest of unix the time zone data can be found in the file zone.tab at

 /usr/share/zoneinfo/zone.tab

As zone.tab and tzdata have the same format and contents, Android's TimeZoneInfo implementation should read time zone data from its locations instead of the general /usr/share/zoneinfo/zone.tab path.

This PR achieves the following:

Splits TimeZoneInfo.Unix.cs into TimeZoneInfo.AnyUnix.cs, TimeZoneInfo.Unix.cs (non-Android), and TimeZoneInfo.Android.cs (Android specific)
Modifies which directory is used in PopulateAllSystemTimeZones GetTimeZoneDirectory() following the order of https://github.com/mono/mono/blob/7791e4d94206dd63acff49268b56a01c0b69983c/mcs/class/corlib/System/TimeZoneInfo.Android.cs#L68-L73
Renames ZoneTabFileName to TimeZoneFileName and split definitions for Android or non-Android (TimeZoneInfo.Unix.cs)

Author:	mdh1418
Assignees:	-
Labels:	`area-System.Runtime`, `os-android`
Milestone:	-

…Zone into separate Unix and Android functions.

…Zones to map to a method in the mono port

…Core in the mono port, so created TryGetTimeZoneFromLocalMachineCore for both TimeZoneInfo.Unix.cs and Android.cs

…meZoneFromLocalMachineCore

eerhardt · 2021-07-16T21:29:34Z

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs

+        */
+        private sealed class AndroidTzData
+        {
+            private unsafe struct AndroidTzDataHeader


Suggested change

private unsafe struct AndroidTzDataHeader

private readonly struct AndroidTzDataHeader

error CS8340: Instance fields of readonly structs must be readonly.
/Users/mdhwang/runtime_droid/src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs(314,17): error CS0191: A readonly field cannot be assigned to (except in a constructor or init-only setter of the type in which the field is defined or a variable initializer)

I went ahead and removed this struct altogether, using out parameters for indexOffset and dataOffset in LoadTzFile and LoadHeader

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs

eerhardt · 2021-07-16T21:36:24Z

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs

+                Span<byte> buffer = stackalloc byte[length];
+                ReadTzDataIntoBuffer(File.OpenRead(_tzFilePath), offset, buffer);
+                byte[] tzBuffer = buffer.ToArray();


No reason to make a stack buffer here, copy the file into the buffer, and then ToArray it.

It is unnecessary copying

We don't know that the length of the file will fit on the stack. In general, stackalloc's should not be unbounded.

Suggested change

Span<byte> buffer = stackalloc byte[length];

ReadTzDataIntoBuffer(File.OpenRead(_tzFilePath), offset, buffer);

byte[] tzBuffer = buffer.ToArray();

byte[] tzBuffer = new byte[length];

ReadTzDataIntoBuffer(File.OpenRead(_tzFilePath), offset, tzBuffer);

Thanks. I see, was it unnecessary copying because we need to still return a byte[] here, whereas in LoadTzFile we didn't "need" to use a byte[] and a Span<byte> would suffice?

Is there a general rule for how large a stackalloc should be?

was it unnecessary copying because we need to still return a byte[] here, whereas in LoadTzFile we didn't "need" to use a byte[] and a Span would suffice?

Correct. In LoadTzFile, you just need a buffer to read the file data into, and then you are parsing the values out of it from the buffer. The buffer didn't need to stick around.

Here, since the method expects you to return a byte[], you might as well just read directly into the byte[] you are going to return.

Is there a general rule for how large a stackalloc should be?

Looking around the codebase, we usually don't let a stackalloc go above 1K of memory.

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs

mdh1418 · 2021-07-19T15:51:50Z

Made a few more changes, including parsing the data entry bytes to construct a string immediately, removing the structs all together, removed unsafe keywords, and updated the description of this PR. Can I get another review @eerhardt @jkotas @steveisok

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs

Mitchell Hwang added 4 commits June 24, 2021 15:27

Initial Android implementation

1014cc6

Split implementations for Android specific GetTimeZoneIds

d161105

Differentiate Unix and Android TimeZoneInfo by PopulateAllSystemTimeZ…

cfa0b33

…ones

Utilize Android paths for timezone data

caa39a6

mdh1418 requested review from steveisok and akoeplinger June 28, 2021 17:53

mdh1418 marked this pull request as ready for review June 28, 2021 18:13

Clean up mono style to match runtime style

ee51c67

steveisok reviewed Jun 28, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.AnyUnix.cs Outdated Show resolved Hide resolved

marek-safar reviewed Jun 29, 2021

View reviewed changes

marek-safar added the area-System.Globalization label Jun 29, 2021

marek-safar added os-android and removed area-System.Globalization labels Jun 29, 2021

marek-safar added the area-System.Runtime label Jun 29, 2021

Pulled in parsing implementation from mono and broke out GetLocalTime…

ce91611

…Zone into separate Unix and Android functions.

steveisok added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jun 29, 2021

Steve Pfister and others added 7 commits June 29, 2021 16:32

TimeZoneInfo.Android builds properly now. Found PopulateAllSystemTime…

4ed8616

…Zones to map to a method in the mono port

Found TryGetTimeZoneFromLocalMachine to map to FindSystemTimeZoneById…

555d0b5

…Core in the mono port, so created TryGetTimeZoneFromLocalMachineCore for both TimeZoneInfo.Unix.cs and Android.cs

Fixed typo on type

d59fb1a

Added cacheddata param

84dcbb7

Initial implementation of PopulateAllSystemTimeZonesCore and TryGetTi…

cb9efd4

…meZoneFromLocalMachineCore

Clean up missing variables and wrong function calls

b68e290

Fix cachedData parameter overload

65615ca

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

eerhardt reviewed Jul 16, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

Remove AndroidTzDataHeader and fix typos

311ecaa

marek-safar approved these changes Jul 17, 2021

View reviewed changes

Mitchell Hwang added 5 commits July 19, 2021 09:50

Add const keyword, move field declaration, use length comparison

a07da96

Change data entry id to string

a47ade2

Parse id account for null terminating character

cc06d96

Remove AndroidTzDataEntry struct and use out parameters

00b0abe

Remove unsafe keywords

5290490

jkotas reviewed Jul 19, 2021

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/TimeZoneInfo.Unix.Android.cs Outdated Show resolved Hide resolved

Close file handle in GetTimeZoneData

95f30f6

mdh1418 mentioned this pull request Jul 19, 2021

Failing SendReceive_MemoryArrayTask.SendAsync_ConcurrentDispose_SucceedsOrThrowsAppropriateException test #55085

Closed

This was referenced Jul 19, 2021

[HTTP/3] SendMoreThanStreamLimitRequests_Succeeds times out #55957

Closed

Connect_DualMode_DnsConnect_RetrievedEndPoints_Success - SocketException: Transport endpoint is not connected #55709

Closed

mdh1418 mentioned this pull request Jul 20, 2021

Mac Catalyst: TimeZoneInfo.FindSystemTimeZoneById fails #55941

Closed

karelz mentioned this pull request Jul 20, 2021

Test failure TcpAcceptSocket_WhenServerBoundToWildcardAddress_LocalEPIsSpecific - different port #56042

Closed

steveisok approved these changes Jul 21, 2021

View reviewed changes

steveisok merged commit 46d9b31 into dotnet:main Jul 21, 2021

mdh1418 deleted the port_android_timezoneinfo branch July 26, 2021 12:58

karelz mentioned this pull request Jul 28, 2021

[WinHttpHandler] Long running test: ResponseHeadersRead_SynchronizationContextNotUsedByHandler on Win7/Win8 #54034

Closed

ghost locked as resolved and limited conversation to collaborators Aug 25, 2021

jeffhandley added area-System.DateTime and removed area-System.Runtime labels Nov 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Android][libraries] TimeZoneInfo Android imp #54845

[Android][libraries] TimeZoneInfo Android imp #54845

mdh1418 commented Jun 28, 2021 •

edited

Loading

dotnet-issue-labeler bot commented Jun 28, 2021

mdh1418 commented Jun 28, 2021

steveisok commented Jun 29, 2021

marek-safar Jun 29, 2021

mdh1418 Jul 1, 2021

ghost commented Jun 29, 2021

ghost commented Jun 29, 2021

ghost commented Jun 29, 2021

eerhardt Jul 16, 2021 •

edited

Loading

mdh1418 Jul 16, 2021

eerhardt Jul 16, 2021

mdh1418 Jul 16, 2021

eerhardt Jul 18, 2021

mdh1418 commented Jul 19, 2021

	private unsafe struct AndroidTzDataHeader
	private readonly struct AndroidTzDataHeader

[Android][libraries] TimeZoneInfo Android imp #54845

[Android][libraries] TimeZoneInfo Android imp #54845

Conversation

mdh1418 commented Jun 28, 2021 • edited Loading

dotnet-issue-labeler bot commented Jun 28, 2021

mdh1418 commented Jun 28, 2021

steveisok commented Jun 29, 2021

marek-safar Jun 29, 2021

Choose a reason for hiding this comment

mdh1418 Jul 1, 2021

Choose a reason for hiding this comment

ghost commented Jun 29, 2021

ghost commented Jun 29, 2021

ghost commented Jun 29, 2021

eerhardt Jul 16, 2021 • edited Loading

Choose a reason for hiding this comment

mdh1418 Jul 16, 2021

Choose a reason for hiding this comment

eerhardt Jul 16, 2021

Choose a reason for hiding this comment

mdh1418 Jul 16, 2021

Choose a reason for hiding this comment

eerhardt Jul 18, 2021

Choose a reason for hiding this comment

mdh1418 commented Jul 19, 2021

mdh1418 commented Jun 28, 2021 •

edited

Loading

eerhardt Jul 16, 2021 •

edited

Loading