The field LastModified field for the s3Object is not in GMT from C# as described in the docs

[screig]

Description

Interrogate an s3 bucket using the AWSSDK.S3 C# nuget package, the LastModifed times you get back are not in UTC (GMT). The times are in fact in some unknown timezone.

https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/TS3Object.html

Reproduction Steps

Query a bucket and some file from two different regions, if the time was truly in UTC then the LastModified value would be the same. Here is the code where I print the times out of some bucket.

foreach (S3Object entry in filter_objects)
...
...
                if ((DateTime.Today - dt).Days < 15)
                {
                    _log.Warn($"RECENT ONETICK FILE -- date = {dt.ToString("dd-MMM-yy")},  filename = {entry.Key},  size = {entry.Size.ToString().PadRight(10)},  uploaded on = {entry.LastModified.ToString("dd-MMM-yy HH:mm", CultureInfo.InvariantCulture)} (UTC time)  ");
                    Thread.Sleep(50); // Just to help with the logging order.
                }

Logs

here I am running the code in London

2021-07-20 12:32:13,475 191 [1] WARN  OneTick.OneTick_Backfiller (null) - ################################################################
2021-07-20 12:32:13,538 255 [1] WARN  OneTick.OneTick_Backfiller (null) - Starting OneTick.OneTick_Backfiller
2021-07-20 12:32:13,541 257 [1] WARN  OneTick.OneTick_Backfiller (null) - ################################################################
2021-07-20 12:32:13,543 259 [1] WARN  OneTick.OneTick_Backfiller (null) - Current time zone = GMT Standard Time
2021-07-20 12:32:13,546 262 [1] WARN  OneTick.OneTick_Backfiller (null) - Local time is now : 20-Jul-21 12:32
2021-07-20 12:32:13,548 264 [1] WARN  OneTick.OneTick_Backfiller (null) - UTC time is now : 20-Jul-21 11:32
2021-07-20 12:32:14,017 733 [1] WARN  OneTick.s3_Interrogator (null) - ******************** Interrogating at the ONETICK s3 bucket ********************
2021-07-20 12:32:26,113 12830 [1] WARN  OneTick.s3_Interrogator (null) - RECENT ONETICK FILE -- date = 06-Jul-21,  filename = caxton-bars/out/caxton_bars_last_5_days_20210706.csv.gz,  size = 113401744 ,  uploaded on = 07-Jul-21 07:40 (UTC time)

And here in the US

2021-07-20 07:36:48,997 254 [1] WARN  OneTick.OneTick_Backfiller (null) - ################################################################
2021-07-20 07:36:49,086 343 [1] WARN  OneTick.OneTick_Backfiller (null) - Starting OneTick.OneTick_Backfiller
2021-07-20 07:36:49,105 362 [1] WARN  OneTick.OneTick_Backfiller (null) - ################################################################
2021-07-20 07:36:49,112 369 [1] WARN  OneTick.OneTick_Backfiller (null) - Current time zone = Eastern Standard Time
2021-07-20 07:36:49,113 370 [1] WARN  OneTick.OneTick_Backfiller (null) - Local time is now : 20-Jul-21 07:36
2021-07-20 07:36:49,114 371 [1] WARN  OneTick.OneTick_Backfiller (null) - UTC time is now : 20-Jul-21 11:36
2021-07-20 07:36:49,642 899 [1] WARN  OneTick.s3_Interrogator (null) - ******************** Interrogating at the ONETICK s3 bucket ********************
2021-07-20 07:36:50,917 2174 [1] WARN  OneTick.s3_Interrogator (null) - RECENT ONETICK FILE -- date = 06-Jul-21,  filename = caxton-bars/out/caxton_bars_last_5_days_20210706.csv.gz,  size = 113401744 ,  uploaded on = 07-Jul-21 02:40 (UTC time)

Note how the last modified time is different by five hours. So it is not GMT, but in some unknown timezone. If I at least knew the timezone I could convert it to GMT.

Environment

<PackageReference Include="AWSSDK.S3" Version="3.7.1.14" />
<TargetFramework>netstandard2.0</TargetFramework>
 Windows 10 

Resolution

---

This is a 🐛 bug-report

[ashishdhingra]

Hi @screig,

Good morning.

Based on my testing, the LastModified time is the Local time. For example, using the following code:

using System;
using Amazon.S3;
using Amazon.S3.Model;

namespace S3GetObjectTest
{
    class Program
    {
        private static string bucketName = "bucket_name";

        static void Main(string[] args)
        {
            var s3Client = new AmazonS3Client();
            var listObjectResponse = s3Client.ListObjectsAsync(new ListObjectsRequest() {
                BucketName= bucketName
            }).Result;

            foreach(var s3Object in listObjectResponse.S3Objects)
            {
                Console.WriteLine($"Key: {s3Object.Key}, LastModified: {s3Object.LastModified}");
            }
        }
    }
}

produces the following output in Debug window:

Notice the Kind of DateTime is System.DateTimeKind.Local.

Also refer https://en.wikipedia.org/wiki/ISO_8601. ISO 8601 format can be used to represent local or UTC time. In our case, it's represented in local time.

Hope this helps.

Thanks,
Ashish

[screig]

Hi

The AWS docs do state that the time should be in GMT, no?

The date and time the object was last modified. The date retrieved from S3 is in ISO8601 format. A GMT formatted date is passed back to the user.

[ashishdhingra]

Hi

The AWS docs do state that the time should be in GMT, no?

The date and time the object was last modified. The date retrieved from S3 is in ISO8601 format. A GMT formatted date is passed back to the user.

@screig The docs state that the GMT formatted date is passed to user, but this is on the HTTP response stream. The date in C# is represented as normal DateTime object, not a formatted string. If you are able to inspect the HTTP response using some network monitoring tool (e.g. Fiddler), you should be able to see GMT formatted date in raw HTTP response.

[screig]

The date in C# is represented as normal DateTime object, not a formatted string.

I have no idea why you are bringing strings in to this? Are you suggesting that the DateTime object cannot hold the timezone information. Using your example, both of these "normal DateTime objects", no? The first one has the time as GMT(UTC) as per the docs and it storing the tz.

The docs says it should return a DateTime object and that it should be in the GMT(UTC) timezone. This function currently does not do that, that is a bug. That's simply a fact. Even now I don't know what the local timezone is. Is it the local time of the insertion region or the local time of the extraction region?

This function should a DateTime object with contents like the one below. Where the time is in UTC and the tz is set in "Kind" member as UTC. Where as I mentioned before GMT=UTC.

Please fix this. Or change the docs.

[ashishdhingra]

Could be similar as #1224

[wjrogers]

Here's the problem spot, which I found while investigating this issue for my previous report (#1224). This DateTime.Parse() call converts to local time:

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Transform/SimpleTypeUnmarshaller.cs

Line 399 in 08c86fc

return DateTime.Parse(text, CultureInfo.InvariantCulture);

[boblodgett]

I have verified this issue and other DateTime issues have been corrected in 4.0.0.0-preview.5 released today: https://github.com/aws/aws-sdk-net/releases/tag/4.0.0.0-preview.5

The changes were made in AWS SDK for .NET version 4 because of the number of breaking changes required to fix DateTime handling.

This PR details all the changes made to the SDK's DateTime handling to ensure AWS SDK for .NET is UTC first: #3572

Please give preview 5 a try if you are able to do so.