BUG: preserve fold in Timestamp.replace

[AlexKirko]

• closes BUG: pd.Timestamp loses fold when removing timezone which is inconsistent with datetime.datetime #37610
• 1 tests added / 1 passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Problem

We currently lose fold information (whether the Timestamp corresponds to the first or second instance of wall clock time in a DST transition) when calling Timestamp.replace.

Solution

A simple addition to the code of Timestamp.replace should fix this.

Test

I added the test for the use case I came up with in the issue discussion. The OP example is losing fold when deleting timezone information with Timetsamp.replace, and that's not really a bug, but replacing a valid dateutil timezone with itself and losing fold definitely is.

The proposed solution fixes the original example as well. I just don't think we should be tracking it in tests, as it's not clear to me why fold must be preserved in a Timestamp with no timezone information (but it is the convention recommended in PEP 495, fold section to let users keep invalid fold and to just ignore it).

Some details

IIRC, we ignore fold, when it doesn't do anything, so we should be safe preserving fold while replacing tzinfo with None, as in the OP example. I remember this coming up when we introduced fold support, and we made sure that the fold-aware functions don't care what fold is outside of a DST transition with a dateutil timezone (this was done to satisfy the requirements of PEP 495, fold section).

[AlexKirko]

All green, should be ready for a review.

[jreback]

should go here: pandas/tests/scalar/timestamp/test_unary_ops.py (there is another replace test)

[AlexKirko]

Done.

[jreback]

cc @pganssle @jbrockmendel

[pganssle]

A few things:

1. It should be much more obvious that this thing started with the fold set correctly. This could fail for reasons unrelated to the thing you are testing.
2. You should test that it works both ways fold=0 → fold=0 and fold=1 → fold=1.
3. When you do replace(tzinfo=tz), you are creating an identical timestamp. I suspect that it would be very reasonable for a future version of Python or Pandas to optimize that case and simply return the original Timestamp/datetime if nothing is changed, making this not an ideal test.
4. This method-chaining style is not going to be great in terms of tracebacks if the test fails. The same line covers three separate things that could fail: constructing the Timestamp, executing replace and accessing the .fold property. I would do these on separate lines.

Probably something more like this:

zone = gettz("America/New_York")

ts = Timestamp(2020, 11, 1, 1, 30, fold=fold, tzinfo=zone)
ts_replaced = ts.replace(microseconds=1)
assert ts_replaced.fold == fold

Where fold is parameterized over [0, 1].

[AlexKirko]

@pganssle Thanks! I had made changes to the test, instead testing our behavior versus datetime, but your suggestion makes more sense. This way, we don't care whether datetime works properly (on the slim chance that it breaks). Introduced the changes. Hope you don't mind me keeping the OP timezone.

Please take a look at the new version.

[AlexeyDmitriev]

You can make the replacing of tzinfo work as a reasonable test if you change to the other timezone with the same time of DST change, e.g Moscow to Novosibirsk

[AlexKirko]

@AlexeyDmitriev That would work, but also make the test a bit less readable (we'd need to comment that we are swapping between two DST-zones, and both times are in a fold). I think leaving the current test should be okay.

[pganssle]

Why tz= instead of tzinfo=? I seem to remember that there are vague plans to remove tz= as redundant or something?

[jbrockmendel]

tz kwarg allows for a string whereas tzinfo requires a tzinfo object

[pganssle]

But that's not what's happening here...

[AlexKirko]

Thanks, agreed. Passing a tzinfo object into the tz argument gives the same result (as all tzinfo gives us is a Cython type-check before we set tz, tzinfo = tzinfo, None), but passing tzinfo is more readable and doc-compliant.
Changed it, please take a look.

[AlexKirko]

@jreback @jbrockmendel @pganssle
Accepted all suggestions, all green, detailed comments above.

[jbrockmendel]

Looks fine to me, but I'm going to agree with whatever @pganssle says here

[jreback]

thanks @AlexKirko