datetime calculations with zoneinfo don't work

[jribbens]

Bug report

Doing calculations with datetime objects with zoneinfo timezones doesn't work - they completely ignore the timezone. For example:

>>> LONDON = zoneinfo.ZoneInfo('Europe/London')
>>> d0 = datetime.datetime(2022, 3, 27, 0, tzinfo=LONDON)
>>> print(d0)
2022-03-27 00:00:00+00:00
>>> print(d0 + datetime.timedelta(seconds=3600+1800))
2022-03-27 01:30:00+00:00

2022-03-27 01:30 is a time that doesn't exist in the Europe/London timezone.

Similarly:

>>> d1 = datetime.datetime(2022, 3, 27, 3, tzinfo=LONDON)
>>> print(d1 - d0)
3:00:00

The difference between d1 and d0 is 2 hours, not 3 hours.

I guess this might not be fixable, in which case at the very least the datetime module documentation needs to be updated to say that you simply cannot do arithmetic with datetime objects unless they are timezone-naive or using the UTC timezone.

Your environment

• CPython versions tested on: 3.9
• Operating system and architecture: Ubuntu 20.04.4

[pganssle]

@jribbens This is working as intended. Arithmetic in datetimes is under-defined, and datetime has chosen semantics that are inconsistent with your intuitions in this case, see my blog post on the subject.

There's not really anything to be done about it, though in the future I think it would be nice for us to add some methods for explicitly doing "absolute time" calculations.

I'm going to close this as the behavior is working as intended and there's nothing we can change about it. Thank you for taking the time to report your issue — this was a very clear and well-written bug report. I do wish there were a better way to resolve this.

[jribbens]

@pganssle Thanks. I read your blog-post before filing the bug report (which is partly why I chose the examples I did, which cannot be explained by claiming it's implementing "wall time"). I think your blog is incorrect in describing Python's datetime addition/subtraction as "wall time" - I don't think that was the intent with which it was programmed, nor is it what it outputs. I'd just describe what it does as "wrong".

As per my original note above, I appreciate that this can't necessarily be fixed in the code, but I think it is a mistake to close this bug report as the documentation certainly can and in my opinion should be improved. You might say that this is a very long-standing issue (dating back to Python 2.3 when datetime was originally added) but actually the addition of the zoneinfo module changes things significantly and adds new issues - now Python appears to support time zones, people might expect it to actually work, and the documentation for zoneinfo in particular is actively misleading.

So I reiterate my suggestion that the documentation should be improved. The datetime documentation for datetime arithmetic should be much clearer than the vague "No time zone adjustments are done" and say explicitly that "The wrong result will be produced if an object that is timezone aware is used, unless that timezone has a fixed offset from UTC", and the documentation for zoneinfo that blithely recommends doing arithmetic on timezone-aware datetimes and even shows it being done and the wrong result being produced without mentioning that the result is wrong really must be changed. It should show the datetime being converted to UTC, the arithmetic being performed, and then conversion back to the original time zone.

[pganssle]

Thanks. I read your blog-post before filing the bug report (which is partly why I chose the examples I did, which cannot be explained by claiming it's implementing "wall time"). I think your blog is incorrect in describing Python's datetime addition/subtraction as "wall time" - I don't think that was the intent with which it was programmed, nor is it what it outputs. I'd just describe what it does as "wrong".

I think you may have failed to understand what my blog post is saying, and what the documentation says. It is definitely not "wrong", it's just not the definition you expect. The value returned by subtraction between two datetime objects in the same zone doesn't have anything to do with the amount of time that has elapsed between the two datetimes, though that may coincidentally be the case. This is certainly deliberate, see "supported operations" in the documentation, specifically point 3:

Subtraction of a datetime from a datetime is defined only if both operands are naive, or if both are aware. If one is aware and the other is naive, TypeError is raised.

If both are naive, or both are aware and have the same tzinfo attribute, the tzinfo attributes are ignored, and the result is a timedelta object t such that datetime2 + t == datetime1. No time zone adjustments are done in this case.

If both are aware and have different tzinfo attributes, a-b acts as if a and b were first converted to naive UTC datetimes first. The result is (a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None) - b.utcoffset()) except that the implementation never overflows.

Note that this is the case where "both are aware and have the same tzinfo", so subtraction is equivalent to dt2.replace(tzinfo=None) - dt1.replace(tzinfo=None). The reason for this is detailed in my blog post — it's the inverse of addition, which doesn't operate on elapsed durations but rather on the calendar itself. It operates on abstract civil times, which is a valid definition of arithmetic, if not one that you find intuitive.

If I had time, I'd certainly write a more thorough guide to datetime artihmetic with zoneinfo (I have written something similar in the pytz_deprecation_shim migration guide, but that is very focused on migration away from pytz and needs to be translated into something that doesn't mention pytz, to be more timeless). Of course, if I had more time, I'd probably prioritize defining an interface that makes it easier to choose your desired arithmetic semantics. Maybe one day someone will get around to these things. 😓

I'd be happy to accept a PR to the zoneinfo docs and possibly the datetime docs that give a better explanation of some of these concepts, though I would encourage you not to take the tack that the results you are getting are "wrong" — they are the right answers to a different question than the one you thought you were asking.

[jribbens]

@pganssle I am certainly not understanding how on earth you could say the examples I gave in this bug report are not straight-up wrong. The result of the first calculation is a datetime that doesn't exist. There is no sane definition that can be applied that makes that not wrong.

And even your own explanation of subtraction "The value returned by subtraction between two datetime objects in the same zone doesn't have anything to do with the amount of time that has elapsed between the two datetimes" is simply another way of saying it's wrong.

This has nothing to do with "my desired semantics". I am afraid I cannot suggest documentation to fix this problem if your view is that the results given are "the right answer" to a secret undocumented question that is presumably a non-Euclidian horror that mortal man is not meant to wot of.

I find it astonishing that it is viewed as acceptable that the documentation is extremely misleading and explicitly advises code that will produce bizarre results nobody expects and will inevitably cause many bugs in Python programs for years to come.

[pganssle]

I'm sorry that you feel that way. This state of affairs is actually deliberate and represents a design trade-off. Probably it's not the choice I'd make today for many reasons, but it's not an invalid choice. The people who originally designed these modules are quite smart and so I tend to have quite a bit of humility when I see what strikes me as a bizarre choice, and I often find that this pays dividends.

There's approximately no chance that this will change, as it would be a major breaking change to do so. Rest assured that I'm aware of the fact that people find it counter-intuitive, which is why I'm constantly writing articles and giving talks on this subject. If you're at PyCon this year feel free to track me down and maybe I can explain it better in person. The perspective of someone who has changed their mind is always extremely valuable, since they know what about the explanation made things "click".

Thank you again for your feedback.

[jribbens]

@pganssle Ok just a final comment since you still seem to be missing my point: I'm saying it's the documentation that needs changing, not the code (and mostly I'm saying that the new documentation for zoneinfo needs changing, rather than the long-standing datetime documentation).

[horsemankukka]

Consider the following (Europe had DST change last night, the night of 25 March/26 March).

>>> dt1 = datetime(2023, 3, 25, 12, tzinfo=ZoneInfo("Europe/Paris"))
>>> dt2 = datetime(2023, 3, 26, 12, tzinfo=ZoneInfo("Europe/Paris"))
>>> dt3 = datetime(2023, 3, 26, 12, tzinfo=ZoneInfo("Europe/Berlin"))
>>> str(dt2 - dt1) # same ZoneInfo, so naive
'1 day, 0:00:00'
>>> str(dt3 - dt1) # diff ZoneInfo, but same offsets
'23:00:00'

Europe/Paris and Europe/Berlin are the same timezone (CET/CEST) in spring 2023. But somehow only when the key string to ZoneInfo is different this calculation succeeds (to the intuition of some, including myself).

A footnote (according to my intuition) may not be enough. I think the docs should have a warning box to this effect:

Warning
Even though datetime substraction seems to “just work” with different ZoneInfos, aware datetimes should usually be converted to UTC before any substraction or addition, and after that converted to target zone to ensure offset changes will be correct in the final result.

That would seem to be the preferred logic, as the docs have warning boxes for all methods that have utc in their name, suggesting making them aware. I am not however fully confident this is a fully correct explanation (folds?), so I would prefer someone more experienced to formulate the content of the box.

Including a recipe for absolute addition and absolute substraction in the box would probably be of great help.