Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix timezone handling for datetime to unixtime conversions #2213

Merged
merged 3 commits into from
Aug 2, 2022

Conversation

joekohlsdorf
Copy link
Contributor

Pull Request check-list

  • Does $ tox pass with this change (including linting)?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

Description of change

datetime objects are supported to set expire, these can have timezones.
mktime was used to convert these to unixtime. mktime in Python however is not timezone aware, it expects the input to be UTC and redis-py did not convert the datetime timestamps to UTC before calling mktime.

This can lead to:

  1. When the datetime timestamp is within DST, mktime fails with "OverflowError: mktime argument out of range" because UTC doesn't have DST. This depends on libc versions.
  2. Setting incorrect expire times because the input datetime object has a timezone but is passed to mktime without converting to UTC first.

Since Python 3.3 datetime has a timestamp function which converts to UTC before calling mktime internally.
I changed all direct uses of mktime to use this instead.

How to reproduce the issues

DST:

import datetime
import time
from dateutil.tz import tzfile

dt = datetime.datetime(2000, 1, 1, 0, 0, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/America/Santiago'))
print(dt.timestamp())
# this will fail with OverflowError because the timezone has DST flag set to 1 at this date
print(time.mktime(dt.timetuple()))

No DST, conversion works but results in incorrect Unix timestamp due to non-timezone aware conversion:

import datetime
import time
from dateutil.tz import tzfile

dt = datetime.datetime(2000, 6, 1, 0, 0, 0, 0, tzinfo=tzfile('/usr/share/zoneinfo/America/Santiago'))
print(dt.timestamp())
# conversion works because DST flag set to 0 but unixtime is wrong
print(time.mktime(dt.timetuple()))

datetime objects are supported to set expire, these can have timezones.
mktime was used to convert these to unixtime. mktime in Python however is not timezone aware, it expects the input to be UTC and redis-py did not convert the datetime timestamps to UTC before calling mktime.

This can lead to:
1) Setting incorrect expire times because the input datetime object has a timezone but is passed to mktime without converting to UTC first.
2) When the datetime timestamp is within DST, mktime fails with "OverflowError: mktime argument out of range" because UTC doesn't have DST. This depends on libc versions.
@@ -1762,14 +1761,12 @@ def getex(
if exat is not None:
pieces.append("EXAT")
if isinstance(exat, datetime.datetime):
s = int(exat.microsecond / 1000000)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed because the maxvalue for microsecond is 999999 resulting in s being 0 for any valid value.

@codecov-commenter
Copy link

codecov-commenter commented Jun 3, 2022

Codecov Report

Merging #2213 (3b1a498) into master (bedf3c8) will decrease coverage by 0.00%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master    #2213      +/-   ##
==========================================
- Coverage   91.83%   91.82%   -0.01%     
==========================================
  Files         108      108              
  Lines       27690    27683       -7     
==========================================
- Hits        25429    25421       -8     
- Misses       2261     2262       +1     
Impacted Files Coverage Δ
redis/commands/core.py 82.11% <83.33%> (-0.01%) ⬇️
tests/test_asyncio/test_commands.py 98.06% <100.00%> (-0.01%) ⬇️
tests/test_commands.py 89.81% <100.00%> (ø)
tests/test_asyncio/test_search.py 98.35% <0.00%> (-0.33%) ⬇️
tests/test_cluster.py 96.91% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bedf3c8...3b1a498. Read the comment docs.

pieces.append(exat)
if pxat is not None:
pieces.append("PXAT")
if isinstance(pxat, datetime.datetime):
ms = int(pxat.microsecond / 1000)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed because timestamp() returns a fractional result and multiplying with 1000 will therefore already consider ms.

@dvora-h
Copy link
Collaborator

dvora-h commented Aug 2, 2022

@joekohlsdorf Sorry it took so long to respond. The PR looks good! I just merged master in to check if it works with the latest changes and we can approve & merge it

@dvora-h dvora-h merged commit 4ed8aba into redis:master Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants