API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

mroeschke · 2018-09-09T01:34:58Z

closes Timestamp.tz_localize() NonExistentTimeError handling #8917
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Currently, users do not have any control over nonexistent datetime handling when tz_localizeing like they do ambiguous times. This adds a new keyword nonexistent to tz_localize so that users now can:

'raise': Raise an error (default)
'NaT': Replace nonexistent times with 'NaT'
'shift': Shift nonexistent times forward to the closest existing time

pep8speaks · 2018-09-09T01:35:04Z

Hello @mroeschke! Thanks for submitting the PR.

There are no PEP8 issues in the file pandas/core/arrays/datetimes.py !
There are no PEP8 issues in the file pandas/core/generic.py !
There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_timezones.py !
There are no PEP8 issues in the file pandas/tests/scalar/timestamp/test_timezones.py !
There are no PEP8 issues in the file pandas/tests/series/test_timezones.py !

codecov · 2018-09-09T05:35:40Z

Codecov Report

Merging #22644 into master will increase coverage by <.01%.
The diff coverage is 94.11%.

@@            Coverage Diff             @@
##           master   #22644      +/-   ##
==========================================
+ Coverage   92.22%   92.22%   +<.01%     
==========================================
  Files         169      169              
  Lines       50911    50922      +11     
==========================================
+ Hits        46954    46965      +11     
  Misses       3957     3957

Flag	Coverage Δ
#multiple	`90.65% <94.11%> (ø)`	⬆️
#single	`42.28% <23.52%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimes.py	`97.45% <100%> (+0.05%)`	⬆️
pandas/core/generic.py	`96.79% <83.33%> (-0.05%)`	⬇️
pandas/util/testing.py	`86.73% <0%> (+0.09%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c26375...8cf16e2. Read the comment docs.

pandas/_libs/tslibs/nattype.pyx

pandas/_libs/tslibs/timestamps.pyx

mroeschke · 2018-09-19T17:51:38Z

@jreback re: overlap between errors and nonexistent

The original issue mentions Timestamp.tz_localize() NonExistentTimeError handling #8917 (comment) having the ability to control over ambiguous times vs nonexistent times independently.
1a) nonexistent and ambiguous can handle their own errors in this PR independently
This PR has the ability to shift the nonexistent time to a real time (like how ambiguous can take True/False

So I would propose that eventually we can depreciate errors and keep both ambiguous and nonexistent

jreback · 2018-10-14T18:12:22Z

lgtm @jorisvandenbossche any more comments?

jorisvandenbossche · 2018-10-15T10:00:27Z

pandas/_libs/tslibs/nattype.pyx

@@ -565,6 +565,8 @@ class NaTType(_NaT):
            - 'raise' will raise an AmbiguousTimeError for an ambiguous time

        nonexistent : 'shift', 'NaT', default 'raise'
+            A nonexistent time doesn't not exist in a particular timezone
+            where clocks moved forward due to DST.
            - 'shift' will shift the nonexistent time forward to the closest


Sorry, rst formatting nitpick: there needs to be a blank line between the first sentences, and the start of this list ... (getting rst right can be annoying ..)

mroeschke · 2018-10-17T16:59:41Z

Thanks @jorisvandenbossche. Added those blank lines for rendering.

jreback · 2018-10-18T16:02:21Z

pandas/tests/indexes/datetimes/test_timezones.py

+    @pytest.mark.parametrize('tz', ['Europe/Warsaw', 'dateutil/Europe/Warsaw'])
+    @pytest.mark.parametrize('method, exp', [
+        ['shift', '2015-03-29 03:00:00'],
+        ['NaT', pd.NaT],


do you have tests that exericse the assertion when you pass a nonexistent keyword that is invalid?

Just added a test for an invalid nonexistent keyword.

jorisvandenbossche · 2018-10-19T08:04:57Z

pandas/_libs/tslibs/timestamps.pyx

@@ -978,14 +979,26 @@ class Timestamp(_Timestamp):
            - 'NaT' will return NaT for an ambiguous time
            - 'raise' will raise an AmbiguousTimeError for an ambiguous time

-        errors : 'raise', 'coerce', default 'raise'
+        nonexistent : 'shift', 'NaT', default 'raise'
+            A nonexistent time doesn't not exist in a particular timezone


Suggested change

A nonexistent time doesn't not exist in a particular timezone

A nonexistent time does not exist in a particular timezone

jorisvandenbossche · 2018-10-19T08:05:33Z

pandas/core/arrays/datetimes.py

@@ -639,15 +639,27 @@ def tz_localize(self, tz, ambiguous='raise', errors='raise'):
            - 'raise' will raise an AmbiguousTimeError if there are ambiguous
              times

-        errors : {'raise', 'coerce'}, default 'raise'
+        nonexistent : 'shift', 'NaT' default 'raise'
+            A nonexistent time doesn't not exist in a particular timezone


Suggested change

A nonexistent time doesn't not exist in a particular timezone

A nonexistent time does not exist in a particular timezone

jorisvandenbossche · 2018-10-19T08:05:52Z

pandas/core/generic.py

@@ -8659,6 +8659,17 @@ def tz_localize(self, tz, axis=0, level=None, copy=True,
            - 'NaT' will return NaT where there are ambiguous times
            - 'raise' will raise an AmbiguousTimeError if there are ambiguous
              times
+        nonexistent : 'shift', 'NaT', default 'raise'
+            A nonexistent time doesn't not exist in a particular timezone


Suggested change

A nonexistent time doesn't not exist in a particular timezone

A nonexistent time does not exist in a particular timezone

mroeschke · 2018-10-19T19:38:41Z

Thanks for catching that typo @jorisvandenbossche

jreback · 2018-10-24T12:41:48Z

one more rebase and I think ok to go

jreback · 2018-10-25T11:46:33Z

thanks @mroeschke

…ndas * repo_org/master: (23 commits) DOC: Add docstring validations for "See Also" section (pandas-dev#23143) TST: Fix test assertion (pandas-dev#23357) BUG: Handle Period in combine (pandas-dev#23350) REF: SparseArray imports (pandas-dev#23329) CI: Migrate some CircleCI jobs to Azure (pandas-dev#22992) DOC: update the is_month_start/is_month_end docstring (pandas-dev#23051) Partialy fix issue pandas-dev#23334 - isort pandas/core/groupby directory (pandas-dev#23341) TST: Add base test for extensionarray setitem pandas-dev#23300 (pandas-dev#23304) API: Add sparse Acessor (pandas-dev#23183) PERF: speed up CategoricalIndex.get_loc (pandas-dev#23235) fix and test incorrect case in delta_to_nanoseconds (pandas-dev#23302) BUG: Handle Datetimelike data in DataFrame.combine (pandas-dev#23317) TST: re-enable gbq tests (pandas-dev#23303) Switched references of App veyor to azure pipelines in the contributing CI section (pandas-dev#23311) isort imports-io (pandas-dev#23332) DOC: Added a Multi Index example for the Series.sum method (pandas-dev#23279) REF: Make PeriodArray an ExtensionArray (pandas-dev#22862) DOC: Added Examples for Series max (pandas-dev#23298) API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option (pandas-dev#22644) BUG: Let MultiIndex.set_levels accept any iterable (pandas-dev#23273) (pandas-dev#23291) ...

…xamples * repo_org/master: (83 commits) DOC: Add docstring validations for "See Also" section (pandas-dev#23143) TST: Fix test assertion (pandas-dev#23357) BUG: Handle Period in combine (pandas-dev#23350) REF: SparseArray imports (pandas-dev#23329) CI: Migrate some CircleCI jobs to Azure (pandas-dev#22992) DOC: update the is_month_start/is_month_end docstring (pandas-dev#23051) Partialy fix issue pandas-dev#23334 - isort pandas/core/groupby directory (pandas-dev#23341) TST: Add base test for extensionarray setitem pandas-dev#23300 (pandas-dev#23304) API: Add sparse Acessor (pandas-dev#23183) PERF: speed up CategoricalIndex.get_loc (pandas-dev#23235) fix and test incorrect case in delta_to_nanoseconds (pandas-dev#23302) BUG: Handle Datetimelike data in DataFrame.combine (pandas-dev#23317) TST: re-enable gbq tests (pandas-dev#23303) Switched references of App veyor to azure pipelines in the contributing CI section (pandas-dev#23311) isort imports-io (pandas-dev#23332) DOC: Added a Multi Index example for the Series.sum method (pandas-dev#23279) REF: Make PeriodArray an ExtensionArray (pandas-dev#22862) DOC: Added Examples for Series max (pandas-dev#23298) API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option (pandas-dev#22644) BUG: Let MultiIndex.set_levels accept any iterable (pandas-dev#23273) (pandas-dev#23291) ...

…add shift option (pandas-dev#22644)

Matt Roeschke added 14 commits September 3, 2018 23:12

ENH: Add handling of nonexistent times

bf5e7bf

Merge remote-tracking branch 'upstream/master' into normalize_tz

36d13c7

Merge remote-tracking branch 'upstream/master' into normalize_tz

a5ea445

correct misspelling

8753d00

Merge remote-tracking branch 'upstream/master' into normalize_tz

e1a6c6a

Merge remote-tracking branch 'upstream/master' into normalize_tz

a7c86c8

Merge remote-tracking branch 'upstream/master' into normalize_tz

1884c7b

change method of handling nonexistent times

a6a05df

Add another comment

c4dc8aa

Add tests for timestamps

1bc81db

Add tests for datetimeindex

c81d58c

Add series test and entry in timeseries.rst

b2c8429

Add whatsnew

a65987d

Clean up docstring

710014c

mroeschke added Enhancement Timezones Timezone data dtype labels Sep 9, 2018

Fix nat doc

93159e5

Merge remote-tracking branch 'upstream/master' into normalize_tz

a0ffcdd

mroeschke added this to the 0.24.0 milestone Sep 11, 2018

Matt Roeschke added 4 commits September 11, 2018 14:02

Merge remote-tracking branch 'upstream/master' into normalize_tz

219256f

Merge remote-tracking branch 'upstream/master' into normalize_tz

d435481

Merge remote-tracking branch 'upstream/master' into normalize_tz

7c849b6

Merge remote-tracking branch 'upstream/master' into normalize_tz

56ac4fe

jreback requested changes Sep 18, 2018

View reviewed changes

pandas/_libs/tslibs/nattype.pyx Show resolved Hide resolved

pandas/_libs/tslibs/timestamps.pyx Show resolved Hide resolved

Matt Roeschke added 2 commits September 19, 2018 09:44

Merge remote-tracking branch 'upstream/master' into normalize_tz

b7b09bd

add versionadded

94a72a5

Remove whitespace

39b769e

Add back check_stacklevel

fe575fe

jorisvandenbossche reviewed Oct 15, 2018

View reviewed changes

Add blank line for rendering

3482f92

Merge remote-tracking branch 'upstream/master' into normalize_tz

f0e43e2

jreback reviewed Oct 18, 2018

View reviewed changes

Matt Roeschke added 2 commits October 18, 2018 14:43

Merge remote-tracking branch 'upstream/master' into normalize_tz

b98d4cf

Validate nonexistent argument

e6c5b2d

jorisvandenbossche reviewed Oct 19, 2018

View reviewed changes

Matt Roeschke added 2 commits October 19, 2018 09:55

Merge remote-tracking branch 'upstream/master' into normalize_tz

83423ad

Fix type

1ca0ab2

Matt Roeschke added 2 commits October 24, 2018 09:57

Merge remote-tracking branch 'upstream/master' into normalize_tz

5bcc977

Merge remote-tracking branch 'upstream/master' into normalize_tz

8cf16e2

mroeschke mentioned this pull request Oct 25, 2018

Rounding valid timestamps near daylight savings jumps should not throw NonExistentTimeError #23324

Closed

jreback merged commit 0a2d501 into pandas-dev:master Oct 25, 2018

mroeschke deleted the normalize_tz branch October 25, 2018 15:48

mroeschke mentioned this pull request Oct 29, 2018

BUG/ENH: Handle NonexistentTimeError in date rounding #23406

Merged

4 tasks

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

API/ENH: tz_localize handling of nonexistent times: rename keyword + …

593794f

…add shift option (pandas-dev#22644)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

API/ENH: tz_localize handling of nonexistent times: rename keyword + …

f46b9fe

…add shift option (pandas-dev#22644)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

API/ENH: tz_localize handling of nonexistent times: rename keyword + …

d15ceaf

…add shift option (pandas-dev#22644)

jreback mentioned this pull request Jul 27, 2019

DEPR: execute deprecations for str.cat in v1.0 #27611

Merged

3 tasks

mroeschke mentioned this pull request Nov 28, 2019

DEPR: Remove errors argument in tz_localize #29911

Merged

5 tasks

jreback mentioned this pull request Nov 29, 2019

DEPR: deprecations log for removed issues #13777

Closed

quangngd mentioned this pull request Nov 17, 2023

BUG: Bad shift_forward around UTC+0 when DST #56017

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

mroeschke commented Sep 9, 2018 •

edited

Loading

pep8speaks commented Sep 9, 2018

codecov bot commented Sep 9, 2018 •

edited

Loading

mroeschke commented Sep 19, 2018 •

edited

Loading

jreback commented Oct 14, 2018

jorisvandenbossche Oct 15, 2018

mroeschke commented Oct 17, 2018

jreback Oct 18, 2018

mroeschke Oct 18, 2018

jorisvandenbossche Oct 19, 2018

jorisvandenbossche Oct 19, 2018

jorisvandenbossche Oct 19, 2018

mroeschke commented Oct 19, 2018

jreback commented Oct 24, 2018

jreback commented Oct 25, 2018

	A nonexistent time doesn't not exist in a particular timezone
	A nonexistent time does not exist in a particular timezone

API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

Conversation

mroeschke commented Sep 9, 2018 • edited Loading

pep8speaks commented Sep 9, 2018

codecov bot commented Sep 9, 2018 • edited Loading

Codecov Report

mroeschke commented Sep 19, 2018 • edited Loading

jreback commented Oct 14, 2018

jorisvandenbossche Oct 15, 2018

Choose a reason for hiding this comment

mroeschke commented Oct 17, 2018

jreback Oct 18, 2018

Choose a reason for hiding this comment

mroeschke Oct 18, 2018

Choose a reason for hiding this comment

jorisvandenbossche Oct 19, 2018

Choose a reason for hiding this comment

jorisvandenbossche Oct 19, 2018

Choose a reason for hiding this comment

jorisvandenbossche Oct 19, 2018

Choose a reason for hiding this comment

mroeschke commented Oct 19, 2018

jreback commented Oct 24, 2018

jreback commented Oct 25, 2018

mroeschke commented Sep 9, 2018 •

edited

Loading

codecov bot commented Sep 9, 2018 •

edited

Loading

mroeschke commented Sep 19, 2018 •

edited

Loading