ENH: Added stride/offset aliases in to_datetime #26631

timcera · 2019-06-03T14:17:30Z

Added support for stride and offset aliases to the 'unit' keyword in
'to_datetime'. Documented support for additional 'unit' intervals that
were available.

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

codecov · 2019-06-03T14:56:43Z

Codecov Report

Merging #26631 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26631      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46576    46572       -4     
- Misses       4116     4120       +4

Flag	Coverage Δ
#multiple	`90.41% <ø> (ø)`	⬆️
#single	`41.8% <ø> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/tools/datetimes.py	`85.05% <ø> (ø)`	⬆️
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d124ea...9c67f70. Read the comment docs.

codecov · 2019-06-03T14:56:43Z

Codecov Report

Merging #26631 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26631      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46576    46572       -4     
- Misses       4116     4120       +4

Flag	Coverage Δ
#multiple	`90.41% <ø> (ø)`	⬆️
#single	`41.8% <ø> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/tools/datetimes.py	`85.05% <ø> (ø)`	⬆️
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d124ea...14e987b. Read the comment docs.

WillAyd

Does this close an existing issue? Also tests are critical to make sure this actually works

mroeschke

Yearly and monthly offsets were deprecated recently. #16344.

Week offsets are on the chopping block too so it's best not to advertise them as well. #14024 (comment)

timcera · 2019-06-04T00:13:07Z

Does this close an existing issue? Also tests are critical to make sure this actually works

I couldn't find an existing issue. I had an issue that I didn't report, however. I was trying to use to_datetime to make an hourly series and blindly used to_datetime([1, 2], unit='H') didn't work, which was puzzling. So looked at the code finding that to_datetime([1, 2], unit='h') did work, but 'h' is not an offset alias. Other strange things where, 'm' meant minutely, and 'M' meant monthly.

After working with the code found it easy to add stride.

Passed current tests, so I didn't break old behaviors.

Where should I put new tests?

timcera · 2019-06-04T00:22:07Z

Yearly and monthly offsets were deprecated recently. #16344.

Week offsets are on the chopping block too so it's best not to advertise them as well. #14024 (comment)

There isn't a deprecation warning in to_datetime(). Will add deprecation warning for 'Y', 'A', 'M', and 'W'.

jreback

we have a bunch of tests for this already in pandas/tests/indexes/timedeltas/test_tools.py

so I suppose some of these were not caught if you actually need to do this?

also would need to do an asv run on appropriate benchmarks for this change

jreback · 2019-06-08T21:22:10Z

pandas/_libs/tslibs/timedeltas.pyx

+        unit = 'L'
+    elif unit == 'us':
+        unit = 'U'
+


make this an else

I don't understand. I actually want "unit" to be unchanged if not one of the tested values so what would an "else" do? Wouldn't it force all "unit" values to be "U"? I have in the past used a dictionary instead to do this same thing if that is a better and more readable approach.

else: unit = unit.upper()

jreback · 2019-06-08T21:22:39Z

pandas/core/tools/datetimes.py

+    unit : string, default is 'N'
+        The unit of the arg. Uses a subset of the pandas offset aliases.
+
+        - 'Y', 'A' for yearly (long term average of 365.2425 days)


remove Y, A, M here I think we deprecated these anyhow

Already my working version is a deprecation warning for "Y", "A", "M" and "W". The "W" deprecation is discussed earlier in this conversation.

so don't mention the deprecated ones

timcera · 2019-06-10T18:03:46Z

we have a bunch of tests for this already in pandas/tests/indexes/timedeltas/test_tools.py

No test for the added stride, also almost all of the tests are only for unit='D' and unit='s'.

so I suppose some of these were not caught if you actually need to do this?

I think reasonable to add tests for stride and for other accepted units. Working on it.

also would need to do an asv run on appropriate benchmarks for this change

I have not done this before. Will look around on how to make that happen.

jreback · 2019-06-10T22:22:00Z

pandas/_libs/tslibs/timedeltas.pyx

+        unit = 'L'
+    elif unit == 'us':
+        unit = 'U'
+


else: unit = unit.upper()

jreback · 2019-06-10T22:22:23Z

pandas/core/tools/datetimes.py

+    unit : string, default is 'N'
+        The unit of the arg. Uses a subset of the pandas offset aliases.
+
+        - 'Y', 'A' for yearly (long term average of 365.2425 days)


so don't mention the deprecated ones

jreback · 2019-06-10T22:22:53Z

pandas/_libs/tslibs/timedeltas.pyx

+
+    unit, stride = _base_and_stride(unit)
+
+    # Normalize old or lowercase codes to standard offset aliases.


I am not 100% sure we are actually testing all of these, can you see where we are and validate

jreback · 2019-07-03T19:53:55Z

can you merge master and respond to comments

timcera · 2019-07-09T17:38:17Z

Almost done.

Ran into this numpy/numpy#8325, which affects several of the tests and for some time thought it was my code failing instead. Tried to resolve/fix but just spun wheels instead.

Ran the benchmarks multiple times and got different answers each time. For example, see the attached files of the results summaries from two benchmark runs. Both done overnight while my Chromebook was not used for anything else.
summary_results_2.txt
summary_results_1.txt
Don't know what to do about that. Copying pandas working directory to a beefier machine and see what happens.

Should I squash all commits into one?

jreback · 2019-07-09T21:33:29Z

Almost done.

Ran into this numpy/numpy#8325, which affects several of the tests and for some time thought it was my code failing instead. Tried to resolve/fix but just spun wheels instead.

Ran the benchmarks multiple times and got different answers each time. For example, see the attached files of the results summaries from two benchmark runs. Both done overnight while my Chromebook was not used for anything else.
summary_results_2.txt
summary_results_1.txt
Don't know what to do about that. Copying pandas working directory to a beefier machine and see what happens.

Should I squash all commits into one?

we don't use numpy directly for (very little) nan operations, and not for timedeltas so not sure where you are hitting this.

timcera · 2019-07-10T04:13:07Z

we don't use numpy directly for (very little) nan operations, and not for timedeltas so not sure where you are hitting this.

Initial bug report is #26964 that shows the problem in pandas. Closed it when I found that numpy had the same bug and thought, maybe incorrectly, that pandas was using numpy and therefore getting the same behavior.

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

The 'origin' in to_datetime can be any resolution. New tests for origin resolution and strided unit codes. Deprecated warning for to_datetime units: 'A', 'Y', 'M', 'W'. Edited docstring to reflect strided unit codes and origin resolution. Replaced re test with simple lstrip. Moved pattern to only other file where it is used. Added 'd' to represent daily, but default should be 'D'. Changed 'd' to 'D' in tests.

…iases

TomAugspurger · 2019-07-29T18:01:39Z

Looks like a git merge issue. Let us know if you need help fixing.

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

The 'origin' in to_datetime can be any resolution. New tests for origin resolution and strided unit codes. Deprecated warning for to_datetime units: 'A', 'Y', 'M', 'W'. Edited docstring to reflect strided unit codes and origin resolution. Replaced re test with simple lstrip. Moved pattern to only other file where it is used. Added 'd' to represent daily, but default should be 'D'. Changed 'd' to 'D' in tests.

…iases

timcera · 2019-07-30T15:26:16Z

Yep. Need help with git. I though everything looked good here, but obviously it is a mess.

I could send the log of what I had done if that is helpful.

TomAugspurger · 2019-08-01T21:41:25Z

In theory, a

git fetch upstream
git merge upstream/master

should get you straightened out.

WillAyd · 2019-08-28T16:52:57Z

@timcera can you merge master again? Looks like old builds are no longer available

TomAugspurger · 2019-09-10T20:31:33Z

numpydev is failing. Does this PR introduce new warnings from Numpy? Running with pytest -wError may help.

jreback · 2019-10-06T23:45:16Z

closing as stale; if you want to continue working, pls merge upstream/master and ping to re-open.

ENH: Added stride/offset aliases in to_datetime

9c67f70

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

WillAyd requested changes Jun 3, 2019

View reviewed changes

mroeschke requested changes Jun 3, 2019

View reviewed changes

gfyoung added Enhancement Datetime Datetime data dtype labels Jun 5, 2019

jreback requested changes Jun 8, 2019

View reviewed changes

jreback requested changes Jun 10, 2019

View reviewed changes

timcera mentioned this pull request Jun 20, 2019

BUG: NaNs converted to 1970-01-01 instead of NaT #26964

Closed

timcera added 3 commits July 28, 2019 14:16

ENH: Added stride/offset aliases in to_datetime

bd3d10f

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

Merge branch 'date_aliases' of github.com:timcera/pandas into date_al…

14e987b

…iases

timcera added 3 commits July 30, 2019 07:14

ENH: Added stride/offset aliases in to_datetime

6037971

Added support for stride and offset aliases to the 'unit' keyword in 'to_datetime'. Documented support for additional 'unit' intervals that were available.

Merge branch 'date_aliases' of github.com:timcera/pandas into date_al…

1892f9e

…iases

Merge remote-tracking branch 'upstream/master' into date_aliases

b00576a

Merge remote-tracking branch 'upstream/master' into date_aliases

c35549b

jreback closed this Oct 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Added stride/offset aliases in to_datetime #26631

ENH: Added stride/offset aliases in to_datetime #26631

timcera commented Jun 3, 2019

codecov bot commented Jun 3, 2019

codecov bot commented Jun 3, 2019 •

edited

Loading

WillAyd left a comment

mroeschke left a comment •

edited

Loading

timcera commented Jun 4, 2019

timcera commented Jun 4, 2019

jreback left a comment

jreback Jun 8, 2019

timcera Jun 10, 2019

jreback Jun 10, 2019

jreback Jun 8, 2019

timcera Jun 10, 2019 •

edited

Loading

jreback Jun 10, 2019

timcera commented Jun 10, 2019

jreback Jun 10, 2019

jreback Jun 10, 2019

jreback Jun 10, 2019

jreback commented Jul 3, 2019

timcera commented Jul 9, 2019

jreback commented Jul 9, 2019

timcera commented Jul 10, 2019

TomAugspurger commented Jul 29, 2019

timcera commented Jul 30, 2019

TomAugspurger commented Aug 1, 2019

WillAyd commented Aug 28, 2019

TomAugspurger commented Sep 10, 2019

jreback commented Oct 6, 2019


		unit, stride = _base_and_stride(unit)

		# Normalize old or lowercase codes to standard offset aliases.

ENH: Added stride/offset aliases in to_datetime #26631

ENH: Added stride/offset aliases in to_datetime #26631

Conversation

timcera commented Jun 3, 2019

codecov bot commented Jun 3, 2019

Codecov Report

codecov bot commented Jun 3, 2019 • edited Loading

Codecov Report

WillAyd left a comment

Choose a reason for hiding this comment

mroeschke left a comment • edited Loading

Choose a reason for hiding this comment

timcera commented Jun 4, 2019

timcera commented Jun 4, 2019

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timcera Jun 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timcera commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jul 3, 2019

timcera commented Jul 9, 2019

jreback commented Jul 9, 2019

timcera commented Jul 10, 2019

TomAugspurger commented Jul 29, 2019

timcera commented Jul 30, 2019

TomAugspurger commented Aug 1, 2019

WillAyd commented Aug 28, 2019

TomAugspurger commented Sep 10, 2019

jreback commented Oct 6, 2019

codecov bot commented Jun 3, 2019 •

edited

Loading

mroeschke left a comment •

edited

Loading

timcera Jun 10, 2019 •

edited

Loading