PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock #25765

hksonngan · 2019-03-18T10:14:44Z

closes Performance degradation when dumping large dataframe with np.datetime to csv #25708
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry
Slicing in line 2181 only slice of i8values that make a whole of values array is processed in _is_dates_only function. This change was from pull diff reduction for 24024 #24543

        values = self.values
        i8values = self.values.view('i8')

        if slicer is not None:
            i8values = i8values[..., slicer]

        from pandas.io.formats.format import _get_format_datetime64_from_values
        fmt = _get_format_datetime64_from_values(values, date_format)

codecov · 2019-03-18T10:50:39Z

Codecov Report

Merging #25765 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25765      +/-   ##
==========================================
+ Coverage   91.25%   91.25%   +<.01%     
==========================================
  Files         172      172              
  Lines       52977    52978       +1     
==========================================
+ Hits        48342    48344       +2     
+ Misses       4635     4634       -1

Flag	Coverage Δ
#multiple	`89.82% <100%> (ø)`	⬆️
#single	`41.74% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/internals/blocks.py	`94.08% <100%> (ø)`	⬆️
pandas/util/testing.py	`89.08% <0%> (+0.09%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 707c720...54502c0. Read the comment docs.

codecov · 2019-03-18T10:50:43Z

Codecov Report

Merging #25765 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25765      +/-   ##
==========================================
- Coverage   91.99%   91.98%   -0.01%     
==========================================
  Files         175      175              
  Lines       52382    52383       +1     
==========================================
- Hits        48188    48184       -4     
- Misses       4194     4199       +5

Flag	Coverage Δ
#multiple	`90.54% <100%> (ø)`	⬆️
#single	`40.74% <0%> (-0.14%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/internals/blocks.py	`94.08% <100%> (ø)`	⬆️
pandas/io/gbq.py	`75% <0%> (-12.5%)`	⬇️
pandas/core/frame.py	`96.9% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.61% <0%> (-0.11%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c18c8be...d135af2. Read the comment docs.

jreback

tests pls

jreback · 2019-03-18T12:10:46Z

or if this is a perf fix, then show the updated asv's (and/or add one)

pep8speaks · 2019-03-21T00:34:41Z

Hello @hksonngan! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-04-19 00:10:51 UTC

hksonngan · 2019-03-21T00:36:39Z

@jreback I add asv's test, test with my macair 2012.

· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)
[  0.00%] · For pandas commit db6993cd <master^2> (round 1/2):
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt....
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 25.00%] · For pandas commit 4472dc9d <25708-fix-reregression-slicing> (round 1/2):
[ 25.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 25.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 50.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 50.00%] · For pandas commit 4472dc9d <25708-fix-reregression-slicing> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· io.csv.ToCSVDatetimeBig.time_frame                                    912±50ms
[ 75.00%] · For pandas commit db6993cd <master^2> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· io.csv.ToCSVDatetimeBig.time_frame                                   1.28±0.2s

hksonngan · 2019-03-21T17:53:39Z

@jreback ping

jreback · 2019-03-22T11:59:43Z

@hksonngan I think you ran asv dev, you need to run asv continuous here it should the ratio of the change.

jreback · 2019-03-22T11:59:55Z

also pls add a whatsnew note. in performance improvements.

hksonngan · 2019-03-22T21:09:52Z

@jreback I think for small under 10.000obs is not gain to notice
This is with oservation > 1.000.000obs, we can see change so much. This is test with 1mil obs, I don't know why asv doesn't report ratio.

asv continuous -s -f 1.1 upstream/master 25708-fix-reregression-slicing -b ^io.csv.ToCSVDatetimeBig
· Creating environments
· Discovering benchmarks.
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Installing 0df6b571 <25708-fix-reregression-slicing> into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt....
· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)
[  0.00%] · For pandas commit db6993cd <master^2> (round 1/2):
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt......
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 25.00%] · For pandas commit 0df6b571 <25708-fix-reregression-slicing> (round 1/2):
[ 25.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.....
[ 25.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 50.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 50.00%] · For pandas commit 0df6b571 <25708-fix-reregression-slicing> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· io.csv.ToCSVDatetimeBig.time_frame                       16.6±0.2s
[ 75.00%] · For pandas commit db6993cd <master^2> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt......
[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· io.csv.ToCSVDatetimeBig.time_frame                         22.4±1s

BENCHMARKS NOT SIGNIFICANTLY CHANGED.

jreback

also i am not clear you are actually fixing anything. pls run your example with timeit on master and with the PR.

doc/source/whatsnew/v0.25.0.rst

jreback

pls show a timeit before/after this PR

asv_bench/benchmarks/io/csv.py

jreback · 2019-04-05T00:51:30Z

can you merge master and update

hksonngan · 2019-04-11T07:13:43Z

@jreback Here is CProfile upstream/master and branch fix.

Upstream/Master
Fix in branch

doc/source/whatsnew/v0.25.0.rst

jreback

pls also show the asv's results

doc/source/whatsnew/v0.25.0.rst

jreback · 2019-04-15T16:00:20Z

lgtm. ping on green.

hksonngan · 2019-04-15T16:17:25Z

@jreback ping.

hksonngan · 2019-04-15T18:14:16Z

'''
E:\dev\pandas\asv_bench>asv continuous -f 1.1 upstream/master HEAD -b ^io.csv.ToCSVDatetimeBig
· Creating environments
· Discovering benchmarks
·· Uninstalling from virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
·· Building 04afb4dd <25708-fix-reregression-slicing> for virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
·· Installing 04afb4dd <25708-fix-reregression-slicing> into virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)
[ 0.00%] · For pandas commit 7b3bf2d (round 1/2):
[ 0.00%] ·· Building for virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 0.00%] ·· Benchmarking virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 25.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 25.00%] · For pandas commit 04afb4dd <25708-fix-reregression-slicing> (round 1/2):
[ 25.00%] ·· Building for virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 25.00%] ·· Benchmarking virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 50.00%] ··· Running (io.csv.ToCSVDatetimeBig.time_frame--).
[ 50.00%] · For pandas commit 04afb4dd <25708-fix-reregression-slicing> (round 2/2):
[ 50.00%] ·· Benchmarking virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· io.csv.ToCSVDatetimeBig.time_frame ok
[ 75.00%] ··· ========== ============
obs
---------- ------------
1000 7.81±3ms
1000000 4.73±0.01s
10000000 49.1±0.4s
========== ============

[ 75.00%] · For pandas commit 7b3bf2d (round 2/2):
[ 75.00%] ·· Building for virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[ 75.00%] ·· Benchmarking virtualenv-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytest-scipy-sqlalchemy-tables-xlrd-xlsxwriter-xlwt
[100.00%] ··· io.csv.ToCSVDatetimeBig.time_frame ok
[100.00%] ··· ========== =============
obs
---------- -------------
1000 0±8000000ns
1000000 5.68±0.06s
10000000 2.16±0.01m
========== =============

   before           after         ratio
 [7b3bf2dc]       [04afb4dd]
 <master-upstream>       <25708-fix-reregression-slicing>

 2.16±0.01m        49.1±0.4s     0.38  io.csv.ToCSVDatetimeBig.time_frame(10000000)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
'''

…ype data

jreback · 2019-04-19T01:06:52Z

thanks @hksonngan

TomAugspurger · 2019-04-19T22:48:12Z

Can someone check the perf for this benchmark: http://pandas.pydata.org/speed/pandas/index.html#series_methods.IsInForObjects.time_isin_long_series_long_values_floats?commits=f53bb061939fb54d6c826c4db408c458d3d2a1db

Not sure if real or a spurious regression.

hksonngan · 2019-04-21T06:38:11Z

@TomAugspurger this is real, because of setting this params = [1000, 10000, 100000].

TomAugspurger · 2019-04-21T11:09:03Z

I don’t think so. The regression is for a different benchmark, no?

…ype data (pandas-dev#25765)

hksonngan changed the title ~~fixed regression slicing values~~ fixed regression slicing values to_native_types of DateTimeBlock Mar 18, 2019

jreback requested changes Mar 18, 2019

View reviewed changes

jreback added the Indexing Related to indexing on series/frames, not to indexes themselves label Mar 18, 2019

hksonngan changed the title ~~fixed regression slicing values to_native_types of DateTimeBlock~~ fixed performance degration becasue doesn not slice values array in to_native_types of DateTimeBlock Mar 19, 2019

hksonngan changed the title ~~fixed performance degration becasue doesn not slice values array in to_native_types of DateTimeBlock~~ fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock Mar 19, 2019

hksonngan changed the title ~~fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock~~ PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock Mar 19, 2019

jreback requested changes Mar 23, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

jreback requested changes Mar 26, 2019

View reviewed changes

asv_bench/benchmarks/io/csv.py Outdated Show resolved Hide resolved

jreback requested changes Apr 12, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

jreback requested changes Apr 15, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

jreback added this to the 0.25.0 milestone Apr 15, 2019

jreback approved these changes Apr 15, 2019

View reviewed changes

tunthoi and others added 2 commits April 18, 2019 17:05

Improved performance of :meth:DataFrame.to_csv when write datetime dt…

dc5acfe

…ype data

Merge branch 'master' into 25708-fix-reregression-slicing

d135af2

jreback merged commit f53bb06 into pandas-dev:master Apr 19, 2019

hksonngan deleted the 25708-fix-reregression-slicing branch April 19, 2019 14:11

yhaque1213 pushed a commit to yhaque1213/pandas that referenced this pull request Apr 22, 2019

Improved performance of :meth:DataFrame.to_csv when write datetime dt…

62e6671

…ype data (pandas-dev#25765)

ryanreh99 added a commit to ryanreh99/pandas that referenced this pull request Apr 22, 2019

Improved performance of :meth:DataFrame.to_csv when write datetime dt…

7aa8bb7

…ype data (pandas-dev#25765)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock #25765

PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock #25765

hksonngan commented Mar 18, 2019 •

edited

Loading

codecov bot commented Mar 18, 2019

codecov bot commented Mar 18, 2019 •

edited

Loading

jreback left a comment

jreback commented Mar 18, 2019

pep8speaks commented Mar 21, 2019 •

edited

Loading

hksonngan commented Mar 21, 2019 •

edited

Loading

hksonngan commented Mar 21, 2019

jreback commented Mar 22, 2019

jreback commented Mar 22, 2019

hksonngan commented Mar 22, 2019 •

edited

Loading

jreback left a comment

jreback left a comment

jreback commented Apr 5, 2019

hksonngan commented Apr 11, 2019

jreback left a comment

jreback commented Apr 15, 2019

hksonngan commented Apr 15, 2019

hksonngan commented Apr 15, 2019

jreback commented Apr 19, 2019

TomAugspurger commented Apr 19, 2019

hksonngan commented Apr 21, 2019

TomAugspurger commented Apr 21, 2019

PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock #25765

PERF: fixed performance degration becasue doesn't slice values array in to_native_types of DateTimeBlock #25765

Conversation

hksonngan commented Mar 18, 2019 • edited Loading

codecov bot commented Mar 18, 2019

Codecov Report

codecov bot commented Mar 18, 2019 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

jreback commented Mar 18, 2019

pep8speaks commented Mar 21, 2019 • edited Loading

Comment last updated at 2019-04-19 00:10:51 UTC

hksonngan commented Mar 21, 2019 • edited Loading

hksonngan commented Mar 21, 2019

jreback commented Mar 22, 2019

jreback commented Mar 22, 2019

hksonngan commented Mar 22, 2019 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented Apr 5, 2019

hksonngan commented Apr 11, 2019

jreback left a comment

Choose a reason for hiding this comment

jreback commented Apr 15, 2019

hksonngan commented Apr 15, 2019

hksonngan commented Apr 15, 2019

jreback commented Apr 19, 2019

TomAugspurger commented Apr 19, 2019

hksonngan commented Apr 21, 2019

TomAugspurger commented Apr 21, 2019

hksonngan commented Mar 18, 2019 •

edited

Loading

codecov bot commented Mar 18, 2019 •

edited

Loading

pep8speaks commented Mar 21, 2019 •

edited

Loading

hksonngan commented Mar 21, 2019 •

edited

Loading

hksonngan commented Mar 22, 2019 •

edited

Loading