Fix for DataFrame.hist() with by- and weights-keyword #11441

Twizzledrizzle · 2015-10-27T14:23:49Z

will make the following work

import numpy as np
import pandas as pd
d = {'one' : ['A', 'A', 'B', 'B', 'C'],
     'two' : [4., 3., 2., 1., np.nan],
     'three' : [10., 8., np.nan, 5., 7.]}     
df = pd.DataFrame(d)
df.hist('two', by='one', weights='three', bins=range(0, 10))
# or
df.hist('two', by=df.one.values, weights='three', bins=range(0, 10))

will make the following work import pandas as pd d = {'one' : ['A', 'A', 'B', 'C'], 'two' : [4., 3., 2., 1.], 'three' : [10., 8., 5., 7.]} df = pd.DataFrame(d) df.hist('two', by='one', weights='three', bins=range(0, 10))

Twizzledrizzle · 2015-10-27T14:26:55Z

Because of my poor git skills, this is a continuation for
#11028

upgraded to work with the newest pandas release

I am at loss writing a test for this. @jreback I could not find the place you recommended. The place where weights are tested right now seems to be test_generic.py?

However I am unsure of how to construct a test the best way, I have never written one, so pointers are very welcome!

Also fix if NaN's make an entire group empty

… in the DataFrame

Twizzledrizzle · 2015-10-27T19:30:43Z

Note, I added the column 'weighst' to the supplied DataFrame if an array was supplied instead of the column name... Can this be done in a nicer way? I need it in the groupby logic.

if a column is named 'weights' it will be overwritten... :(

TomAugspurger · 2015-10-27T20:25:24Z

@Twizzledrizzle no worries about the git snafu, we've all been there. People are usually in the chat if you need help walking through git.

I think Jeff was talking about the logic in DataFrame.sample in pandas/core/frame.py. I haven't had a chance to look at your code to see if it's similar enough to refactor into a common function.

For the tests you can look to see how similar tests are done in pandas/tests/test_graphics.py. Basically you call the function your testing and assert that something about the result equals an expected value that you hardcode in the test.

I'll take a look later, but it won't be until next week sometime.

…nymore Also made the logic better when using weights without group by by aligning NaN's, and finding out if weights is supplied by column or by array.

Twizzledrizzle · 2015-10-27T20:42:16Z

Thanks @TomAugspurger I will take a look.

A lot of commits right now, finding things while working through the travis errors... I hope the server does not crash!

TomAugspurger · 2015-10-27T20:47:25Z

Yeah, we'll come up with a better way to solve that weights column.

FYI you can run the tests locally, will be quicker than waiting on Travis. Make sure you have nose installed (pip install nose or conda install nose) and then run the tests with nosetests test_graphics from the pandas/tests directory.

Run specific tests with nosetests test_graphics:TestClass.test_function (I think)

Twizzledrizzle · 2015-10-28T09:40:55Z

I have python 2.7 and every tests seems to work fine. Why could python 3.4 fail on

nosetests test_data:TestGoogle.test_get_multi1

Does not seem to have anything to do with my pull request?

jreback · 2015-10-28T10:25:53Z

pandas/tools/plotting.py

-               xrot=None, ylabelsize=None, yrot=None, ax=None, sharex=False,
-               sharey=False, figsize=None, layout=None, bins=10, **kwds):
+def hist_frame(data, column=None, weights=None, by=None, grid=True,
+               xlabelsize=None, xrot=None, ylabelsize=None, yrot=None, ax=None,


don't change the order of parameters, just add at th eend

jreback · 2015-10-28T10:27:15Z

need tests

jreback · 2015-10-28T10:28:04Z

@Twizzledrizzle ignore that goggle multi failure....its an unreliable test on travis

Uses dropna(subset=...) to delete where nan's over the columns supplied Also doing this in the beginning so we do not have to duplicate this logic

Twizzledrizzle · 2015-10-28T13:31:05Z

Made the code more readable now I think, I hope it looks better.

Regarding tests, are we talking tests like this? In test_graphics_others.py? (Looks like some other tests are done one the hist functionality there).

ax = _check_plot_works(df.hist, column='two', by='one', weights='three', bins=range(0, 10))

Twizzledrizzle · 2015-10-28T21:04:16Z

Oh also, as a side, the new function df.plot.hist() will miss a lot of this functionality.

df.plot.hist() seem to have problems using the by- keyword. Also missing column keyword. I tried working with the code, but it was more complex than the one I have been editing. I could not get my head around making the by-keyword work.

Should I open a discussion around this perhaps in a separate thread?

Twizzledrizzle · 2015-10-30T13:45:10Z

pandas/tools/plotting.py

+    subset_cols_drop_nan = []
+    if weights is not None:
+        if isinstance(weights, np.ndarray):
+            # weights supplied as an array instead of a part of the dataframe


Hmm, this will not work if weights is a >1 dimensional ndarray... Need to think on this

jreback · 2015-12-06T19:17:31Z

@TomAugspurger status of this?

Twizzledrizzle · 2015-12-06T20:54:34Z

I am awaiting with interest what will happen with

#8018

before putting to much time in this pull request.

jreback · 2016-01-11T13:44:52Z

@TomAugspurger status of this

jreback · 2016-03-12T17:47:59Z

@TomAugspurger status?

TomAugspurger · 2016-03-21T12:28:30Z

I think waiting on #8018, which will conflict with the case where by is not None here.

jreback · 2016-09-09T22:45:26Z

status?

jreback · 2016-12-21T23:05:01Z

@TomAugspurger can this be independent #8018 ?

jreback · 2017-03-20T13:49:52Z

@TomAugspurger ?

jreback · 2017-04-03T15:25:51Z

closing as stale

Fix for DataFrame.hist() with by- and weights-keyword

20a02c3

will make the following work import pandas as pd d = {'one' : ['A', 'A', 'B', 'C'], 'two' : [4., 3., 2., 1.], 'three' : [10., 8., 5., 7.]} df = pd.DataFrame(d) df.hist('two', by='one', weights='three', bins=range(0, 10))

Twizzledrizzle mentioned this pull request Oct 27, 2015

Fix for DataFrame.hist() with by- and weights-keyword #11028

Closed

nosterlu added 2 commits October 27, 2015 19:17

Fix for handling both ndarray and series.

5bbba11

Also fix if NaN's make an entire group empty

Fix for if weights is supplied by an array instead of the column name…

787577e

… in the DataFrame

Fixed bug where weights were not used, since not included in **kwds a…

b6bcb5c

…nymore Also made the logic better when using weights without group by by aligning NaN's, and finding out if weights is supplied by column or by array.

jreback reviewed Oct 28, 2015
View reviewed changes

jreback added the Visualization plotting label Oct 28, 2015

changed weights= to end of functions

cbe68ec

Uses dropna(subset=...) to delete where nan's over the columns supplied Also doing this in the beginning so we do not have to duplicate this logic

Added the first test (in my life)

868fbf0

Twizzledrizzle mentioned this pull request Oct 30, 2015

By-keyword, Inconcistencies between the new df.plot.hist and its df counterpart #11483

Open

Twizzledrizzle reviewed Oct 30, 2015
View reviewed changes

jreback closed this Apr 3, 2017

nmusolino mentioned this pull request Mar 30, 2018

Using 'by' and 'weights' together with DataFrame.hist() results in ValueError: weights should have the same shape as x #9540

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for DataFrame.hist() with by- and weights-keyword #11441

Fix for DataFrame.hist() with by- and weights-keyword #11441

Twizzledrizzle commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

TomAugspurger commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

TomAugspurger commented Oct 27, 2015

Twizzledrizzle commented Oct 28, 2015

jreback Oct 28, 2015

jreback commented Oct 28, 2015

jreback commented Oct 28, 2015

Twizzledrizzle commented Oct 28, 2015

Twizzledrizzle commented Oct 28, 2015

Twizzledrizzle Oct 30, 2015

jreback commented Dec 6, 2015

Twizzledrizzle commented Dec 6, 2015

jreback commented Jan 11, 2016

jreback commented Mar 12, 2016

TomAugspurger commented Mar 21, 2016

jreback commented Sep 9, 2016

jreback commented Dec 21, 2016

jreback commented Mar 20, 2017

jreback commented Apr 3, 2017

Fix for DataFrame.hist() with by- and weights-keyword #11441

Fix for DataFrame.hist() with by- and weights-keyword #11441

Conversation

Twizzledrizzle commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

TomAugspurger commented Oct 27, 2015

Twizzledrizzle commented Oct 27, 2015

TomAugspurger commented Oct 27, 2015

Twizzledrizzle commented Oct 28, 2015

jreback Oct 28, 2015

Choose a reason for hiding this comment

jreback commented Oct 28, 2015

jreback commented Oct 28, 2015

Twizzledrizzle commented Oct 28, 2015

Twizzledrizzle commented Oct 28, 2015

Twizzledrizzle Oct 30, 2015

Choose a reason for hiding this comment

jreback commented Dec 6, 2015

Twizzledrizzle commented Dec 6, 2015

jreback commented Jan 11, 2016

jreback commented Mar 12, 2016

TomAugspurger commented Mar 21, 2016

jreback commented Sep 9, 2016

jreback commented Dec 21, 2016

jreback commented Mar 20, 2017

jreback commented Apr 3, 2017