Calculate normalized freqs in value_counts correctly when bins is not None

[krsnik93]

• closes value_counts unexpected behaviour - bins and dropna #25970
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@df3d9b2). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #26923   +/-   ##
=========================================
  Coverage          ?   91.86%           
=========================================
  Files             ?      180           
  Lines             ?    50745           
  Branches          ?        0           
=========================================
  Hits              ?    46619           
  Misses            ?     4126           
  Partials          ?        0

Flag	Coverage Δ
#multiple	90.46% <100%> (?)
#single	41.12% <0%> (?)

Impacted Files	Coverage Δ
pandas/core/algorithms.py	94.72% <100%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df3d9b2...81bbfa6. Read the comment docs.

[jreback]

can you add a whatsnew note, reshape section of bug fixes

[jreback]

can you pull the dtypes into another level of parameterization

[pep8speaks]

Hello @krsnik93! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-09-09 07:34:18 UTC

[jreback]

@krsnik93 can you merge master & add a whatsnew note (bug fix in reshaping is ok). ping on green.

[krsnik93]

I have a failing test in tests/groupby/test_value_counts.py. The test compares Series.value_counts against SeriesGroupBy.value_counts. For this issue, I modified Series.value_counts and I think the change is correct. However, I think SeriesGroupBy.value_counts should change as well. This is the troubling bit:

pandas/pandas/core/groupby/generic.py

Lines 1261 to 1275 in c275dbf

	# for compat. with libgroupby.value_counts need to ensure every
	# bin is present at every index level, null filled with zeros
	diff = np.zeros(len(out), dtype='bool')
	for lab in labels[:-1]:
	diff |= np.r_[True, lab[1:] != lab[:-1]]
	ncat, nbin = diff.sum(), len(levels[-1])
	left = [np.repeat(np.arange(ncat), nbin),
	np.tile(np.arange(nbin), ncat)]
	right = [diff.cumsum() - 1, labels[-1]]
	_, idx = _get_join_indexers(left, right, sort=False, how='left')
	out = np.where(idx != -1, out[idx], 0)

For some reason, it takes by-group counts of unbinned data (either NaN values or outside of bin breaks) and resets those counts to 0 (last line). I do not understand why is this done, but the whole method is kind hard to read and I am not able to make the required changes.

[TomAugspurger]

cc @jschendel if you have time to take a look at #26923 (comment).

[WillAyd]

@krsnik93 can you merge master and fix up merge conflicts?

[WillAyd]

Closing as this looks stale, but @krsnik93 please ping if you'd like to continue and can open back up