POC: Use khash sets instead of maps for isin #53059
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
realead
reviewed
May 4, 2023
|
Just found out about the perf tool to measure branch prediction misses. Just ran this for a benchmark:
On main I get the following: Performance counter stats for 'python -c import pandas as pd; import numpy as np; N = 1_000_000; ser = pd.Series(np.arange(N)); vals = np.arange(N); ser.isin(vals)':
1,965.47 msec task-clock # 2.357 CPUs utilized
147 context-switches # 74.791 /sec
0 cpu-migrations # 0.000 /sec
31,457 page-faults # 16.005 K/sec
3,936,570,319 cpu_core/cycles/ # 2.003 G/sec
3,802,321,801 cpu_atom/cycles/ # 1.935 G/sec (52.15%)
7,214,481,236 cpu_core/instructions/ # 3.671 G/sec
8,924,412,911 cpu_atom/instructions/ # 4.541 G/sec (52.15%)
1,376,744,013 cpu_core/branches/ # 700.464 M/sec
1,457,321,116 cpu_atom/branches/ # 741.460 M/sec (52.15%)
23,755,435 cpu_core/branch-misses/ # 12.086 M/sec
76,473 cpu_atom/branch-misses/ # 38.908 K/sec (52.15%)
19,950,214,260 cpu_core/slots/ # 10.150 G/sec
6,988,756,509 cpu_core/topdown-retiring/ # 34.9% Retiring
3,077,912,645 cpu_core/topdown-bad-spec/ # 15.4% Bad Speculation
5,841,369,682 cpu_core/topdown-fe-bound/ # 29.2% Frontend Bound
4,090,935,951 cpu_core/topdown-be-bound/ # 20.5% Backend Bound
607,579,968 cpu_core/topdown-heavy-ops/ # 3.0% Heavy Operations # 31.9% Light Operations
2,935,493,211 cpu_core/topdown-br-mispredict/ # 14.7% Branch Mispredict # 0.7% Machine Clears
2,838,977,426 cpu_core/topdown-fetch-lat/ # 14.2% Fetch Latency # 15.0% Fetch Bandwidth
2,956,225,784 cpu_core/topdown-mem-bound/ # 14.8% Memory Bound # 5.7% Core Bound
0.833991099 seconds time elapsed
1.010541000 seconds user
0.958761000 seconds sysVersus this PR: Performance counter stats for 'python -c import pandas as pd; import numpy as np; N = 1_000_000; ser = pd.Series(np.arange(N)); vals = np.arange(N); ser.isin(vals)':
1,954.42 msec task-clock # 2.370 CPUs utilized
215 context-switches # 110.007 /sec
2 cpu-migrations # 1.023 /sec
26,302 page-faults # 13.458 K/sec
3,927,968,672 cpu_core/cycles/ # 2.010 G/sec
3,786,117,128 cpu_atom/cycles/ # 1.937 G/sec (52.56%)
7,217,333,673 cpu_core/instructions/ # 3.693 G/sec
8,849,613,601 cpu_atom/instructions/ # 4.528 G/sec (52.56%)
1,378,602,285 cpu_core/branches/ # 705.376 M/sec
1,443,771,308 cpu_atom/branches/ # 738.720 M/sec (52.56%)
23,818,420 cpu_core/branch-misses/ # 12.187 M/sec
299,996 cpu_atom/branch-misses/ # 153.496 K/sec (52.56%)
19,880,779,164 cpu_core/slots/ # 10.172 G/sec
6,961,062,372 cpu_core/topdown-retiring/ # 35.0% Retiring
3,064,425,117 cpu_core/topdown-bad-spec/ # 15.4% Bad Speculation
5,844,723,817 cpu_core/topdown-fe-bound/ # 29.4% Frontend Bound
4,026,716,906 cpu_core/topdown-be-bound/ # 20.2% Backend Bound
600,833,950 cpu_core/topdown-heavy-ops/ # 3.0% Heavy Operations # 32.0% Light Operations
2,926,332,561 cpu_core/topdown-br-mispredict/ # 14.7% Branch Mispredict # 0.7% Machine Clears
2,818,938,163 cpu_core/topdown-fetch-lat/ # 14.2% Fetch Latency # 15.2% Fetch Bandwidth
2,943,103,050 cpu_core/topdown-mem-bound/ # 14.8% Memory Bound # 5.4% Core Bound
0.824527840 seconds time elapsed
1.034003000 seconds user
0.924299000 seconds sysSo not sure this really makes a big difference for branch predictions. Will have to research more |
|
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
|
Closing for now but would be nice to pick up again in the future |
|
Expanded this to other primitives aside from int64 / uint64. Across the board seems like this helps, although there are some regressions. Needs more investigation Haven't done the PyObject case yet as the naming conventions there don't follow the same conventions as described here. Might need to tackle that in a follow up |
|
OK got everything set up. Some of the prior regressions were due to improper macro use / declarations. Surprised those didn't throw compiler errors...but there are many layers of indirection between tempita / khash. Something to investigate another day... Here are the results for a full run of the isin benchmarks - looks like this does help with scalability as larger datasets are showing 20-50% improvement @mroeschke @jbrockmendel @realead for review |
WillAyd
added
Performance
mroeschke
approved these changes
Dec 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a POC towards what @realead described in #39799
The IsIn benchmarks overall seemed a bit unreliable, but I could consistently get results of
algos.isin.IsinWithArangeSortedthat look like:>>> asv continuous upstream/main HEAD -b algos.isin.IsinWithArangeSorted before after ratio [38881793] [939529bb] <main> <khash-set-poc> - 47.3±1μs 42.9±0.3μs 0.91 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.int64'>, 2000) - 715±30μs 601±4μs 0.84 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.int64'>, 100000) - 107±5μs 89.6±1μs 0.84 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.int64'>, 8000) - 26.5±0.4ms 12.2±0.7ms 0.46 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.int64'>, 1000000) SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY. PERFORMANCE INCREASED.The performance improvement on the largest dataset might be in line with @realead expectation that
For big datasets, the running time of the above algorithms is dominated by cache-misses. Thus having twice as many cache-misses, because also values are touched could mean a factor 2 slowdown