Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize window function's find_ranges_in_range #595

Merged
merged 1 commit into from
Jun 23, 2021

Conversation

jimexist
Copy link
Member

@jimexist jimexist commented Jun 22, 2021

Which issue does this PR close?

Optimize window function's find_ranges_in_range

Based on #564 so review that first
Closes #.

Rationale for this change

benchmark:

Benchmarking window empty over, aggregate functions: Collecting 100 samples in estimated 7.0674 s (200 iterati                                                                                                              window empty over, aggregate functions
                        time:   [33.115 ms 33.588 ms 34.129 ms]
                        change: [-17.056% -15.622% -14.056%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

Benchmarking window empty over, built-in functions: Collecting 100 samples in estimated 5.9518 s (200 iteratio                                                                                                              window empty over, built-in functions
                        time:   [27.812 ms 28.113 ms 28.429 ms]
                        change: [-23.131% -21.112% -19.100%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Benchmarking window order by, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 20.8s, or reduce sample count to 20.
Benchmarking window order by, aggregate functions: Collecting 100 samples in estimated 20.778 s (100 iteration                                                                                                              window order by, aggregate functions
                        time:   [191.15 ms 196.85 ms 202.61 ms]
                        change: [-13.162% -10.553% -7.7583%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking window order by, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 17.9s, or reduce sample count to 20.
Benchmarking window order by, built-in functions: Collecting 100 samples in estimated 17.919 s (100 iterations                                                                                                              window order by, built-in functions
                        time:   [192.66 ms 199.27 ms 206.93 ms]
                        change: [-8.1837% -5.1872% -1.3218%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

Benchmarking window partition by, u64_wide, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 27.8s, or reduce sample count to 10.
Benchmarking window partition by, u64_wide, aggregate functions: Collecting 100 samples in estimated 27.777 s                                                                                                               window partition by, u64_wide, aggregate functions
                        time:   [261.49 ms 265.26 ms 268.94 ms]
                        change: [-18.841% -17.347% -15.891%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking window partition by, u64_narrow, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 14.1s, or reduce sample count to 30.
Benchmarking window partition by, u64_narrow, aggregate functions: Collecting 100 samples in estimated 14.061                                                                                                               window partition by, u64_narrow, aggregate functions
                        time:   [133.94 ms 135.53 ms 137.16 ms]
                        change: [-19.379% -17.908% -16.493%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking window partition by, u64_wide, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 22.2s, or reduce sample count to 20.
Benchmarking window partition by, u64_wide, built-in functions: Collecting 100 samples in estimated 22.179 s (                                                                                                              window partition by, u64_wide, built-in functions
                        time:   [213.65 ms 215.64 ms 217.65 ms]
                        change: [-20.451% -19.390% -18.336%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

Benchmarking window partition by, u64_narrow, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.8s, or reduce sample count to 40.
Benchmarking window partition by, u64_narrow, built-in functions: Collecting 100 samples in estimated 11.849 s                                                                                                              window partition by, u64_narrow, built-in functions
                        time:   [120.97 ms 122.49 ms 124.03 ms]
                        change: [-22.048% -20.534% -19.035%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking window partition and order by, u64_wide, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 45.8s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_wide, aggregate functions: Collecting 100 samples in estimated                                                                                                              window partition and order by, u64_wide, aggregate functions
                        time:   [434.30 ms 437.73 ms 441.20 ms]
                        change: [-6.4053% -5.1483% -3.8173%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking window partition and order by, u64_narrow, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 64.5s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_narrow, aggregate functions: Collecting 100 samples in estimat                                                                                                              window partition and order by, u64_narrow, aggregate functions
                        time:   [607.27 ms 610.83 ms 614.46 ms]
                        change: [-5.2088% -4.0570% -2.8806%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking window partition and order by, u64_wide, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 39.4s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_wide, built-in functions: Collecting 100 samples in estimated                                                                                                               window partition and order by, u64_wide, built-in functions
                        time:   [387.44 ms 390.74 ms 394.56 ms]
                        change: [-0.4466% +0.8918% +2.3908%] (p = 0.21 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

Benchmarking window partition and order by, u64_narrow, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 38.4s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_narrow, built-in functions: Collecting 100 samples in estimate                                                                                                              window partition and order by, u64_narrow, built-in functions
                        time:   [384.46 ms 389.51 ms 395.09 ms]
                        change: [+2.4390% +4.1822% +5.9735%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

What changes are included in this PR?

Are there any user-facing changes?

@jimexist jimexist changed the title Window partition optimize ranges optimize window function's find_ranges_in_range Jun 22, 2021
@jimexist jimexist force-pushed the window-partition-optimize-ranges branch from 827a331 to 4711836 Compare June 22, 2021 10:11
@jimexist jimexist force-pushed the window-partition-optimize-ranges branch from 4711836 to b2ed86f Compare June 23, 2021 00:30
@alamb
Copy link
Contributor

alamb commented Jun 23, 2021

Makes sense to me

@alamb alamb added the datafusion Changes in the datafusion crate label Jun 23, 2021
@alamb alamb merged commit 35a0f30 into apache:master Jun 23, 2021
@jimexist jimexist deleted the window-partition-optimize-ranges branch June 23, 2021 13:29
@houqp houqp added the performance Make DataFusion faster label Jul 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datafusion Changes in the datafusion crate performance Make DataFusion faster
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants