adding an atime test case for gshift as gforce optimized shift #5205

[DorisAmoakohene]

In this test case the user mentions that the shift() function in data.table version 1.9.6 is slow when applied to many groups. The exact cause or specific use case is not elaborated upon in the issue.visit here to read more

The pull request address the issue by allowing GForce optimization for lapply even without using .SD1.

[github-actions]

Generated via commit 187c79c

Download link for the artifact containing the test results: ↓ atime-results.zip

Time taken to finish the standard R installation steps: 3 minutes and 29 seconds

Time taken to run atime::atime_pkg on the tests: 9 minutes and 7 seconds

[tdhock]

Hi I converted to draft because I think you should first merge your first PR, and then use what you learn from that experience to improve your other PRs, before asking others to review them.

[DorisAmoakohene]

Hi I converted to draft because I think you should first merge your first PR, and then use what you learn from that experience to improve your other PRs, before asking others to review them.

I have made most corrections to this PR for review and merge

[tdhock]

please resolve conflicts and fix typo shif -> shift

[DorisAmoakohene]

please resolve conflicts and fix typo shif -> shift

I have resolved this. Thank you

[tdhock]

currently we see this result

there are three issues

1. there is not a huge time separation between Regression and Fixed at largest N, but there is some separation at smaller N, why? Can you please investigate, perhaps with a larger N? and double check the historical commit IDs?
2. There is no Before, so please either add that, or change to Fast and Slow.
3. why does Regression use less memory than Fixed?

[DorisAmoakohene]

currently we see this result

there are three issues

1. there is not a huge time separation between Regression and Fixed at largest N, but there is some separation at smaller N, why? Can you please investigate, perhaps with a larger N? and double check the historical commit IDs?
2. There is no Before, so please either add that, or change to Fast and Slow.
3. why does Regression use less memory than Fixed?

Alright

[DorisAmoakohene]

currently we see this result

there are three issues

1. there is not a huge time separation between Regression and Fixed at largest N, but there is some separation at smaller N, why? Can you please investigate, perhaps with a larger N? and double check the historical commit IDs?
2. There is no Before, so please either add that, or change to Fast and Slow.
3. why does Regression use less memory than Fixed?

I investigated further with a larger sample size (N), but found no difference, it stops when it gets to 10000

. Even when I reverted to the commit to the commit before the last commits, where the PR was merge nothing changes.

[tdhock]

the original benchmarks were reported by @ben-schwen in #5205 (comment)
there are two sizes in the benchmarks.

• number of rows of DT
• number of values in sample (y column)

your current setup has DT = data.table(x = sample(N), y = sample(1e2, N, TRUE)) so N controls number of rows of DT, not number of values in sample (y column) which is currently fixed at 1e2. Maybe you could try varying that instead?
Or varying both?

[tdhock]

also @ben-schwen do you think the memory differences we observe are normal/expected?