Skip to content

Commit f88c49b

Browse files
authored
[Data] Increase scale of map_groups release tests from SF10 to SF100 (#58710)
> Thank you for contributing to Ray! 🚀 > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. > ⚠️ Remove these instructions before submitting your PR. > 💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. #58711 decreased the scale of the `map_groups` tests from scale-factor 100 to scale-factor 10 because some of the `map_groups` release tests were failing. However, after more investigation, I realized that the only variant that doesn't work with scale-factor 100 is the hash shuffle with autoscaling variant (see #58734). This PR re-increases the scale and only disables the cases that fail. ## Related issues > Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
1 parent eb28037 commit f88c49b

File tree

1 file changed

+15
-5
lines changed

1 file changed

+15
-5
lines changed

release/release_data_tests.yaml

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -178,22 +178,32 @@
178178

179179
matrix:
180180
setup:
181-
# This test consistently fails on fixed-size clusters due to head OOM from
182-
# too many objects references on the head node. So, we only run it on
183-
# autoscaling clusters.
184-
scaling: [autoscaling]
181+
scaling: [fixed_size]
185182
shuffle_strategy: [sort_shuffle_pull_based, hash_shuffle]
186183
columns:
187184
- "column08 column13 column14" # 84 groups
188185
- "column02 column14" # 7M groups
189186

187+
adjustments:
188+
# Ray Data OOMs with hash shuffle on autoscaling clusters. So, only run
189+
# autoscaling variants with sort shuffle. For more information, see
190+
# https://github.com/ray-project/ray/issues/58734.
191+
- with:
192+
scaling: autoscaling
193+
shuffle_strategy: sort_shuffle_pull_based
194+
columns: "column08 column13 column14"
195+
- with:
196+
scaling: autoscaling
197+
shuffle_strategy: sort_shuffle_pull_based
198+
columns: "column02 column14"
199+
190200
cluster:
191201
cluster_compute: "{{scaling}}_all_to_all_compute.yaml"
192202

193203
run:
194204
timeout: 3600
195205
script: >
196-
python groupby_benchmark.py --sf 10 --map-groups --group-by {{columns}}
206+
python groupby_benchmark.py --sf 100 --map-groups --group-by {{columns}}
197207
--shuffle-strategy {{shuffle_strategy}}
198208
199209
###############

0 commit comments

Comments
 (0)