Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename get_default_shuffle_algorithm to get_default_shuffle_method #7902

Merged
merged 6 commits into from
Jun 13, 2023
Merged

Rename get_default_shuffle_algorithm to get_default_shuffle_method #7902

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d028e04
Rename to get_default_shuffle_method
hendrikmakait Jun 12, 2023
47c4fcd
Temporarily point to dask/dask PR [skip-caching]
jrbourbeau Jun 12, 2023
bd310aa
gpuCI
jrbourbeau Jun 12, 2023
a260ce7
pre-commit [skip-caching]
jrbourbeau Jun 12, 2023
cec93c4
gpuCI again [skip-caching]
jrbourbeau Jun 12, 2023
43f746a
Revert "Temporarily point to dask/dask PR [skip-caching]"
hendrikmakait Jun 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion continuous_integration/environment-3.10.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ dependencies:
- zict # overridden by git tip below
- zstandard >=0.9.0
- pip:
- git+https://github.com/dask/dask
# - git+https://github.com/dask/dask
- git+https://github.com/hendrikmakait/dask.git@default-shuffle-ignores-fuse
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just flagging: Revert me

- git+https://github.com/dask/s3fs
- git+https://github.com/dask/zict
- git+https://github.com/fsspec/filesystem_spec
Expand Down
3 changes: 2 additions & 1 deletion continuous_integration/environment-3.11.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ dependencies:
- zict # overridden by git tip below
- zstandard >=0.9.0
- pip:
- git+https://github.com/dask/dask
# - git+https://github.com/dask/dask
- git+https://github.com/hendrikmakait/dask.git@default-shuffle-ignores-fuse
- git+https://github.com/dask/s3fs
- git+https://github.com/dask/zict
- git+https://github.com/fsspec/filesystem_spec
Expand Down
3 changes: 2 additions & 1 deletion continuous_integration/environment-3.9.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ dependencies:
- zict
- zstandard >=0.9.0
- pip:
- git+https://github.com/dask/dask
# - git+https://github.com/dask/dask
- git+https://github.com/hendrikmakait/dask.git@default-shuffle-ignores-fuse
- git+https://github.com/dask/crick # Only tested here
- keras
3 changes: 2 additions & 1 deletion continuous_integration/environment-mindeps.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ dependencies:
# Distributed depends on the latest version of Dask
- pip
- pip:
- git+https://github.com/dask/dask
# - git+https://github.com/dask/dask
- git+https://github.com/hendrikmakait/dask.git@default-shuffle-ignores-fuse
# test dependencies
- pytest
- pytest-cov
Expand Down
19 changes: 7 additions & 12 deletions distributed/tests/test_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,7 @@
import dask.bag as db
from dask import delayed
from dask.optimization import SubgraphCallable
from dask.utils import (
get_default_shuffle_algorithm,
parse_timedelta,
stringify,
tmpfile,
)
from dask.utils import get_default_shuffle_method, parse_timedelta, stringify, tmpfile

from distributed import (
CancelledError,
Expand Down Expand Up @@ -3229,25 +3224,25 @@ def test_default_get(loop_in_thread):
# These may change in the future but the selection below should not
distributed_default = "p2p" if has_pyarrow else "tasks"
local_default = "disk"
assert get_default_shuffle_algorithm() == local_default
assert get_default_shuffle_method() == local_default
with Client(s["address"], set_as_default=True, loop=loop) as c:
assert dask.base.get_scheduler() == c.get
assert get_default_shuffle_algorithm() == distributed_default
assert get_default_shuffle_method() == distributed_default

assert dask.base.get_scheduler() == pre_get
assert get_default_shuffle_algorithm() == local_default
assert get_default_shuffle_method() == local_default

c = Client(s["address"], set_as_default=False, loop=loop)
assert dask.base.get_scheduler() == pre_get
assert get_default_shuffle_algorithm() == local_default
assert get_default_shuffle_method() == local_default
c.close()

c = Client(s["address"], set_as_default=True, loop=loop)
assert get_default_shuffle_algorithm() == distributed_default
assert get_default_shuffle_method() == distributed_default
assert dask.base.get_scheduler() == c.get
c.close()
assert dask.base.get_scheduler() == pre_get
assert get_default_shuffle_algorithm() == local_default
assert get_default_shuffle_method() == local_default

with Client(s["address"], loop=loop) as c:
assert dask.base.get_scheduler() == c.get
Expand Down