Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dataset pickling #7

Merged
merged 2 commits into from
Apr 2, 2024
Merged

Fix dataset pickling #7

merged 2 commits into from
Apr 2, 2024

Commits on Mar 26, 2024

  1. Add test to trigger pickling error 

    Computing a dataset using "processes" or a Client scheduler currently
fails due to a pickling error. This adds a corresponding test that will
need to be fixed.
    @aazuspan
    aazuspan committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    8464a96 View commit details
    Browse the repository at this point in the history

  2. Fix failing pickling test 

    `PosixPath.open()` returns a `BufferedReader` that is not pickleable,
breaking dask computation with `Client` or "processes" schedulers. This
still uses `resources` to identify data file paths, but opens directly
from the paths instead of file buffers to fix pickling.
    @aazuspan
    aazuspan committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    1995369 View commit details
    Browse the repository at this point in the history