extended pmap with kw args #3715
Conversation
|
Great addition. Could you update the docs? |
|
We need better names though. Perhaps hold on the doc change until we can bikeshed this? |
|
I'm a little confused about exactly what these mean – are they not always the opposite of each other? It seems like |
|
Well it's possible that one could want to not stop on errors and also not retry failed tasks. |
|
So the reasonable options in the presence of failures are:
|
| np = nprocs() | ||
| n = length(lsts[1]) | ||
| results = cell(n) | ||
| queue = [1:n] | ||
| i = 1 | ||
| errors = Set() |
There was a problem hiding this comment.
Using a bool flag here did not work, hence the Set object.
|
stop and fail immediately, - if on_err_stop = true, value of on_err_retry should be ignored retry failed jobs - on_err_retry should be true AND on_err_stop should be false don't retry but keep going, returning incomplete results. - on_err_retry is false, on_err_stop is false - results will have exception objects at the failed indexes Will update the pull request. Also, I think the names are OK. A bit long but clear. |
|
Updated. Renamed to |
|
Bump. Are the revised names acceptable? Anything else needs to be done? |
extended pmap with kw args
|
Do you think we should change the behavior of |
|
I suggest leaving it as is. When people start doing massively parallel stuff, they usually would like to be tolerant of a few requests failing. For example, a Google like distributed query on a web index. |
Stdlib: Pkg URL: https://github.com/JuliaLang/Pkg.jl.git Stdlib branch: master Julia branch: master Old commit: 85f1e5564 New commit: 3c86ba27e Julia version: 1.11.0-DEV Pkg version: 1.11.0 Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: JuliaLang/Pkg.jl@85f1e55...3c86ba2 ``` $ git log --oneline 85f1e5564..3c86ba27e 3c86ba27e add `add --weak/extra Foo` to add to [weakdeps] or [extras] (#3708) 2e640f92f respect --color=no in Pkg.precompile (#3740) cbd5d08ad Automatically add compat entries when adding deps to a package (#3732) 03de920b3 rm old manual handling of `--compiled-modules` (#3738) 314d5497b Use realpaths for temp dirs during tests. Fix SparseArrays `why` breakage (#3734) a6531d4be environments.md: update Julia version (#3715) a509bc062 Revise the API of is_manifest_current. (#3701) 60b7b7995 rm incorrect kwargs in add docstring (#3733) ``` Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
on_err_retry, defaulttrue, specifies if pmap should retry a failed task on a different worker.on_err_stop, defaultfalse, specifies if pmap should not execute any pending tasks in case of an exception an any currently running tasks.Default values mimic current behavior.
runtests.jlhas been updated to stop on the first error.