WISH: Allow for mc.cores=0 in parallel::mclapply() and friends

[HenrikBengtsson]

Background

From help("options") we have that mc.cores is default as:

mc.cores:
a integer giving the maximum allowed number of additional R processes allowed to be run in parallel to the current R process. Defaults to the setting of the environment variable MC_CORES if set. Most applications which use this assume a limit of 2 if it is unset.

From this definition, I would interpret mc.cores = 0, 1, 2 to mean:

• mc.cores = 0: Only the main R process may run.
• mc.cores = 1: The main R process plus one more forked process may run.
• mc.cores = 2: The main R process plus two more forked processes may run.

Comment: This means that from a computational point of view it makes little sense to use mc.cores = 1 iff you're using parallel::mclapply() and friends, because it forks off a single R processing (with the main process only polling/waiting for it to finish) and performs the same calculation that you could have done in the main R process alone. In this sense, mc.cores = 1 could effectively be doing/implemented the same as mc.cores = 0. (However, you could imagine implementations that are making full use of exactly two R processes. This is for instance possible to do using the future package.)

On compute clusters with schedulers such as PBS and Slurm, you submit jobs and request the number of cores you would need. If you request a single-core process, it makes sense to do all calculations in the main R process. Thus, we should really use mc.cores = 0 whenever allocated single-core R sessions. If we use mc.cores = 1, we are actually consuming two processes.

Problem

Currently, mc.cores = 0 gives an error when used by parallel::mclapply() and friends , e.g.

>  parallel::mclapply(1:3, FUN=rnorm, mc.cores=0L)
Error in parallel::mclapply(1:3, FUN = rnorm, mc.cores = 0L) :  'mc.cores' must be >= 1

> options(mc.cores=0L)
> parallel::mclapply(1:3, FUN=rnorm)
Error in parallel::mclapply(1:3, FUN = rnorm) : 'mc.cores' must be >= 1

This means that in order to write code that is agile to cluster settings and work with any number of allocated cores, we need to tedious coding such as:

if (mc.cores == 0L) {
  y <- lapply(X, FUN=...)
  } else {
  y <- mclapply(X, FUN=..., mc.cores=mc.cores)
}

It is clear that not everyone is aware that mc.cores specifies additional R process. For instance, it is not uncommon to see mc.cores = detectCores() where the developer probably intended mc.cores = detectCores() - 1. (PS. It is not really a good thing to use detectCores() this way, cf. the help).

Wish / Suggestion

Add support for mc.cores = 0 by mclapply() and friends in the parallel package. Specifically:

• Update mclapply() and friends to allow for mc.cores = 0, which should fall back to lapply() or similarly. Actually, mc.cores = 1 could do the same thing.
• Clarify in help pages that mc.cores = 0 is a properly fine setting.
• Consistently handle when mc.cores is a missing value.
• Update mclapply() on Windows to have argument mc.cores = 0 and not mc.cores = 1 as done currently.

Details

The current implementation of parallel::mclapply() already falls back to using base::lapply() whenever is called by a multicore child process and recursive multicore processing is not explicitly enabled;

> parallel::mclapply
function (X, FUN, ..., mc.preschedule = TRUE, mc.set.seed = TRUE,
    mc.silent = FALSE, mc.cores = getOption("mc.cores", 2L),
    mc.cleanup = TRUE, mc.allow.recursive = TRUE)
{
    cores <- as.integer(mc.cores)
    if (is.na(cores) || cores < 1L)
        stop("'mc.cores' must be >= 1")
    .check_ncores(cores)
    if (isChild() && !isTRUE(mc.allow.recursive))
        return(lapply(X = X, FUN = FUN, ...))
[...]

Thus, it would take very little to extend it to also support mc.cores = 0, e.g.

    if (is.na(cores) || cores < 0L)
        stop("'mc.cores' must be >= 0")
    if (mc.cores == 0 || (isChild() && !isTRUE(mc.allow.recursive)))
        return(lapply(X = X, FUN = FUN, ...))
[...]

We may even want to use if (mc.cores <= 1 || ...) as suggested above.

UPDATE 2016-02-04: As @ilarischeinin points in his comment below, with mclapply(..., mc.preschedule=TRUE) (the default), a bit further down in the code it actually already says:

    ## mc.preschedule = TRUE from here on.
    if (length(X) < cores) cores <- length(X)
    if (cores < 2L) return(lapply(X = X, FUN = FUN, ...))

Thus, it's clear that here the developer has had similar thoughts.

Continuing, On R for Windows, which does not support multicore processing / forking of processes, mclapply() falls back to calling lapply();

> parallel::mclapply
function (X, FUN, ..., mc.preschedule = TRUE, mc.set.seed = TRUE,
    mc.silent = FALSE, mc.cores = 1L, mc.cleanup = TRUE,
mc.allow.recursive = TRUE)
{
    cores <- as.integer(mc.cores)
    if (cores < 1L)
        stop("'mc.cores' must be >= 1")
    if (cores > 1L)
        stop("'mc.cores' > 1 is not supported on Windows")
    lapply(X, FUN, ...)
}

It would not be hard to update this one accordingly, i.e.

    if (is.na(cores) || cores < 0L)
        stop("'mc.cores' must be >= 0")
    if (cores > 0L)
        stop("'mc.cores' > 0 is not supported on Windows")
    lapply(X, FUN, ...)

Interestingly, looking at parallel::pvec(), we can see that the developer also thinks it is unnecessary to fork off a process if mc.cores = 1 (see also paragraph on mclapply(..., mc.preschedule=TRUE) above);

> pvec
function (v, FUN, ..., mc.set.seed = TRUE, mc.silent = FALSE,
    mc.cores = getOption("mc.cores", 2L), mc.cleanup = TRUE)
{
    if (!is.vector(v))
        stop("'v' must be a vector")
    cores <- as.integer(mc.cores)
    if (cores < 1L)
        stop("'mc.cores' must be >= 1")
    if (cores == 1L)
        return(FUN(v, ...))
[...]

Thus, also here it is easy to update to support mc.cores = 0.

[ilarischeinin]

I agree the documentation and behavior is confusing.

When run with mc.preschedule=TRUE (default) and mc.cores=1L, mclapply() will actually not fork a separate process:

Line 117 in mclapply():

    if (cores < 2L) return(lapply(X = X, FUN = FUN, ...))

i.e. same behavior you noted for pvec(). However, I guess technically this doesn't go against the documentation, since it says:

a integer giving the maximum allowed number of additional R processes allowed to be run in parallel to the current R process.

So, when mc.cores=1L, allowed additional processes are either 0 (as with mclapply(mc.preschedule=TRUE) or 1 (as with mclapply(mc.preschedule=FALSE).

But like I said, that is confusing and also inconsistent. It makes no sense to fork one additional process and have the main process simply wait.

The idea of the main process simply waiting I guess is the rational behind using mc.cores=detectCores() instead of - 1L. The point isn't so much to define how many simultaneous processes are allowed, but how many processes are allowed to compute at the same time. (Hence the two different behaviors of mclapply(mc.cores=1L) of one or two simultaneous processes, but only one actually doing anything.)

[HenrikBengtsson]

Thanks for the comments @ilarischeinin and for spotting the mcapply(mc.preschedule=TRUE) case; I've updated my top comment accordingly. To me this indicates that the developer(s) had similar ideas but at some point we ended up with a requirement of mc.cores >= 1.

I've also fixed a typo in my code suggestion: It should be if (mc.cores == 0 || (isChild() && !isTRUE(mc.allow.recursive))) and nothing else.

About the number of active versus computing processes: Yes, it's true that the main process shouldn't consume that much of the CPU since it is "just" spawning and polling for results. Having said this, I can imagine that there are obnoxiously strict compute clusters that would kick you out if you ran one process more than you requested (regardless of it's CPU usage). Moreover, this is holds for mclapply(). If you use for instance mcparallel(), there is nothing preventing you from running for computations in the main process as well. This can certainly becomes a reality with futures, where a main loop spawns of background processes and collects and post-processes results as they come back in while still polling for results of non-finished processes. This is why I think it is very important that the definition of mc.cores is very transparent and clear.