R: parallel computing unstable with Brew version of R

[ywwry66]

Here is a simple script for LDA (linear discriminant analysis) simulation in R, and my goal is to run 20 replicates in parallel.

library(MASS)
library(parallel)
library(doParallel)
num_cores <- detectCores()
registerDoParallel(cores = num_cores)

data <- list()
reps <- 20

## generate data
for (i in 1:reps) {
    data[[i]] <-
        list(x = rbind(mvrnorm(100, rep(0, 50), diag(50)),
                       mvrnorm(100, rep(1, 50), diag(50))),
             xnew = rbind(mvrnorm(100, rep(0, 50), diag(50)),
                          mvrnorm(100, rep(1, 50), diag(50))),
             y = rbind(rep(0, 100), rep(1, 100)),
             ynew = rbind(rep(0, 100), rep(1, 100)))
}

## function producing prediction error for LDA
lda_prediction_error <- function(data) {
    fit_lda <- lda(data$x, as.factor(data$y))
    ypred <- predict(fit_lda, data$xnew)$class
    pe_lda <- mean(data$ynew != ypred)
    return(pe_lda)
}

foreach(i = 1:reps, .verbose = TRUE) %dopar% lda_prediction_error(data[[i]])

mclapply(data, lda_prediction_error, mc.cores = num_cores)

However, R hangs forever with either foreach or mclapply. In Activity Monitor, I can see 4 R processes are created but all use 0% cpu.

If I use sequential computing (use %do% instead of %dopar%, or use lapply instead of mclapply), the code works well.

If I generate data with rnorm instead of mvrnorm, the code also works well.

Then I tested the code under different scenarios:

1. Build R from source with Homebrew formula: brew install -s r
2. Install official R.app: brew cask install r
3. Build R from source by myself: download R from https://cran.r-project.org/src/base/R-4/R-4.0.2.tar.gz, then ./configure --without-x and make
4. R in WSL2 installed with sudo apt install r-base

The code above hangs in scenario 1, but works in 2, 3, 4, 5. From these experiments I think it is the Homebrew r formula that causes R to hang. I don't have the insight to find the culprit in the formula, so I am asking for help here. I am also interested to see if this issue is reproducible.

[ywwry66]

I just rebuilt R with Homebrew r formula default args:

1. Download R from https://cran.r-project.org/src/base-prerelease/R-devel_2020-09-17_r79226.tar.gz. I use the daily snapshot because the current stable version won't build with CLT 12
2. Build with ./configure --enable-memory-profiling --without-cairo --without-tcltk --without-x --with-aqua --with-lapack --enable-R-shlib SED=/usr/bin/sed --disable-java --with-blas=-L/usr/local/Cellar/openblas/0.3.10_1/lib-lopenblas && make
   The R code above actually works in this version of R. So I am confused what could possibly go wrong with the Homebrew r formula.

[SMillerDev]

Maybe they fixed it in the newer release, maybe it's one of the other modifications Homebrew is doing. If you really want to get to the bottom of this I'd recommend contacting the R developers. Homebrew just packages software, we're generally not much help figuring out the internal working of software.

[ywwry66]

Thanks for the suggestion. I just hacked the homebrew r formula a bit to change the url to the latest dev r version (haven't used ruby before but it seems not hard once I get my hands on that), and install with brew install -s r. The parallel computing code works perfectly with this version.

So, at this point, with all the experiments, I can't really tell whether the problem lies in homebrew r formula or upstream. I think I will mark this as solved for now and wait until the new version of R comes out.

[jonchang]

I'm going to take a look at it sometime this week. R on Homebrew has a lot of issues that unfortunately few people have had time to look into. For the time being using the cask version of R is probably the way to go if you need this specific functionality. As a bonus, you'll be able to use pre-compiled binary packages from CRAN.

[ywwry66]

Thanks. I did not see your answer when I replied to SMillerDev. And as I said, I did more testing with the daily snapshot R version in conjunction with Homebrew r formula. The result that the parallel computing code worked is actually surprising to me. Hopefully this piece of info is also useful to you.

Based on the number of downloads from analytics, people (homebrew users, strictly speaking) still prefer r cellar than r cask, and I am one of them, despite of the fact that every package needs to be compiled locally. It is somewhat satisfying to see the packages compile successfully haha.

[ywwry66]

Issue solved in R 4.0.3.

Edit: see Homebrew/homebrew-core#75506