OpenMP not detected on Mac (Help from Mac users kindly needed!) #63

Asquidy · 2020-11-27T23:47:00Z

Hi, I'm trying to use fixest with multiple threads but it doesn't work. See the code below. Note that I do have OpenMP installed and it works on other R packages such as data.table. Any thoughts on why this may happen?


> library(fixest)
> fenegbin(Euros ~ log(dist_km)|Origin+Destination+Product+Year, trade, nthreads = 2)
Negative Binomial ML estimation, Dep. Var.: Euros
Observations: 38,325 
Fixed-effects: Origin: 15,  Destination: 15,  Product: 20,  Year: 10
Standard-errors: Clustered (Origin) 
             Estimate Std. Error z value  Pr(>|z|)    
log(dist_km)  -1.7108   0.166347 -10.284 < 2.2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Over-dispersion parameter: theta = 0.5487878  
Log-likelihood:  -646,586.98   Adj. Pseudo-R2: 0.03464 
           BIC: 1,293,796.64     Squared Cor.: 0.43761 
Warning message:
In fixest_env(fml = fml, data = data, family = famil...:
 OpenMP not detected: cannot use 2 threads, single-threaded mode instead.

The text was updated successfully, but these errors were encountered:

lrberge · 2020-11-30T13:17:57Z

Hi, that looks very platform specific. Unfortunately, I have no access to MAC so it'll be difficult to debug.

I've googled around and it may be a flag problem. The flag _OPENMP isn't there on your install.

Does the package fst have multi-threading enabled?

Related issues:
fstpackage/fst#242
Rdatatable/data.table#4417
Rdatatable/data.table#4410

Asquidy · 2020-11-30T13:54:26Z

Thanks, fst doesn't find OpenMP either. Will let you know if I find a solution.

lrberge · 2021-03-01T13:30:27Z

Hi, did you manage to find a solution?

Asquidy · 2021-03-01T14:01:59Z

Unfortunately not

…

On Mon, Mar 1, 2021 at 8:30 AM Laurent Bergé ***@***.***> wrote: Hi, did you manage to find a solution? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#63 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHCXYBDUQERLZ6YGE3DHQLTBOJILANCNFSM4UFMXKTQ> .

lrberge · 2021-03-01T14:05:39Z

OK, thanks. I'm changing the title (don't know if it'll help though :-)).

grantmcdermott · 2021-03-02T23:13:43Z

I too am not on Mac. But I recall that a related issue came up on the data.table GH repo. TL;DR is that OpenMP isn't natively supported by the MacOS toolchain used by R 4.0.0+. People are aware of it (workarounds have been provided) and from reading the thread I think the latest version of R might solve it?

See here and links therein: Rdatatable/data.table#4348

Asquidy · 2021-03-03T17:36:23Z

@grantmcdermott thanks for this link. Weirdly, data.table works just fine for me when I change the Makevars file, but fixest doesn't.

lrberge · 2021-03-04T17:12:44Z

I think the guys from data.table have applied patches to make it work.

But I don't know enough on these topics really. Further I don't think it's worth the extra legwork if next versions of R will fix it (which is really great news and makes sense given the prominence of OpenMP).

pbaylis · 2021-03-04T18:02:14Z

I just got OpenMP working on my Macbook Pro for data.table (following their instructions), but not fst or fixest. According to this thread there is some way to do it in fst but I gave up after five minutes. The same thread also indicates that the data.table people got it working using patches.

But yeah, if R is going to fix this anyway soon, doesn't seem worth the extra effort.

lrberge · 2021-03-05T10:59:26Z

And here's the ad hoc DT PR Rdatatable/data.table#3984

adamaltmejd · 2021-03-24T20:25:44Z

Think I got it working with the following Makevars file. For it to work you will first need to run (if you haven't already). Only tested with R installed with Homebrew, not the 4.1 binary.

xcode-select --install
brew install llvm gcc gettext libomp

Install homebrew here: https://brew.sh

~/.R/Makevars

XC_LOC:=$(shell xcrun --show-sdk-path)
LLVM_LOC:=$(shell brew --prefix llvm)
GCC_LOC:=$(shell brew --prefix gcc)
GETTEXT_LOC:=$(shell brew --prefix gettext)
OMP_LOC:=$(shell brew --prefix libomp)

CC=$(LLVM_LOC)/bin/clang
CXX=$(LLVM_LOC)/bin/clang++
CXX11=$(LLVM_LOC)/bin/clang++
CXX14=$(LLVM_LOC)/bin/clang++
CXX17=$(LLVM_LOC)/bin/clang++
CXX1X=$(LLVM_LOC)/bin/clang++

CFLAGS=-g -O2 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O2 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L"$(LLVM_LOC)/lib" -L"$(GETTEXT_LOC)/lib" -Wl,-rpath,$(LLVM_LOC)/lib --sysroot="$(XC_LOC)" -lomp
CPPFLAGS=-I"$(GETTEXT_LOC)/include" -I"$(LLVM_LOC)/include" -I"$(OMP_LOC)/include" -isysroot "$(XC_LOC)" -Xclang -fopenmp

FC=$(GCC_LOC)/bin/gfortran
F77=$(GCC_LOC)/bin/gfortran
FLIBS=-L$(GCC_LOC)/lib/gcc/10/ -lm

> data(iris)
> library(fixest)
> summary(feols(Sepal.Length ~ Petal.Length | Species[Sepal.Width], iris))
OLS estimation, Dep. Var.: Sepal.Length
Observations: 150
Fixed-effects: Species: 3
Varying slopes: Sepal.Width (Species: 3)
Standard-errors: Clustered (Species)
             Estimate Std. Error t value Pr(>|t|)
Petal.Length 0.822045    0.10061  8.1706 0.014651 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
RMSE: 0.298903     Adj. R2: 0.863326
                 Within R2: 0.518738
>

Asquidy · 2021-03-25T15:26:53Z

@adamaltmejd Thanks. I think the problem is that it isn't able to do multi-thread operations, not that it doesn't run at all. Can you try the following?

 summary(feols(Sepal.Length ~ Petal.Length | Species[Sepal.Width], iris), nthreads = 4)

adamaltmejd · 2021-03-25T15:29:40Z

Oops! Forgot to add that part to the post:

> library(fixest)
> data(iris)
>  summary(feols(Sepal.Length ~ Petal.Length | Species[Sepal.Width], iris), nthreads = 4)
OLS estimation, Dep. Var.: Sepal.Length
Observations: 150
Fixed-effects: Species: 3
Varying slopes: Sepal.Width (Species: 3)
Standard-errors: Clustered (Species)
             Estimate Std. Error t value Pr(>|t|)
Petal.Length 0.822045    0.10061  8.1706 0.014651 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
RMSE: 0.298903     Adj. R2: 0.863326
                 Within R2: 0.518738

lrberge · 2021-03-25T15:41:57Z

@adamaltmejd you rock!

@Asquidy: could you check if it works for you? If so, I'll pin an issue describing the way to go for Mac users.

Asquidy · 2021-03-25T15:46:04Z

It still doesn't work. FYI, I'm using Big Sur and R 4.0.3.

lrberge · 2021-03-25T15:51:16Z

Too bad :-(, at least we know it can work w/t extra changes in the package for some Mac users.

adamaltmejd · 2021-03-25T17:00:04Z

Strange. Are you using R installed with Homebrew? And github fixest built from source? For example installed by running remotes::install_github("lrberge/fixest")? Did you get openMP to work with data.table and fst? Latest Homebrew R version is 4.0.4 so try updating.

Asquidy · 2021-03-25T18:01:39Z

I've tried many of these things (and yes data.table works). Unfortunately, installing fixest from github doesn't work. I can't justify the time it takes to try to debug this even more.

…

On Thu, Mar 25, 2021 at 1:00 PM Adam Altmejd ***@***.***> wrote: Strange. Are you using R installed with Homebrew? And github fixest built from source? For example installed by running remotes::install_github("lrberge/fixest")? Did you get openMP to work with data.table and fst? Latest Homebrew R version is 4.0.4 so try updating. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHCXYAMHPKTBH3RG3BX7ULTFNT2VANCNFSM4UFMXKTQ> .

adamaltmejd · 2021-03-25T18:42:24Z

Yeah probably the use case where one wants to run code on own computer instead of server and really need multicore is quite rare.

matthewgson · 2021-05-22T15:50:27Z

Mac user here. Since CRAN on Mac uses Apple Xcode command line tools after R 4.0, which does not support OpenMP, you might want to compile the package from your own computer, not downloading the binary from CRAN. This can be done by adding type="source" in install.packages(). Seems fixest is running smoothly on my machine (Big Sur, R 4.0.3). I believe this will fix OpenMP issue with your fst as well (make sure CXX11 is defined on your .Makevars). Hope this helps!

lrberge · 2021-06-01T06:54:55Z

Thanks @matthewgson! Since the problem does not seem general but rather quite specific to the OP, I'm closing. Thanks all!

adamaltmejd · 2021-06-01T08:54:49Z

I can btw confirm that the Makefile posted above also works with the 4.1 big sur binary for arm M1 macs at least for me. Just make sure to build fixest from source.

grantmcdermott · 2022-12-09T22:13:03Z

@adamaltmejd's solution above has generally worked really well for me.

Just adding some info since I ran brew update today and got caught out by some broken symlinks following the OpenMP and LLVM updates. To fix, I needed to:

Explicitly reinstall z3 brew reinstall z3 (Adam, maybe worth adding z3 to your initial brew install call?)

Update my compile flags (the brew output tells you to do this, but worth pointing out nonetheless since I think Homebrew's default location changed).

export LDFLAGS="-L/usr/local/opt/llvm/lib"
export CPPFLAGS="-I/usr/local/opt/llvm/include"
export LDFLAGS="-L/usr/local/opt/libomp/lib"
export CPPFLAGS="-I/usr/local/opt/libomp/include"

Asquidy · 2022-12-10T02:08:56Z

Can confirm that this now works!

adamaltmejd · 2022-12-11T13:06:52Z

Couple of things:

It's unfortunately a recurring problem with compiled R packages. Whenever brew updates llvm and you try to install new R packages some of the dependencies will point to old libraries which will lead to compilation errors. Usually really annoying to identify which packages need to be rebuilt so i tend to just rebuild my whole R library. Edit: the z3 issue happened to me too even though i had not installed z3 by myself. Its a llvm dependency so it installs automatically with llvm. Very strange that it doesn't get reconfigured by brew automatically.

The code above uses brew --prefix llvm because the brew directory can be in different places. On a clean install M1 mac its /opt/homebrew/opt/llvm.

I would be careful about exporting those compile flags outside of R, not sure how that will work with other brew-compiled software that usually uses the built-in compilation toolchain.

RoyalTS · 2024-04-02T21:04:12Z

Does the above recipe still work for people or has it gone out of date over the last 18 months?

grantmcdermott · 2024-04-09T19:17:36Z

Still works for me, although I haven't tested on one of the newer Macs with the "M" silicon-based chips.

adamaltmejd · 2024-04-09T20:11:52Z

Same on an M1 with Sonoma.

caleb-kwon · 2024-07-09T05:38:17Z

Confirming that this works on the M3 Max (Sonoma 14.5). Thanks @adamaltmejd!

Think I got it working with the following Makevars file. For it to work you will first need to run (if you haven't already). Only tested with R installed with Homebrew, not the 4.1 binary.

xcode-select --install
brew install llvm gcc gettext libomp

Install homebrew here: https://brew.sh

~/.R/Makevars

XC_LOC:=$(shell xcrun --show-sdk-path)
LLVM_LOC:=$(shell brew --prefix llvm)
GCC_LOC:=$(shell brew --prefix gcc)
GETTEXT_LOC:=$(shell brew --prefix gettext)
OMP_LOC:=$(shell brew --prefix libomp)

CC=$(LLVM_LOC)/bin/clang
CXX=$(LLVM_LOC)/bin/clang++
CXX11=$(LLVM_LOC)/bin/clang++
CXX14=$(LLVM_LOC)/bin/clang++
CXX17=$(LLVM_LOC)/bin/clang++
CXX1X=$(LLVM_LOC)/bin/clang++

CFLAGS=-g -O2 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O2 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L"$(LLVM_LOC)/lib" -L"$(GETTEXT_LOC)/lib" -Wl,-rpath,$(LLVM_LOC)/lib --sysroot="$(XC_LOC)" -lomp
CPPFLAGS=-I"$(GETTEXT_LOC)/include" -I"$(LLVM_LOC)/include" -I"$(OMP_LOC)/include" -isysroot "$(XC_LOC)" -Xclang -fopenmp

FC=$(GCC_LOC)/bin/gfortran
F77=$(GCC_LOC)/bin/gfortran
FLIBS=-L$(GCC_LOC)/lib/gcc/10/ -lm

> data(iris)
> library(fixest)
> summary(feols(Sepal.Length ~ Petal.Length | Species[Sepal.Width], iris))
OLS estimation, Dep. Var.: Sepal.Length
Observations: 150
Fixed-effects: Species: 3
Varying slopes: Sepal.Width (Species: 3)
Standard-errors: Clustered (Species)
             Estimate Std. Error t value Pr(>|t|)
Petal.Length 0.822045    0.10061  8.1706 0.014651 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
RMSE: 0.298903     Adj. R2: 0.863326
                 Within R2: 0.518738
>

lrberge changed the title ~~OpenMP not detected on Mac~~ OpenMP not detected on Mac (Help from Mac users kindly needed!) Mar 1, 2021

lrberge mentioned this issue Mar 2, 2021

Benchmark against Python's pyhdfe FixedEffects/FixedEffectModels.jl#91

Closed

lrberge closed this as completed Jun 1, 2021

lrberge mentioned this issue Dec 2, 2021

clang: error: unsupported option '-fopenmp' #241

Closed

grantmcdermott mentioned this issue Sep 4, 2022

Topics for generic "misc" tips stata2r/stata2r.github.io#26

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenMP not detected on Mac (Help from Mac users kindly needed!) #63

OpenMP not detected on Mac (Help from Mac users kindly needed!) #63

Asquidy commented Nov 27, 2020

lrberge commented Nov 30, 2020

Asquidy commented Nov 30, 2020

lrberge commented Mar 1, 2021

Asquidy commented Mar 1, 2021 via email

lrberge commented Mar 1, 2021

grantmcdermott commented Mar 2, 2021 •

edited

Loading

Asquidy commented Mar 3, 2021

lrberge commented Mar 4, 2021

pbaylis commented Mar 4, 2021 •

edited

Loading

lrberge commented Mar 5, 2021

adamaltmejd commented Mar 24, 2021

Asquidy commented Mar 25, 2021

adamaltmejd commented Mar 25, 2021

lrberge commented Mar 25, 2021

Asquidy commented Mar 25, 2021

lrberge commented Mar 25, 2021

adamaltmejd commented Mar 25, 2021

Asquidy commented Mar 25, 2021 via email

adamaltmejd commented Mar 25, 2021

matthewgson commented May 22, 2021

lrberge commented Jun 1, 2021

adamaltmejd commented Jun 1, 2021

grantmcdermott commented Dec 9, 2022

Asquidy commented Dec 10, 2022

adamaltmejd commented Dec 11, 2022 •

edited

Loading

RoyalTS commented Apr 2, 2024

grantmcdermott commented Apr 9, 2024

adamaltmejd commented Apr 9, 2024

caleb-kwon commented Jul 9, 2024

OpenMP not detected on Mac (Help from Mac users kindly needed!) #63

OpenMP not detected on Mac (Help from Mac users kindly needed!) #63

Comments

Asquidy commented Nov 27, 2020

lrberge commented Nov 30, 2020

Asquidy commented Nov 30, 2020

lrberge commented Mar 1, 2021

Asquidy commented Mar 1, 2021 via email

lrberge commented Mar 1, 2021

grantmcdermott commented Mar 2, 2021 • edited Loading

Asquidy commented Mar 3, 2021

lrberge commented Mar 4, 2021

pbaylis commented Mar 4, 2021 • edited Loading

lrberge commented Mar 5, 2021

adamaltmejd commented Mar 24, 2021

Asquidy commented Mar 25, 2021

adamaltmejd commented Mar 25, 2021

lrberge commented Mar 25, 2021

Asquidy commented Mar 25, 2021

lrberge commented Mar 25, 2021

adamaltmejd commented Mar 25, 2021

Asquidy commented Mar 25, 2021 via email

adamaltmejd commented Mar 25, 2021

matthewgson commented May 22, 2021

lrberge commented Jun 1, 2021

adamaltmejd commented Jun 1, 2021

grantmcdermott commented Dec 9, 2022

Asquidy commented Dec 10, 2022

adamaltmejd commented Dec 11, 2022 • edited Loading

RoyalTS commented Apr 2, 2024

grantmcdermott commented Apr 9, 2024

adamaltmejd commented Apr 9, 2024

caleb-kwon commented Jul 9, 2024

grantmcdermott commented Mar 2, 2021 •

edited

Loading

pbaylis commented Mar 4, 2021 •

edited

Loading

adamaltmejd commented Dec 11, 2022 •

edited

Loading