Github actions compile troubles

[nbenn]

I assume this is closely related to #226 and at least in part to #238 but I decided to open a separate issue because #226 is marked as resolved (and focuses more on Conda) and #238 is more concerned with a follow up problem. The focus here is compiling with github actions using the check-full.yaml config example of r-lib/actions. The macOS/devel job, which attempts installing all dependencies from source fails because fst cannot be installed

 unable to load shared object '/Users/runner/runners/2.169.1/work/_temp/Library/00LOCK-fst/00new/fst/libs/fst.so':
  dlopen(/Users/runner/runners/2.169.1/work/_temp/Library/00LOCK-fst/00new/fst/libs/fst.so, 6): Symbol not found: _XXH32
  Referenced from: /Users/runner/runners/2.169.1/work/_temp/Library/00LOCK-fst/00new/fst/libs/fst.so
* removing ‘/Users/runner/runners/2.169.1/work/_temp/Library/fst’
  Expected in: flat namespace
 in /Users/runner/runners/2.169.1/work/_temp/Library/00LOCK-fst/00new/fst/libs/fst.so

I did try to add a step for setting up the macOS job as recommended here using

- name: Install macOS-devel system dependencies
   if: matrix.config.os == 'macOS-latest' && matrix.config.r == 'devel'
   run: |
     echo "::add-path::/usr/local/opt/llvm/bin"
     echo "::set-env name=LDFLAGS::-L/usr/local/opt/llvm/lib"
     echo "::set-env name=CPPFLAGS::-I/usr/local/opt/llvm/include"

At one time, I also had brew install llvm in there, which told me

##[warning]Warning: llvm 9.0.1 is already installed and up-to-date
To reinstall 9.0.1, run `brew reinstall llvm`

so I guess llvm is available?

Looking at the compiler output, R CMD Install does not seem to be picking up my env vars

clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c flex_store.cpp -o flex_store.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fst_blockrunner_char.cpp -o fst_blockrunner_char.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fst_compress.cpp -o fst_compress.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fst_error.cpp -o fst_error.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fst_table.cpp -o fst_table.o
...
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fstcore/blockstreamer/blockstreamer_v2.cpp -o fstcore/blockstreamer/blockstreamer_v2.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fstcore/integer64/integer64_v11.cpp -o fstcore/integer64/integer64_v11.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I. -Ifstcore -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress -Ifstcore/ZSTD/compress -I'/Users/runner/runners/2.169.1/work/_temp/Library/Rcpp/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c fstcore/byteblock/byteblock_v13.cpp -o fstcore/byteblock/byteblock_v13.o
ar rcs libFRAME.a fstcore/interface/openmphelper.o fstcore/interface/fststore.o fstcore/logical/logical_v10.o fstcore/integer/integer_v8.o fstcore/byte/byte_v12.o fstcore/double/double_v9.o fstcore/character/character_v6.o fstcore/factor/factor_v7.o fstcore/blockstreamer/blockstreamer_v2.o fstcore/integer64/integer64_v11.o fstcore/byteblock/byteblock_v13.o
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o fst.so RcppExports.o flex_store.o fst_blockrunner_char.o fst_compress.o fst_error.o fst_table.o init.o openmp.o -L. -lFRAME -lCOMPRESSION -lLZ4 -lZSTD -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation

I also tried adding

      echo "::set-env name=CC::/usr/local/opt/llvm/bin/clang
      echo "::set-env name=CXX::/usr/local/opt/llvm/bin/clang++

to my CI config, but this did not resolve my problem. I'm new to github actions, but according to the docs, this is the way to set env vars. It's entirely possible that the r-lib/actions R setup creates a Makevars file or that I do not understand how to define gh action steps or something else entirely. I haven't had the time to investigate further. Maybe someone else has ideas on how to resolve this?

[nbenn]

@MarcusKlik also, feel free to close this again if you think this is more of an issue of how to properly set up gh actions. I mainly raised the issue here b/c I was hoping that something might be improved on the fst side of things, to make to build experience more robust.

[MarcusKlik]

Hi @nbenn, thanks for reporting, I would definitely like to add github CI to the fst repository, so it's very relevant!

To test the check_full.yaml example, I've created a new repository and enabled github actions. The OSX build produces the error that you report above.

The errors are consitent with a linking problem where components build with libc++ are mixed with components build with libstdc++ (see for example this thread), but I haven't be able to fix that yet.

Any ideas or suggestions are very welcome!

(the test repository contains the new fstcore package; from the next release onward, almost all c++ code will be contained in that package which will act as a 'base' package for fst)

thanks!

[MarcusKlik]

All actions run correctly now