Failure to build with zstd with 15.0.3 or ToT on OpenBSD

[brad0]

Ever since #58205 went in it has broken the build on OpenBSD. If I use 15.0.2 it's fine. 15.0.3 or ToT and it's broken. If I use -DLLVM_ENABLE_ZSTD=OFF it skips the test and completes the CMake configuration.

CMake Error at lib/Support/CMakeLists.txt:312 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.

CMake Error at lib/Support/CMakeLists.txt:315 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.

CMake Error at lib/Support/CMakeLists.txt:317 (get_library_name):
  get_library_name Function invoked with incorrect arguments for function
  named: get_library_name

[mgorny]

Could you give us more information how zstd is built on OpenBSD? Which build system is used, which files are installed?

[brad0]

I'm not familiar with zstd. But looking at our port it appears to be using a GNU Makefile.

fubar$ pkg_info -L zstd
Information for inst:zstd-1.5.2

Files:
/usr/local/bin/unzstd
/usr/local/bin/zstd
/usr/local/bin/zstdcat
/usr/local/bin/zstdgrep
/usr/local/bin/zstdless
/usr/local/bin/zstdmt
/usr/local/include/zdict.h
/usr/local/include/zstd.h
/usr/local/include/zstd_errors.h
/usr/local/lib/libzstd.a
/usr/local/lib/libzstd.so.6.1
/usr/local/lib/pkgconfig/libzstd.pc
/usr/local/man/man1/unzstd.1
/usr/local/man/man1/zstd.1
/usr/local/man/man1/zstdcat.1
/usr/local/man/man1/zstdgrep.1
/usr/local/man/man1/zstdless.1

Ah, I see in the build/ dir there is CMake and Meson support for zstd.

[mgorny]

That looks pretty normal.

How about:

grep zstd CMakeCache.txt

[DimitryAndric]

I reported the same problem in https://discourse.llvm.org/t/llvm-15-0-3-tagged/66031/5?u=dimitryandric, and I simply rolled back b484027 to work around it.

[DimitryAndric]

I reported the same problem in https://discourse.llvm.org/t/llvm-15-0-3-tagged/66031/5?u=dimitryandric

That was on macOS, btw. Where zstd is installed by Homebrew, into /opt/homebrew. You have to force it to link statically in that case, otherwise end users can't run it without also installing Homebrew and the zstd package.

[brad0]

I reported the same problem in https://discourse.llvm.org/t/llvm-15-0-3-tagged/66031/5?u=dimitryandric, and I simply rolled back b484027 to work around it.

I saw that, but it would be nice to try and come up with a resolution to the issue. I don't mind using the command line param to disable zstd for the time being.

Does macOS not have something like rpath?

[DimitryAndric]

Well, macOS does have something like rpath, but in the spirit of "think different", it is rather different than how BSDs and Linux implement it. :) You would still have copy libzstd.1.dylib into your final installation directory, and I'm unsure how you do that in CMake land. I'm not going to touch it, in any case.

Btw it seems to work OK with main (as of fcd5098 from just now), but still errors out with release/15.x as of 154e88a, if you specify -D LLVM_USE_STATIC_ZSTD=ON:

command: cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/Users/dim/llvm/llvmorg-15.0.5-0-g154e88af7ec9-1 -D CMAKE_C_COMPILER=clang -D CMAKE_CXX_COMPILER=clang++ -D LLVM_ENABLE_PROJECTS=clang -D LLVM_USE_STATIC_ZSTD=ON /Users/dim/src/llvm/release-15/llvm
-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
...
-- Looking for __x86_64__
-- Looking for __x86_64__ - not found
CMake Error at lib/Support/CMakeLists.txt:309 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.


CMake Error at lib/Support/CMakeLists.txt:312 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.


CMake Error at lib/Support/CMakeLists.txt:314 (get_library_name):
  get_library_name Function invoked with incorrect arguments for function
  named: get_library_name


-- Found Git: /usr/bin/git (found version "2.37.1 (Apple Git-137.1)")
...
-- Configuring incomplete, errors occurred!
See also "/Users/dim/obj/llvmorg-15.0.5-0-g154e88af7ec9-1/CMakeFiles/CMakeOutput.log".
See also "/Users/dim/obj/llvmorg-15.0.5-0-g154e88af7ec9-1/CMakeFiles/CMakeError.log".

It does show zstd in CMakeCache.txt:

% grep -i zstd CMakeCache.txt
//Use zstd for compression/decompression if available. Can be ON,
LLVM_ENABLE_ZSTD:STRING=ON
//Use static version of zstd. Can be TRUE, FALSE
LLVM_USE_STATIC_ZSTD:BOOL=ON
zstd_INCLUDE_DIR:PATH=/opt/homebrew/include
zstd_LIBRARY:FILEPATH=/opt/homebrew/lib/libzstd.dylib
//ADVANCED property for variable: zstd_INCLUDE_DIR
zstd_INCLUDE_DIR-ADVANCED:INTERNAL=1
//ADVANCED property for variable: zstd_LIBRARY
zstd_LIBRARY-ADVANCED:INTERNAL=1

At first sight it looks like @mgorny fixed this in main with 56f94ed ?

[brad0]

Btw it seems to work OK with main (as of fcd5098 from just now), but still errors out with release/15.x as of 154e88a, if you specify -D LLVM_USE_STATIC_ZSTD=ON:

command: cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/Users/dim/llvm/llvmorg-15.0.5-0-g154e88af7ec9-1 -D CMAKE_C_COMPILER=clang -D CMAKE_CXX_COMPILER=clang++ -D LLVM_ENABLE_PROJECTS=clang -D LLVM_USE_STATIC_ZSTD=ON /Users/dim/src/llvm/release-15/llvm
-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
...
-- Looking for __x86_64__
-- Looking for __x86_64__ - not found
CMake Error at lib/Support/CMakeLists.txt:309 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.


CMake Error at lib/Support/CMakeLists.txt:312 (get_property):
  get_property could not find TARGET zstd::libzstd_static.  Perhaps it has
  not yet been created.


CMake Error at lib/Support/CMakeLists.txt:314 (get_library_name):
  get_library_name Function invoked with incorrect arguments for function
  named: get_library_name


-- Found Git: /usr/bin/git (found version "2.37.1 (Apple Git-137.1)")
...
-- Configuring incomplete, errors occurred!
See also "/Users/dim/obj/llvmorg-15.0.5-0-g154e88af7ec9-1/CMakeFiles/CMakeOutput.log".
See also "/Users/dim/obj/llvmorg-15.0.5-0-g154e88af7ec9-1/CMakeFiles/CMakeError.log".

It does show zstd in CMakeCache.txt:

% grep -i zstd CMakeCache.txt
//Use zstd for compression/decompression if available. Can be ON,
LLVM_ENABLE_ZSTD:STRING=ON
//Use static version of zstd. Can be TRUE, FALSE
LLVM_USE_STATIC_ZSTD:BOOL=ON
zstd_INCLUDE_DIR:PATH=/opt/homebrew/include
zstd_LIBRARY:FILEPATH=/opt/homebrew/lib/libzstd.dylib
//ADVANCED property for variable: zstd_INCLUDE_DIR
zstd_INCLUDE_DIR-ADVANCED:INTERNAL=1
//ADVANCED property for variable: zstd_LIBRARY
zstd_LIBRARY-ADVANCED:INTERNAL=1

At first sight it looks like @mgorny fixed this in main with 56f94ed ?

I'm still seeing the same error even with main.

fubar$ grep -i zstd CMakeCache.txt
//Use zstd for compression/decompression if available. Can be ON,
LLVM_ENABLE_ZSTD:STRING=ON
//Use static version of zstd. Can be TRUE, FALSE
LLVM_USE_STATIC_ZSTD:BOOL=FALSE
zstd_INCLUDE_DIR:PATH=/usr/local/include
zstd_LIBRARY:FILEPATH=/usr/local/lib/libzstd.so.6.1
zstd_STATIC_LIBRARY:FILEPATH=/usr/local/lib/libzstd.a
//ADVANCED property for variable: zstd_INCLUDE_DIR
zstd_INCLUDE_DIR-ADVANCED:INTERNAL=1
//ADVANCED property for variable: zstd_LIBRARY
zstd_LIBRARY-ADVANCED:INTERNAL=1
//ADVANCED property for variable: zstd_STATIC_LIBRARY
zstd_STATIC_LIBRARY-ADVANCED:INTERNAL=1

[mgorny]

Ok, I think I know what the problem is, or at least part of the problem. For some reason, your CMake sets path to the library as /usr/local/lib/libzstd.so.6.1 whereas it just gives /usr/lib64/libzstd.so here. The SOVERSION causes the zstd_SHARED_LIBRARY_SUFFIX not to match. Any clue why it behaves like that?

CC @petrhosek.

[brad0]

Ok, I think I know what the problem is, or at least part of the problem. For some reason, your CMake sets path to the library as /usr/local/lib/libzstd.so.6.1 whereas it just gives /usr/lib64/libzstd.so here. The SOVERSION causes the zstd_SHARED_LIBRARY_SUFFIX not to match. Any clue why it behaves like that?

CC @petrhosek.

OpenBSD handles shared libs differently from other ELF OS's. There are no symlinks.