Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

does not build on macOS when using isolated install with colcon #1270

Closed
wjwwood opened this issue May 3, 2022 · 18 comments
Closed

does not build on macOS when using isolated install with colcon #1270

wjwwood opened this issue May 3, 2022 · 18 comments

Comments

@wjwwood
Copy link

wjwwood commented May 3, 2022

When building with just colcon build on macOS I get:

% colcon build
Starting >>> iceoryx_hoofs
Finished <<< iceoryx_hoofs [6.57s]
Starting >>> iceoryx_posh
--- stderr: iceoryx_posh
Cloning into 'src'...
HEAD is now at fededad one-char typo
Submodule 'deps/meta-cmake' (https://github.com/meta-toolkit/meta-cmake.git) registered for path 'deps/meta-cmake'
Cloning into '/Users/william/cyclone_ws/build/iceoryx_posh/dependencies/cpptoml/src/deps/meta-cmake'...
---
Finished <<< iceoryx_posh [22.9s]
Starting >>> iceoryx_binding_c
Finished <<< iceoryx_binding_c [4.79s]
Starting >>> cyclonedds
--- stderr: cyclonedds
dyld: Library not loaded: @rpath/libiceoryx_posh.dylib
  Referenced from: /Users/william/cyclone_ws/install/iceoryx_binding_c/lib/libiceoryx_binding_c.dylib
  Reason: image not found
/bin/sh: line 1: 44301 Abort trap: 6           ../../../bin/idlc -Wno-implicit-extensibility /Users/william/cyclone_ws/src/cyclonedds/src/tools/ddsperf/ddsperf_types.idl
make[2]: *** [src/tools/ddsperf/ddsperf_types.c] Error 134
make[1]: *** [src/tools/ddsperf/CMakeFiles/ddsperf_types_generate.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< cyclonedds [21.7s, exited with code 2]

Summary: 3 packages finished [56.5s]
  1 package failed: cyclonedds
  2 packages had stderr output: cyclonedds iceoryx_posh

I believe this is because of a naive assumption that libiceoryx_binding_c.dylib is installed to the same place as all of its dependencies, like libiceoryx_posh.dylib, which is not the case in an isolated install.

This failure is coming from trying to run idlc here:

add_custom_command(
OUTPUT ${_file_outputs}
COMMAND ${_idlc_executable}
ARGS ${_language} ${IDLC_ARGS} ${IDLC_INCLUDE_DIRS} ${_file}
DEPENDS ${_files} ${_depends})

And I guess it is linked against iceoryx.

There is some logic to try and handle this for starting the daemon when testing, but it is similarly flawed as it only extends the DYLD_LIBRARY_PATH with the place that iceoryx_binding_c is installed, but not it's dependencies:

# The run-time library paths aren't set in iox-roudi so we need to set
# (DY)LD_LIBRARY_PATH. CTest can't start a daemon, so we have to use
# a wrapper to start a process, for which bash makes the most sense
# when targetting just Linux and macOS. However, that means macOS
# will strip DYLD_LIBRARY_PATH from the environment and therefore we
# cannot rely on "set_test_library_paths".
find_library(ICEORYX_LIB iceoryx_binding_c)
get_filename_component(ICEORYX_LIB_PATH ${ICEORYX_LIB} DIRECTORY)
if(APPLE)
set(ICEORYX_ROUDI_LIBPATH "DYLD_LIBRARY_PATH=${ICEORYX_LIB_PATH}:$ENV{DYLD_LIBRARY_PATH}")
else()
set(ICEORYX_ROUDI_LIBPATH "LD_LIBRARY_PATH=${ICEORYX_LIB_PATH}:$ENV{LD_LIBRARY_PATH}")
endif()

I suppose it may fail there too.


Steps to reproduce:

  • mkdir -p cyclone_ws/src
  • cd cyclone_ws
  • Create a cyclone.repos file with these contents:
repositories:
  cyclonedds:
    type: git
    url: https://github.com/eclipse-cyclonedds/cyclonedds.git
    version: 3c81d2856aa088e48d2e9f2365c53c2591ee6fdb
  iceoryx:
    type: git
    url: https://github.com/eclipse-iceoryx/iceoryx
    version: e537097966771585daa373bb15618a5c4493e1dc
  • vcs import ./src < cyclone.repos
  • colcon build

Note: You can work around this by using colcon build --merge-install, but that is not a satisfactory solution for my situation unfortunately.


I guess this issue may be related, though it's hard to tell from the description which is pretty vague: #1198

@eboasson
Copy link
Contributor

eboasson commented May 5, 2022

Given that the only direct dependency of Cyclone is to libiceoryx_binding_c.dylib (hoofs and posh are listed in package.xml just to make CMake's find_package work), I'd say this is more likely an issue with a run-time library path in libiceoryx_binding_c.dylib.

Indeed, there is no LC_RPATH in the iceoryx library:

# otool -l install/iceoryx_binding_c/lib/libiceoryx_binding_c.dylib | grep -C4 -i rpath
    flags 0x0
Load command 4
          cmd LC_ID_DYLIB
      cmdsize 64
         name @rpath/libiceoryx_binding_c.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 5
--
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libiceoryx_posh.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libiceoryx_hoofs.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libiceoryx_platform.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 17

For reference:

# otool -l build/cyclonedds/lib/libddsc.dylib | grep -C4 -i rpath
    flags 0x0
Load command 4
          cmd LC_ID_DYLIB
      cmdsize 48
         name @rpath/libddsc.0.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.10.0
compatibility version 0.0.0
Load command 5
--
  version 0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libiceoryx_binding_c.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 13
--
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1311.100.3
compatibility version 1.0.0
Load command 15
          cmd LC_RPATH
      cmdsize 72
         path /Users/erik/C/qwe/cyclone_ws/install/iceoryx_binding_c/lib (offset 12)
Load command 16
      cmd LC_FUNCTION_STARTS

@dkroenke any idea why there is no rpath entry?

@elBoberido
Copy link

@eboasson we do not set the rpath and there is an issue to change this but we did not yet have time to implement it. eclipse-iceoryx/iceoryx#1287

We can give it a higher priority and fix it the next days if necessary.

cc @elfenpiff @MatthiasKillat

@wjwwood
Copy link
Author

wjwwood commented May 5, 2022

It's not super high priority for me, but I wanted to give a thorough bug report. For now I'm fixing the lib manually with install name tool.

I agree with you @eboasson, but the logic about extending the dyld library path I linked to is still suspect/fragile to me.

@elBoberido
Copy link

@wjwwood I'm just wondering why we do not have this problem on the ROS CI. Do you have a newer/older colcon or cmake than on the CI?

@wjwwood
Copy link
Author

wjwwood commented May 5, 2022

We no longer run macOS CI.

Also, it's possible we use/used --merge-install there, but I don't think that was the case.

@elBoberido
Copy link

@wjwwood we are on it but it make take a few more days eclipse-iceoryx/iceoryx#1357

@wjwwood
Copy link
Author

wjwwood commented May 13, 2022

Thanks for the update, I'm not blocked by this at the moment. 👍

@elBoberido
Copy link

@wjwwood it would be great if you could also check if the branch from ApexAI:iox-#1287-setting-rpath-for-all-iceoryx-targets works for you. We checked it on several platforms but we couldn't reproduce the problem with your instructions in the first place and had to trigger it by different means. If this also works for you we would merge the branch to master.

@wjwwood
Copy link
Author

wjwwood commented May 26, 2022

I'm testing it now, but as a tangent, I recommend not using # in your branch names. It messes with the urls in GitHub and is generally annoying to use.

@wjwwood
Copy link
Author

wjwwood commented May 26, 2022

It fails because the libraries already have an LC_RPATH set:

% vcs import ./src < cyclone.repos
..
=== ./src/cyclonedds (git) ===
Cloning into '.'...
Note: switching to '3c81d2856aa088e48d2e9f2365c53c2591ee6fdb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3c81d285 Remove Discovery/Peers/Group and all related code
=== ./src/iceoryx (git) ===
Cloning into '.'...

% colcon build
Starting >>> iceoryx_hoofs
Finished <<< iceoryx_hoofs [8.41s]
Starting >>> iceoryx_posh
--- stderr: iceoryx_posh
Cloning into 'src'...
HEAD is now at fededad one-char typo
Submodule 'deps/meta-cmake' (https://github.com/meta-toolkit/meta-cmake.git) registered for path 'deps/meta-cmake'
Cloning into '/private/tmp/cyclone_ws/build/iceoryx_posh/dependencies/cpptoml/src/deps/meta-cmake'...
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64) option "-add_rpath @executable_path/../lib" would duplicate path, file already has LC_RPATH for: @executable_path/../lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../iceoryx_hoofs found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../iceoryx_hoofs"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../iceoryx_posh found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../iceoryx_posh"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../iceoryx_hoofs/platform found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../iceoryx_hoofs/platform"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../iceoryx_binding_c found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../iceoryx_binding_c"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../hoofs found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../hoofs"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../posh found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../posh"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../hoofs/platform found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../hoofs/platform"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../binding_c found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../binding_c"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../../hoofs found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../../hoofs"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../../posh found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../../posh"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../../hoofs/platform found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../../hoofs/platform"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/../../binding_c found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/../../binding_c"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/hoofs found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/hoofs"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/posh found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/posh"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/hoofs/platform found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/hoofs/platform"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: @executable_path/binding_c found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath @executable_path/binding_c"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /tmp/cyclone_ws/build/iceoryx_posh found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath /tmp/cyclone_ws/build/iceoryx_posh"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /tmp/cyclone_ws/install/iceoryx_hoofs/lib found in: /tmp/cyclone_ws/install/iceoryx_posh/bin/iox-roudi (for architecture x86_64), required for specified option "-delete_rpath /tmp/cyclone_ws/install/iceoryx_hoofs/lib"
---
Finished <<< iceoryx_posh [21.8s]
Starting >>> iceoryx_binding_c
Finished <<< iceoryx_binding_c [3.76s]
Starting >>> cyclonedds
--- stderr: cyclonedds
dyld: Library not loaded: @rpath/libiceoryx_posh.2.90.0.dylib
  Referenced from: /tmp/cyclone_ws/install/iceoryx_binding_c/lib/libiceoryx_binding_c.2.90.0.dylib
  Reason: image not found
/bin/sh: line 1: 65207 Abort trap: 6           ../../../bin/idlc -Wno-implicit-extensibility /tmp/cyclone_ws/src/cyclonedds/src/tools/ddsperf/ddsperf_types.idl
make[2]: *** [src/tools/ddsperf/ddsperf_types.c] Error 134
make[1]: *** [src/tools/ddsperf/CMakeFiles/ddsperf_types_generate.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< cyclonedds [15.3s, exited with code 2]

Summary: 3 packages finished [50.0s]
  1 package failed: cyclonedds
  2 packages had stderr output: cyclonedds iceoryx_posh

@automech-rb
Copy link

automech-rb commented Aug 3, 2022

Hi,

Is there any update on it. I am getting same error building on jetson docker.

Thanks!

@eboasson
Copy link
Contributor

eboasson commented Aug 4, 2022

@automech-rb it appears not ... but I think I found the problem: it seems CMake either simply sets the run-time link path based to the contents of CMAKE_INSTALL_RPATH, or it copies the build-time path, or it doesn't add any at all. None of these options are palatable ...

I always left as many CMake things to others as I could because CMake and I don't go well together, assuming that the way it set CMAKE_INSTALL_RPATH was correct. It seems that it just strips the reference to the Iceoryx library directory instead.

I don't know how fragile the (Cyclone DDS) patch below is, but "it seems to work for me today" ™️. If it works for you, too, I'll turn it into a PR as better than nothing ...

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 2c7ee5e0..fa08094a 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -57,6 +57,10 @@ if(ENABLE_SHM)
   endif()
   find_package(iceoryx_binding_c ${iceoryx_required})
   set(ENABLE_SHM ${iceoryx_binding_c_FOUND} CACHE STRING "" FORCE)
+  if(ENABLE_SHM AND APPLE)
+    get_filename_component(iceoryx_libdir "${ICEORYX_LIB}" DIRECTORY)
+    set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH};${iceoryx_libdir}")
+  endif()
 endif()
 
 # ones that linger in the sources

There's got to be a better way, though ...

@eboasson
Copy link
Contributor

eboasson commented Aug 4, 2022

@automech-rb Scratch that ... it is useful (see #1353) but it doesn't fix the colcon build problem, which really is internal to Iceoryx and so unaffected by my little patch.

The problem is libiceoryx_binding_c not finding libiceoryx_posh, not Cyclone not finding libiceoryx_binding_c. I mixed them up in my head because I have suffered from the latter quite a bit, patching rpaths after builds many times. But it is a different problem.

For me colcon build --merge-install works well, though, and that's what I often used for ROS builds anyway.

@elBoberido
Copy link

@eboasson @automech-rb we are not sure why it doesn't work since we cannot reproduce the problem. There was this PR eclipse-iceoryx/iceoryx#1357 which fixed it for all scenarios we could reproduce. Do you have more information regarding the issue.

cc @dkroenke @elfenpiff

@eboasson
Copy link
Contributor

eboasson commented Aug 4, 2022

@elBoberido there's only one thing that comes to mind: macOS' SIP, which among other things likes to strip out environment variables with names starting with DYLD_. The installation instructions of ROS 2 used (haven't checked if they still do) to tell you to disable it (which I always refused out of principle 😁). If your machine does have SIP disabled, then that might be the difference.

csrutil status tells you whether it is enabled or disabled.

Another thing to check would be whether in your case there are RPATHS embedded in the libraries, e.g. using something like otool -l .../libiceoryx_binding_c.dylib | grep -A2 RPATH. There are probably more elegant ways, but this works well enough.

I don't know what difference there could be that would plausibly result in these being set differently on one machine then on another while using the same build scripts. The least implausible that springs to mind is a change somewhere in macOS and its toolchain. ROS 2 used to support macOS 10.12, I am on (10.)15 and stuff definitely changes from time to time, like Apple's adding the locations it tried for finding the library to the error message. Indeed, that tells me that @wjwwood copy-pasted from an older version of macOS.

@elfenpiff
Copy link
Contributor

@elBoberido @eboasson

There was this PR eclipse-iceoryx/iceoryx#1357 which fixed it for all scenarios we could reproduce.

Actually, this PR prepared everything to fix this issue but in the end did not set the RPATH which would fix the problem. I created a PR to solve this problem once and for all, see eclipse-iceoryx/iceoryx#1587.

cc: @mossmaurice, @dkroenke since you review the PR.

@dkroenke
Copy link
Contributor

@elfenpiff
Is this confirmed to be solved with the changes from that PR? I can remember a discussion from above #1270 (comment) where it doesn't seems to help and we had a reason to deactivate it since it was not properly tested.

@thijsmie
Copy link
Contributor

thijsmie commented Dec 7, 2022

This issue is now outdated and no more complaints reached us, it seems things are working (also, the last remaining issue was on the iceoryx side). I'm going to close this, please open a new issue if any problems persist.

@thijsmie thijsmie closed this as completed Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants