[scripts|cmake] Fix up install name and rpath on osx dynamic #32200

liviaerxin · 2023-06-24T15:35:58Z

It's supposed to uniform dynamic libraries and executable install name and rpath on osx of arm64-osx-dynamic triplet. Due to many reasons, some individual ports may set the wrong rpath and install name on osx, or not set them. Here's hoping to manage them in a standard way in vcpkg.

It is supposed to solve general problems like #31719, continuing on #31720, also referring #23035

Here's the rule which may be quite different than fixing rpath mentioned at #23035 as this following:

After building and installing a package into /opt/vcpkg/packages/xxxx_arm64-osx-dynamic, it'll do the fix on Mach-O files in lib/, debug/lib/, tools/xxx/, following:

for shared libraries in lib/ and debug/lib/:
- fix install name on itself to be with @rpath prefix
- fix install name in its dependent shared libraries to be with @rpath prefix
  - include libraries only from the same port, assuming other libraries from other managed ports by vcpkg are already fixed!
  - exclude system libraries in /usr/lib/ and /System/Library!
- fix rpath on itself to be @loader_path currently (Or using absolute path CURRENT_INSTALLED_DIR, like /opt/vcpkg/installed/arm64-osx-dynamic/lib or /opt/vcpkg/installed/arm64-osx-dynamic/debug/lib?)
for exe in tools/{package}/bin/
- fix install name in its dependent shared libraries to be with @rpath prefix
  - include libraries only from the same port in lib/, assuming other libraries from other managed ports by vcpkg are already fixed!
  - exclude system libraries in /usr/lib/ and /System/Library!
- fix rpath to be @loader_path/../../../ to be relative with lib/(Or using absolute path CURRENT_INSTALLED_DIR, like /opt/vcpkg/installed/arm64-osx-dynamic/lib?)

Have done a test on [qtbase] and others

Before:

❯ ~/Documents/dlopen_test.out /opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/rcc
dlopen(/opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/rcc, 0x0001): Library not loaded: @rpath/libQt6Core.6.dylib
  Referenced from: <28854ECA-06DA-351C-81B3-C922A23B6832> /opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/rcc
  Reason: tried: '/opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/../lib/libQt6Core.6.dylib' (no such file), '/opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/../lib/libQt6Core.6.dylib' (no such file), '/usr/local/lib/libQt6Core.6.dylib' (no such file), '/usr/lib/libQt6Core.6.dylib' (no such file, not in dyld cache)
dlopen sample%                                                                                                                         

❯ otool -L /opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libQt6Core_debug.6.dylib
/opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libQt6Core_debug.6.dylib:
	@rpath/libQt6Core_debug.6.dylib (compatibility version 6.0.0, current version 6.5.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.50.120)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 61.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
	/opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libb2.1.dylib (compatibility version 2.0.0, current version 2.4.0)
	@rpath/libicui18n.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	@rpath/libicuuc.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	@rpath/libicudata.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libdouble-conversion.3.dylib (compatibility version 3.0.0, current version 3.2.0)
	@rpath/libpcre2-16.0.dylib (compatibility version 12.0.0, current version 12.2.0)
	/opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	@rpath/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

After fix:

❯ ~/Documents/dlopen_test.out /opt/vcpkg/installed/arm64-osx-dynamic/tools/Qt6/bin/rcc
dlopen sample%                                                                                                                                                                                                                                       

❯ otool -L /opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libQt6Core_debug.6.dylib
/opt/vcpkg/installed/arm64-osx-dynamic/debug/lib/libQt6Core_debug.6.dylib:
	@rpath/libQt6Core_debug.6.dylib (compatibility version 6.0.0, current version 6.5.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.50.120)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 61.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
	@rpath/libb2.1.dylib (compatibility version 2.0.0, current version 2.4.0)
	@rpath/libicui18n.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	@rpath/libicuuc.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	@rpath/libicudata.73.dylib (compatibility version 73.0.0, current version 73.1.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libdouble-conversion.3.dylib (compatibility version 3.0.0, current version 3.2.0)
	@rpath/libpcre2-16.0.dylib (compatibility version 12.0.0, current version 12.2.0)
	@rpath/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	@rpath/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	@rpath/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

…install-name-rpath-osx-dynamic

BillyONeal

I like this direction.

BillyONeal · 2023-07-25T20:15:18Z

scripts/ports.cmake

+        if(VCPKG_FIXUP_MACHO_RPATH)
+            include("${SCRIPTS}/cmake/z_vcpkg_fixup_install_name_rpath.cmake")
+        endif()


Suggested change

if(VCPKG_FIXUP_MACHO_RPATH)

include("${SCRIPTS}/cmake/z_vcpkg_fixup_install_name_rpath.cmake")

endif()

z_vcpkg_fixup_install_name_rpath_in_dir()

and include unconditionally at the top. The function should always be available.

Does that imply that linux rpath fixup mechanism should also be included unconditionally?

So let z_vcpkg_fixup_install_name_rpath_in_dir() check the running triplet like when hitting *-osx-dynamic to run the fixup?

BillyONeal · 2023-07-25T20:15:23Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+  endforeach()
+endfunction()
+
+z_vcpkg_fixup_install_name_rpath_in_dir()


Suggested change

z_vcpkg_fixup_install_name_rpath_in_dir()

BillyONeal · 2023-07-25T20:16:30Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+# │       ├── libicudata.72.dylib -> libicudata.72.1.dylib
+# │       ├── libicudata.dylib -> libicudata.72.1.dylib


Note that this pattern is already broken as symlinks aren't cachable. If that's necessary then yet another problem with osx-dyanamic :(

It's not necessary, just showing a build tree from icu.

The script just tries to resolve a install name for the underlying dylib non-link file while keeping orginal symlinks, See at:

# Get the major version string(REGEX REPLACE "(lib[^\\.]+\\.[0-9]+).*\\.(dylib|so)" "\\1.\\2" filename_major "${filename}") set(install_name_file "${file_dir}/${filename_major}") # Try the major version file if(NOT EXISTS "${install_name_file}") # Get the major.minor version if major version file is missing string(REGEX REPLACE "(lib[^\\.]+\\.[0-9]+.[0-9]+).*\\.(dylib|so)" "\\1.\\2" filename_major_minor "${filename}") set(install_name_file "${file_dir}/${filename_major_minor}") # Fall back to if the major version file not found if(NOT EXISTS "${install_name_file}") set(install_name_file "${file}") endif() endif()

BillyONeal · 2023-07-25T20:17:46Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+    DOC "Absolute path of otool cmd"
+  )
+
+  if(NOT otool_cmd)
+    message(WARNING "otool not found!")
+    return()
+  endif()


Suggested change

DOC "Absolute path of otool cmd"

)

if(NOT otool_cmd)

message(WARNING "otool not found!")

return()

endif()

DOC "Absolute path of otool cmd"

REQUIRED

)

This can't depend on what happens to be on the box, it needs to be done, a warning isn't sufficient.

Otherwise we end up binary caching broken bits from a broken machine and restoring them on a hypothetically working machine.

BillyONeal · 2023-07-25T20:18:07Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+    DOC "Absolute path of install_name_tool cmd"
+  )
+
+  if(NOT install_name_tool_cmd)
+    message(WARNING " install_name_tool not found!")
+    return()
+  endif()


Suggested change

DOC "Absolute path of install_name_tool cmd"

)

if(NOT install_name_tool_cmd)

message(WARNING " install_name_tool not found!")

return()

endif()

DOC "Absolute path of install_name_tool cmd"

REQUIRED

)

ditto.

BillyONeal · 2023-07-25T20:18:25Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+#       - exclude system libraries in `/usr/lib/` and `/System/Library`!
+#    - fix rpath to `@loader/../../../`  to be relative with `lib/`
+function(z_vcpkg_fixup_install_name_rpath_in_dir)
+  find_program(


Can we do better than find_program, e.g. fetching this for the user if possible?

Two feasible ways to install otool and install_name_tool tools, bundled in Xcode Command Line Tools:

Run xcode-select --install in terminal.

Goto developer.apple.com/downloads but you have to login first.

The 1st seems more reasonable.

BillyONeal · 2023-07-25T20:19:11Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+        )
+
+        if(NOT set_install_name_rv EQUAL 0)
+          message(WARNING "Failed, install_name_tool -id ${id_name} ${file}, \n${set_install_name_ov}")


Suggested change

message(WARNING "Failed, install_name_tool -id ${id_name} ${file}, \n${set_install_name_ov}")

message(FATAL_ERROR "Failed, install_name_tool -id ${id_name} ${file}, \n${set_install_name_ov}")

BillyONeal · 2023-07-25T20:19:29Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+      )
+
+      if(NOT set_rpath_rv EQUAL 0)
+        message(WARNING "Failed, install_name_tool -add_rpath ${rpath} ${file}, \n${set_rpath_ov}")


Suggested change

message(WARNING "Failed, install_name_tool -add_rpath ${rpath} ${file}, \n${set_rpath_ov}")

message(FATAL_ERROR "Failed, install_name_tool -add_rpath ${rpath} ${file}, \n${set_rpath_ov}")

BillyONeal · 2023-07-25T20:20:04Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+            message(WARNING "Failed, install_name_tool -change ${dep_install_name} ${id_name} ${file}, \n${change_dep_install_name_ov}")
+            continue()
+          else()
+            message(DEBUG "Fix ${file}: ${dep_install_name} to ${id_name}")
+            break()


Suggested change

message(WARNING "Failed, install_name_tool -change ${dep_install_name} ${id_name} ${file}, \n${change_dep_install_name_ov}")

continue()

else()

message(DEBUG "Fix ${file}: ${dep_install_name} to ${id_name}")

break()

message(FATAL_ERROR "Failed, install_name_tool -change ${dep_install_name} ${id_name} ${file}, \n${change_dep_install_name_ov}")

m-kuhn · 2023-09-20T15:50:19Z

Can I help to move this forward?

m-kuhn · 2023-12-13T17:36:51Z

Can anything be done to revive this? It's an immense quality of life improvement for *-osx-dynamic

m-kuhn · 2023-12-14T07:20:28Z

scripts/cmake/z_vcpkg_fixup_install_name_rpath.cmake

+          message(DEBUG "${dep_install_name} match with ${dep_file} of install name ${id_name}")
+
+          execute_process(
+            COMMAND "${install_name_tool_cmd}" -change "${dep_install_name}" "${id_name}" "${file}"


This does not work here (in every case).
E.g. for tools/python3.11 it will not replace libpython3.11.dylib because the tool will already load the library from the install path but we replace the package path. Wouldn't it be better to use the name as detected in dep_install_name (without modification on as done here: https://github.com/microsoft/vcpkg/pull/32200/files#diff-20a469d007bd17c131ed093757f539dffc2fe3d47e89e154d349a8854899f9ceR376)?

JonLiu1993 · 2024-05-14T06:38:31Z

Closing this PR since it seems that no progress is being made. Please ping us to reopen if work is still being done.

autoantwort · 2024-05-22T16:01:12Z

Was also requested on Slack: https://cpplang.slack.com/archives/C7BFF7RCJ/p1716381580159229

m-kuhn · 2024-05-22T16:38:07Z

@autoantwort continued in #36556 I still need to go over the comments by @Neumann-A , help welcome.

liviaerxin added 2 commits June 24, 2023 22:10

Fixup install name and rpath after building dynamic libraries on osx

4fefba4

Update comments

2df07ea

liviaerxin changed the title ~~Fixup install name and rpath after building dynamic libraries on osx~~ [vcpkg]Fixup install name and rpath after building dynamic libraries on osx Jun 24, 2023

liviaerxin changed the title ~~[vcpkg]Fixup install name and rpath after building dynamic libraries on osx~~ [vcpkg] Fixup install name and rpath after building dynamic libraries on osx Jun 24, 2023

JonLiu1993 self-assigned this Jun 25, 2023

JonLiu1993 added category:community-triplet A PR or issue related to community triplets not officially validated by the vcpkg team. category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly labels Jun 25, 2023

JonLiu1993 changed the title ~~[vcpkg] Fixup install name and rpath after building dynamic libraries on osx~~ [scripts|cmake] Fixup install name and rpath after building dynamic libraries on osx Jun 25, 2023

liviaerxin mentioned this pull request Jun 25, 2023

[libb2] Fix using absolute install name in arm64-osx-dynamic triple #31720

Closed

liviaerxin changed the title ~~[scripts|cmake] Fixup install name and rpath after building dynamic libraries on osx~~ [scripts|cmake] Fix up install name and rpath on osx dynamic Jun 26, 2023

liviaerxin and others added 3 commits June 26, 2023 21:32

chore: do fixup if necessary tools found

b94dc25

Merge branch 'master' of https://github.com/microsoft/vcpkg into fix-…

bb9e58a

…install-name-rpath-osx-dynamic

Merge branch 'master' of https://github.com/microsoft/vcpkg into fix-…

fdb79c6

…install-name-rpath-osx-dynamic

JonLiu1993 marked this pull request as draft July 13, 2023 03:00

dg0yt mentioned this pull request Jul 13, 2023

[qt5-base] fix building with mac dynamic triplet #32500

Merged

7 tasks

Merge branch 'master' of https://github.com/microsoft/vcpkg into fix-…

856a124

…install-name-rpath-osx-dynamic

BillyONeal reviewed Jul 25, 2023

View reviewed changes

dg0yt mentioned this pull request Sep 20, 2023

[spatialite] fix building with x64-osx-dynamic #33841

Closed

dg0yt mentioned this pull request Nov 10, 2023

Change z_vcpkg_fixup_rpath_in_dir to behave like other vcpkg helpers #34879

Closed

m-kuhn reviewed Dec 14, 2023

View reviewed changes

m-kuhn mentioned this pull request Feb 3, 2024

[scripts|cmake] Fix up install name and rpath on osx dynamic #36556

Closed

JonLiu1993 closed this May 14, 2024

m-kuhn mentioned this pull request Jul 17, 2024

Fix osx-dynamic install names for updated dependent shared library ids #39889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[scripts|cmake] Fix up install name and rpath on osx dynamic #32200

[scripts|cmake] Fix up install name and rpath on osx dynamic #32200

liviaerxin commented Jun 24, 2023 •

edited

Loading

BillyONeal left a comment

BillyONeal Jul 25, 2023

Osyotr Jul 25, 2023

liviaerxin Jul 26, 2023

BillyONeal Jul 25, 2023

BillyONeal Jul 25, 2023

liviaerxin Jul 26, 2023

BillyONeal Jul 25, 2023

BillyONeal Jul 25, 2023

BillyONeal Jul 25, 2023

liviaerxin Jul 26, 2023

BillyONeal Jul 25, 2023

BillyONeal Jul 25, 2023

BillyONeal Jul 25, 2023

m-kuhn commented Sep 20, 2023

m-kuhn commented Dec 13, 2023

m-kuhn Dec 14, 2023

JonLiu1993 commented May 14, 2024

autoantwort commented May 22, 2024

m-kuhn commented May 22, 2024

		# │ ├── libicudata.72.dylib -> libicudata.72.1.dylib
		# │ ├── libicudata.dylib -> libicudata.72.1.dylib

	message(WARNING "Failed, install_name_tool -id ${id_name} ${file}, \n${set_install_name_ov}")
	message(FATAL_ERROR "Failed, install_name_tool -id ${id_name} ${file}, \n${set_install_name_ov}")

[scripts|cmake] Fix up install name and rpath on osx dynamic #32200

[scripts|cmake] Fix up install name and rpath on osx dynamic #32200

Conversation

liviaerxin commented Jun 24, 2023 • edited Loading

BillyONeal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m-kuhn commented Sep 20, 2023

m-kuhn commented Dec 13, 2023

Choose a reason for hiding this comment

JonLiu1993 commented May 14, 2024

autoantwort commented May 22, 2024

m-kuhn commented May 22, 2024

liviaerxin commented Jun 24, 2023 •

edited

Loading