Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vcpkg] Fix Mach-O RPATH duplicate or empty values ​​on macOS #41146

Closed
wants to merge 5 commits into from

Conversation

FrankXie05
Copy link
Contributor

@FrankXie05 FrankXie05 commented Sep 24, 2024

Fix #41122

  • Changes comply with the maintainer guide.
  • SHA512s are updated for each updated download.
  • The "supports" clause reflects platforms that may be fixed by this new version.
  • Any fixed CI baseline entries are removed from that file.
  • Any patches that are no longer applied are deleted from the port's directory.
  • When updating the upstream version, the "port-version" is reset (removed from vcpkg.json).
  • The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
  • Only one version is added to each modified port's versions file.

This PR fixes an issue where install_name_tool tried to add an RPATH that already existed in some Mach-O files or was empty. This issue caused warnings in the vcpkg_fixup_macho_rpath step when processing files like cldr-plurals.

Changes:

  • Added a check to skip adding the new RPATH if it already existed in the Mach-O file.
  • Updated logic to only call install_name_tool if there is an RPATH to remove, avoiding unnecessary operations. Avoid empty warnings.
  • This fix resolves CMake warnings related to duplicate RPATH entries and ensures more efficient processing of Mach-O files during RPATH fixup.

cc @autoantwort

@FrankXie05 FrankXie05 added the category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly label Sep 24, 2024
@FrankXie05 FrankXie05 added the info:internal This PR or Issue was filed by the vcpkg team. label Sep 24, 2024
@jdpurcell
Copy link

I'm kind of confused by the changes but that could be because I know very little CMake syntax and have only a vague idea of what RPATHs are. But assuming the original code was correct, the idea was to call install_name_tool with -delete_rpath for all of the rpaths in rpath_list (representing the existing values in macho_file), and once more with -add_rpath to set the desired target value, i.e. new_rpath.

Then the optimized code aimed to call install_name_tool only once by batching the arguments together, e.g. if you need to do two deletes and an add, it doesn't require three separate invocations of install_name_tool, rather only one invocation with two delete arguments and one add argument. Furthermore it aimed to remove redundancies, i.e. if the correct target rpath is already present, there's no need to delete and re-add it.

In the proposed changes I see:

if(rpath_list STREQUAL "")
    continue()
endif()

What if rpath_list were empty not because a redundant remove/add was eliminated, but because macho_file had no existing values and we were only supposed to add one? It seems the proposed code wouldn't add the new value in this case. Or in any case for that matter, since the code that constructed the -add_rpath argument was removed.

The more I study it, I wonder if the optimized version was fine, with just a small mistake causing the warnings as described in #41122. Specifically here probably:

if(rpath_args STREQUAL "")
    continue()
endif()

If I write some simple test code like:

set(rpath_args)
if(rpath_args STREQUAL "")
    message(STATUS "empty")
else()
    message(STATUS "populated")
endif()

It prints populated. So that makes me think the optimized version is simply failing to continue() if no arguments were constructed, and that's what can cause the warnings about install_name_tool. If that's the case, the fix might be as simple as:

if("${rpath_args}" STREQUAL "")
    continue()
endif()

Or however else you're supposed to do it in CMake.

Perhaps @m-kuhn or @autoantwort have some thoughts since they've worked on this part before?

@dg0yt
Copy link
Contributor

dg0yt commented Sep 26, 2024

I would appreciate to see a specific test.

@FrankXie05
Copy link
Contributor Author

@jdpurcell
Thank you for your detailed reply! I can understand your confusion regarding the changes, and I’ve taken your points into account. Here’s my explanation and response to your comments:

Clarification on the empty rpath_list: You’re right — in the original logic, if the rpath_list was empty, the new RPATH would not be added. My intent with the change was to ensure we skip redundant operations, but I see now that skipping it entirely when rpath_list is empty would lead to the new RPATH not being added at all. I’ve updated the logic to ensure that in cases where the list is empty, we correctly add the new RPATH.

Handling of empty rpath_args: I also addressed the issue where empty rpath_args was being handled improperly. I’ve changed the comparison to if("${rpath_args}" STREQUAL ""), which should resolve the problem with CMake’s handling of empty values.

Optimized install_name_tool invocation: As you mentioned, the optimized version aimed to batch the -delete_rpath and -add_rpath calls together, and I believe this approach should still work with a small fix. I’ve now ensured that both deletions and additions happen in a single install_name_tool invocation, as per the original intent.

@jdpurcell
Copy link

We can do a simulation to help observe and reason about the behavior.

CMakeLists.txt:

set(macho_files "a;b;c;d;e;f")
foreach(macho_file IN LISTS macho_files)
    # The correct/target rpath value
    set(new_rpath "correct")

    # The rpath values currently present on macho_file
    if (macho_file STREQUAL "a")
        set(rpath_list "")
    elseif (macho_file STREQUAL "b")
        set(rpath_list "wrong1")
    elseif (macho_file STREQUAL "c")
        set(rpath_list "correct")
    elseif (macho_file STREQUAL "d")
        set(rpath_list "wrong1" "wrong2")
    elseif (macho_file STREQUAL "e")
        set(rpath_list "wrong1" "correct")
    elseif (macho_file STREQUAL "f")
        set(rpath_list "correct" "wrong1")
    endif()

    # TODO: Compute arguments

    execute_process(
        COMMAND ./print_args.sh ${rpath_args} "${macho_file}"
    )
endforeach()

print_args.sh:

#!/bin/bash

echo "Arguments: $@"

Make sure we can execute it:

chmod +x print_args.sh

Now let's replace the # TODO with the relevant code from a68201b:

list(FIND rpath_list "${new_rpath}" has_new_rpath)
if(NOT has_new_rpath EQUAL -1)
    list(REMOVE_AT rpath_list ${has_new_rpath})
    set(rpath_args)
else()
    set(rpath_args -add_rpath "${new_rpath}")
endif()
foreach(rpath IN LISTS rpath_list)
    list(APPEND rpath_args "-delete_rpath" "${rpath}")
endforeach()
if(rpath_args STREQUAL "")
    continue()
endif()

Run it:

cmake -P CMakeLists.txt

And observe the output:

Arguments: -add_rpath correct a
Arguments: -add_rpath correct -delete_rpath wrong1 b
Arguments: c
Arguments: -add_rpath correct -delete_rpath wrong1 -delete_rpath wrong2 d
Arguments: -delete_rpath wrong1 e
Arguments: -delete_rpath wrong1 f

Everything looks correct to me, with the exception of file "c", which should have skipped executing the command because no rpath changes were needed. Simply adjusting the final if statement to:

if("${rpath_args}" STREQUAL "")

Produces the output I would expect, with the processing of file "c" skipped:

Arguments: -add_rpath correct a
Arguments: -add_rpath correct -delete_rpath wrong1 b
Arguments: -add_rpath correct -delete_rpath wrong1 -delete_rpath wrong2 d
Arguments: -delete_rpath wrong1 e
Arguments: -delete_rpath wrong1 f

If the other changes in this PR are run in the simulation, several issues are revealed:

  1. rpath_args is not cleared during each iteration of the main loop (through macho_files), so the arguments keep accumulating.
  2. -add_rpath is unconditionally added, even if the correct rpath already exists in the file.

So I don't think any changes are needed aside from correcting the one if, but I did kind of like where you were going with the refactoring. I had an attempt at it as well and came up with this:

# Compute arguments
set(rpath_args "")
set(found_new_rpath FALSE)
foreach(rpath IN LISTS rpath_list)
    if (rpath STREQUAL new_rpath)
        set(found_new_rpath TRUE)
    else()
        list(APPEND rpath_args "-delete_rpath" "${rpath}")
    endif()
endforeach()
if(NOT found_new_rpath)
    list(APPEND rpath_args "-add_rpath" "${new_rpath}")
endif()
if("${rpath_args}" STREQUAL "")
    continue()
endif()

This is a few lines longer than the code it would replace, but I find it a bit easier to read and understand. Running through the simulation, it produces the expected output, just with the arguments in a different order (i.e. the adds come after the removes), which should work fine.

@FrankXie05
Copy link
Contributor Author

@jdpurcell
Thank you very much for your local testing, I updated the code and took your suggestions.
This implementation solves the problems you pointed out, such as preventing parameter accumulation and ensuring that the correct RPATH is processed without redundant operations.

Thanks again for your help! :)

Copy link
Contributor

@dg0yt dg0yt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still no test.

scripts/cmake/z_vcpkg_fixup_rpath_macho.cmake Outdated Show resolved Hide resolved
@FrankXie05 FrankXie05 marked this pull request as ready for review September 29, 2024 03:04
@Cheney-W Cheney-W added the info:reviewed Pull Request changes follow basic guidelines label Sep 30, 2024
@JavierMatosD JavierMatosD added the requires:vcpkg-team-review This PR or issue requires someone on the vcpkg team to take a further look. label Sep 30, 2024
@JavierMatosD
Copy link
Contributor

@BillyONeal, @ras0219-msft , @AugP, @data-queue and I discussed this today.

This looks reasonable, but it needs a test that shows it fails today and not with these changes.
There are existing tests in:

/scripts/test_ports/unit-test-cmake/test-z_vcpkg_calculate_corrected_rpath_macho.cmake

@JavierMatosD JavierMatosD marked this pull request as draft October 3, 2024 22:32
@m-kuhn
Copy link
Contributor

m-kuhn commented Oct 9, 2024

@BillyONeal, @ras0219-msft , @AugP, @data-queue and I discussed this today.

This looks reasonable, but it needs a test that shows it fails today and not with these changes. There are existing tests in:

/scripts/test_ports/unit-test-cmake/test-z_vcpkg_calculate_corrected_rpath_macho.cmake

I think this will need a different kind of test, similar to scripts/test_ports/rpath-test

@FrankXie05
Copy link
Contributor Author

I think this will need a different kind of test, similar to scripts/test_ports/rpath-test

Test results:

vcpkg@vcpkg vcpkg % ./vcpkg install rpath-test
Computing installation plan...
The following packages will be built and installed:
    rpath-test:x64-osx@ci
  * rpath-test-binaries:x64-osx@ci
  * vcpkg-cmake:x64-osx@2024-04-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-osx...
Compiler found: /Library/Developer/CommandLineTools/usr/bin/c++
Restored 1 package(s) from /Users/vcpkg/.cache/vcpkg/archives in 24.8 ms. Use --debug to see more details.
Installing 1/3 vcpkg-cmake:x64-osx@2024-04-23...
Elapsed time to handle vcpkg-cmake:x64-osx: 7.57 ms
vcpkg-cmake:x64-osx package ABI: 64dfcbe479694224a95d2f260b61f5e81f3f6b07c555b4898e34731c2c44a4c5
Installing 2/3 rpath-test-binaries:x64-osx@ci...
Building rpath-test-binaries:x64-osx@ci...
-- Note: rpath-test-binaries only supports dynamic library linkage. Building dynamic library.
-- Found external ninja('1.12.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Set install name id of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/debug/lib/librpath-backend-lib.dylib' to '@rpath/librpath-backend-lib.dylib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/debug/lib/librpath-backend-lib.dylib' to '@loader_path'
-- Set install name id of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/debug/lib/librpath-test-lib.dylib' to '@rpath/librpath-test-lib.dylib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/debug/lib/librpath-test-lib.dylib' to '@loader_path'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/debug/tools/rpath-test-binaries/rpath-test-tool' to '@loader_path/../../lib'
-- Set install name id of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/lib/librpath-backend-lib.dylib' to '@rpath/librpath-backend-lib.dylib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/lib/librpath-backend-lib.dylib' to '@loader_path'
-- Set install name id of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/lib/librpath-test-lib.dylib' to '@rpath/librpath-test-lib.dylib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/lib/librpath-test-lib.dylib' to '@loader_path'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/manual-tools/rpath-test-binaries/debug/rpath-test-tool' to '@loader_path/../../../debug/lib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/manual-tools/rpath-test-binaries/rpath-test-tool' to '@loader_path/../../lib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/tools/rpath-test-binaries/debug/rpath-test-tool' to '@loader_path/../../../debug/lib'
-- Adjusted RPATH of '/Users/vcpkg/Frank/vcpkg/packages/rpath-test-binaries_x64-osx/tools/rpath-test-binaries/rpath-test-tool' to '@loader_path/../../lib'
-- Performing post-build validation
Stored binaries in 1 destinations in 19.7 ms.
Elapsed time to handle rpath-test-binaries:x64-osx: 1.6 s
rpath-test-binaries:x64-osx package ABI: b0f7f0d57f1f17c9355e20bbfefde577e24a798930d99ba410db95f97bdb7f72
Installing 3/3 rpath-test:x64-osx@ci...
Building rpath-test:x64-osx@ci...
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Stored binaries in 1 destinations in 9.68 ms.
Elapsed time to handle rpath-test:x64-osx: 2.6 s
rpath-test:x64-osx package ABI: 45e5a5145bc49840fff228eee3e9d847e018e55266790ffeec088b2b912e8ca4
Total install time: 4.2 

@m-kuhn
Copy link
Contributor

m-kuhn commented Oct 9, 2024

what I meant is: to test this fix, it will require a test which builds binaries that show this problem (and not "just" a test that checks some strings produced by helper cmake functions)

Some affected ports

  • libexiv2
  • protobuf
  • plenty of the qt* ports

@autoantwort
Copy link
Contributor

The code is more or less the same as before, but it fixes the following bug in the existing code:

set(test_var)
if (test_var STREQUAL "")
    message(STATUS "test_var is empty")
else()
    message(STATUS "test_var is not empty")
endif()

outputs "test_var is not empty". So it would be enough to change

if(rpath_args STREQUAL "")

to

if(NOT rpath_args)

m-kuhn added a commit to m-kuhn/vcpkg that referenced this pull request Oct 9, 2024
@BillyONeal
Copy link
Member

I'm closing this because I merged the competing fix #41443 . Thanks for your contribution to vcpkg!

@BillyONeal BillyONeal closed this Oct 9, 2024
@BillyONeal
Copy link
Member

what I meant is: to test this fix, it will require a test which builds binaries that show this problem (and not "just" a test that checks some strings produced by helper cmake functions)

I think this problem should be reducible to string manipulation testing. However @autoantwort's 1 line fix seemed like a much more targeted fix so I merged it despite not much testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:internal This PR or Issue was filed by the vcpkg team. info:reviewed Pull Request changes follow basic guidelines requires:vcpkg-team-review This PR or issue requires someone on the vcpkg team to take a further look.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[gettext] Couldn't adjust RPATH on macOS
8 participants