Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ld-2.24.90 does not support -l:/full/path/to/library.so #694

Closed
tho- opened this issue Nov 13, 2014 · 58 comments
Closed

ld-2.24.90 does not support -l:/full/path/to/library.so #694

tho- opened this issue Nov 13, 2014 · 58 comments

Comments

@tho-
Copy link

tho- commented Nov 13, 2014

pkg-config files are generated with items like -l:/full/path/to/library.so.
(See https://github.com/ros/catkin/blob/indigo-devel/cmake/catkin_package.cmake#L264 )

This does not seem to work with ld-2.24.90. Only -l:library.so works (also, -L/ -l:/full/path/ seems to work, but it's not clear if this is an intended behaviour or not). The ld manual only talks about -l:library.so.

I could not find yet it this is a regression in ld or a deliberate change.

One option could be to change catkin so that it generates proper -L/full/path -lrary in the .pc to link with /full/path/library.so.
Another option is to add an explicit -L/ in front of the -l: items.

Any comment?

@dirk-thomas
Copy link
Member

Using -l:/path/to/lib is necessary to prevent pkg-config from reordering libraries.

This was working well until now. What system are you experiencing this?

Using -L and -l is not really applicable since it changes resolution of libraries and would break workspace chaining. I don't know a different way to maintain the current behavior.

@tho-
Copy link
Author

tho- commented Nov 13, 2014

On Thursday, at 10:13, Dirk Thomas wrote:
| Using -l/path/to/lib is necessary to prevent pkg-config from reordering libraries.

You mean -l:/path/to/lib? (-l/path/to/lib has never worked).

AFAIK, pkg-config does not reorder libraries (this would be a bug). But yes, I
understand your point that -L and -l are not useable.

| This was working well until now. What system are you experiencing this?

ld-2.24.90, from ubuntu-14.10

% ld --version
GNU ld (GNU Binutils for Ubuntu) 2.24.90.20141014
% gcc -xc /dev/null -l:/usr/lib/x86_64-linux-gnu/libc.so
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libc.so
collect2: error: ld returned 1 exit status

@dirk-thomas
Copy link
Member

Yes, -l:/path/to/lib.

It sadly does reorder libraries - please see more info in #300.

If that effects Utopic it will be "problematic" to build any dry packages which depend on pkg-config.

@tho-
Copy link
Author

tho- commented Nov 13, 2014

On Thursday, at 11:13, Dirk Thomas wrote:
| It sadly does reorder libraries - please see more info in #300.

I do not see any reordering issue in #300 ?
#300 is about fixing -l/path/to/lib by using -l:/path/to/lib

Maybe what you mean is that pkg-config does not properly handle items that are
a mere full path (e.g. just /path/to/lib.so), while this is a perfectly valid
ld argument. pkg-config basically only recognizes -L, -l and may do weird thing
with the rest. That is a pkg-config issue.

| If that effects Utopic it will be "problematic" to build any dry packages which
| depend on pkg-config.

Yes, hence this PR :)

I think all this comes from a more general issue that is a misunderstanding of
what pkg-config does (no offense meant!).

The Libs: is not about listing a library's dependencies. Libs: is just meant to
provide information on how to link with a library. The link-time dependencies
are handled by the ELF format itself (or whatever object format the target
uses). In other words, for a package foo providing libfoo, Libs: is usually
just "-L${prefix}/lib -lfoo", no matter what shared libraries libfoo.so was
linked with.

The only scenario where Libs: should contain something more is when the foo
package provides public headers that do not properly hide their dependencies
and do reference symbols from other libraries (because that other dependency is
exposed and becomes a primary dependency of whatever compiles/links with
foo).

Also, there is the Libs.private: which is meant for static linking, because
static archives do not carry their own dependency information and all of them
have to be explicited at link time.

So, if the goal is to find the most simple fix, then adding just a "-L/" in
front of all the -l: stuff seems to work. I'm just a bit worried by the fact
that this looks more like a fragile workaround than a proper fix.

@tho-
Copy link
Author

tho- commented Nov 13, 2014

On Thursday, at 21:05, Anthony Mallet wrote:
| I do not see any reordering issue in #300 ?
| #300 is about fixing -l/path/to/lib by using -l:/path/to/lib

More on this ... because I would really be surprised that pkg-config reorders
stuff (this is against linking semantics). Do you have an example of a .pc not
working?

% cat foo.pc
Name: foo
Version: 1.0
Description: foo

Libs: -L/foo /usr/lib/libfoo.so -lbar /usr/lib/libfoo.so -lbar

% env PKG_CONFIG_PATH=. pkg-config --libs foo
-L/foo /usr/lib/libfoo.so -lbar /usr/lib/libfoo.so -lbar

This looks correct to me.

@tho-
Copy link
Author

tho- commented Nov 20, 2014

See also the issue reported to the binutils people here
https://sourceware.org/bugzilla/show_bug.cgi?id=17606

If somebody wants to help convincing them ... :)

@jbohren
Copy link
Member

jbohren commented Nov 20, 2014

Wow...

@kartikmohta
Copy link

Reviving an old thread, but I'm facing the same issue on Arch with binutils 2.25.0. One way to solve the issue is to only add -l:library.so instead of -l:/full/path/to/library.so to the generated pkg-config files. I haven't tested whether this works on the supported Ubuntu platforms, but for someone looking for a quick solution, here is the hack:

diff --git a/cmake/catkin_package.cmake b/cmake/catkin_package.cmake
index 7d56edf..a8f9257 100644
--- a/cmake/catkin_package.cmake
+++ b/cmake/catkin_package.cmake
@@ -261,7 +261,8 @@ function(_catkin_package)
     if(IS_ABSOLUTE ${library})
       get_filename_component(suffix ${library} EXT)
       if(NOT "${suffix}" STREQUAL "${CMAKE_STATIC_LIBRARY_SUFFIX}")
-        set(library "-l:${library}")
+        get_filename_component(libname ${library} NAME)
+        set(library "-l:${libname}")
       endif()
     else()
       set(library "-l${library}")

@dirk-thomas
Copy link
Member

Just using the library name without the absolute path won't work since a) it does not set the library dirs anywhere and b) even if it would prepend a -L that would still make the lookup try wrong locations first (due to non-deterministic order of the library paths). So I am afraid the proposed patch is not a viable solution.

@kartikmohta
Copy link

Well, according to the ld man page, it says

-l namespec
If namespec is of the form :filename, ld will search the library path for a file called filename, otherwise it will search the library path for a file called libnamespec.a.

so it should only search for the file in the library path. And as per https://sourceware.org/bugzilla/show_bug.cgi?id=17532#c3, the use of full path in the argument is an undocumented ld feature, in actual fact a ld bug, that has since been fixed.
This would require a fix for the next Ubuntu release (15.04) which includes ld 2.25.0. Why is the full library path needed anyway?

@dirk-thomas
Copy link
Member

Using numerous -L and -l arguments in large scale builds like ROS packags with lots of dependencies has the problem that we can not ensure the right search order for libraries.

Especially when intentionally overlaying workspaces (http://www.ros.org/reps/rep-0128.html#overlays). E.g. when a workspace B overlays workspace A that means that any shared library from workspace B should be preferred over any from workspace A. if every package simply exports library dirs and library names the order in which you find and use them results in a non-deterministic order of -L flags on the link line. Therefore each package exports absolute paths to their libraries.

I could not imagine a workaround for the ROS buildsystem which does not involve a significant refactoring of all ROS packages if the linker would suddenly not support absolute paths anymore. Also we are building ROS Jade packages for Ubuntu Vivid for several months now and I have not seen a single failure related to this. So to me it seems that its ld version still supports that use case.

@kartikmohta
Copy link

@dirk-thomas It only occurs when building rosbuild packages, catkin packages build fine so you might not have noticed it. I just tried building a rosbuild package, with a trivial executable as the target, depending on roscpp (with jade on Ubuntu 15.04) and it gave me errors in the linking stage

Linking CXX executable ../bin/test_node                                          
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_signals.so        
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_filesystem.so     
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/liblog4cxx.so              
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_regex.so          
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_date_time.so      
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_system.so         
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libboost_thread.so         
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libpthread.so              
/usr/bin/ld: cannot find -l:/usr/lib/x86_64-linux-gnu/libconsole_bridge.so       
collect2: error: ld returned 1 exit status

@dirk-thomas
Copy link
Member

I didn't notice that you referred to -l:. The reason for that is described in other comments above:

Removing the colon is therefore not an option (at least that was the behavior at that time). And dropping the path and using only the filename will result in the ordering problem described.

@kartikmohta
Copy link

Sadly then seems like rosbuild package building would be broken from Utopic onwards...

@dirk-thomas
Copy link
Member

I am open to suggestions if you can imagine a way to address the problem.

@j-rivero
Copy link

j-rivero commented Apr 2, 2015

It could sound a bit stupid but: what about adding the absolute path to the .pc file without the -l flag, just as the absolute path?

jrivero@nium tmp $ cat foo.pc 
Name: foo
Description: The foo library
Version: 1.0.0
Libs: /tmp/libboost_system.so.1.54.0
jrivero@nium tmp $ g++ foo.cc -o foo `PKG_CONFIG_PATH=. pkg-config --libs foo`

We can discuss (and probably lost the discussion) if we are not breaking the semantic of the Libs: clause of pkg-config, which specifies:

Libs: The link flags specific to this package and any required libraries that don't support pkg-config. The same rule as Cflags applies here.

I've found that Ogre and OpenCV are doing this kind of tricks in OGRE-PCZ.pc (together with rpath). In the case of OpenCV I think that the .pc file kind of buggy: ... ${exec_prefix}/lib/x86_64-linux-gnu/libopencv_videostab.so -lopencv_videostad

@j-rivero
Copy link

j-rivero commented Apr 2, 2015

I was able to create a catkin_workspace with catkin modified with this patch and ros_comm. All seems to be working. Absolute paths are passing to the compiler without the -l

Linking CXX executable /home/jrivero/code/ros/ws/devel/lib/rosbag/play
cd /home/jrivero/code/ros/ws/build/ros_comm/tools/rosbag && /usr/bin/cmake -E cmake_link_script CMakeFiles/play.dir/link.txt --verbose=1
/usr/bin/c++       CMakeFiles/play.dir/src/play.cpp.o  -o /home/jrivero/code/ros/ws/devel/lib/rosbag/play -rdynamic /home/jrivero/code/ros/ws/devel/lib/librosbag.so /home/jrivero/code/ros/ws/devel/lib/librosbag_storage.so -lbz2 -lboost_program_options /home/jrivero/code/ros/ws/devel/lib/libroslz4.so -llz4 /home/jrivero/code/ros/ws/devel/lib/libtopic_tools.so /home/jrivero/code/ros/ws/devel/lib/libroscpp.so -lboost_signals -lboost_filesystem /home/jrivero/code/ros/ws/devel/lib/librosconsole.so /home/jrivero/code/ros/ws/devel/lib/librosconsole_log4cxx.so /home/jrivero/code/ros/ws/devel/lib/librosconsole_backend_interface.so -llog4cxx -lboost_regex /opt/ros/indigo/lib/libroscpp_serialization.so /opt/ros/indigo/lib/librostime.so -lboost_date_time /home/jrivero/code/ros/ws/devel/lib/libxmlrpcpp.so /opt/ros/indigo/lib/libcpp_common.so -lboost_system -lboost_thread -lpthread -lconsole_bridge -Wl,-rpath,/home/jrivero/code/ros/ws/devel/lib:/opt/ros/indigo/lib 

I was not able to find any singe rosbuild package. @kartikmohta could please point me to your rosbuild package or test the patch by yourself?

@dirk-thomas
Copy link
Member

I think we first should reproduce the original problem (the reordering) which was the reason why this was changed in the first place (#300).

@gerkey
Copy link
Contributor

gerkey commented May 11, 2015

I'm looking into this, too. As @tho- asked, can somebody provide a .pc file that will cause pkg-config to reorder the libs?

@gerkey
Copy link
Contributor

gerkey commented May 11, 2015

Removing the -l: prefixes in the .pc files in my Jade installation, to leave bare absolute library paths (cd /opt/ros/jade/lib/pkgconfig; for f in *.pc; do sed -i 's/-l://g' $f; done) fixes my simple example rosbuild package on Utopic. Presumably I've broken something else in exchange; how I can demonstrate that other broken thing?

@tho-
Copy link
Author

tho- commented May 11, 2015

On Monday, at 14:55, gerkey wrote:
| Removing the -l: prefixes in the .pc files in my Jade installation, to leave bare
| absolute library paths (cd /opt/ros/jade/lib/pkgconfig; for f in *.pc; do sed -i 's
| /-l://g' $f; done) fixes my simple example rosbuild package on Utopic.

To strictly comply with pkg-config, I think (to be checked again) that Libs:
may only contain -L/-l flags. Absolute path to libraries is not supported,
although it will/should work in practice?

I think that the reordering issues come from libtool, which is too ... clever
(hum) and separates -L/-l and other flags (like absolute paths to libs) and
then does it's own reordering. So the issue is that if you use -lstuff
/full/path/to/other/stuff, you might end-up linking in the wrong order
(other/stuff before stuff or vice-versa). If you use only -L/-l flags, then
libtoo should still do the right thing. So there is no issue if each library is
installed only once in a unique place.

[I know that libtool is not used by ros packages, but it happens that I do use
it myself in combination with autoconf (hard or impossible to use autoconf
without libtool those days) in some of my (not ros) packages.]

The reordering, when using ony -L/-l and no absolute paths, is only an issue if
you try to solve the scenario where -lstuff might be taken from different -Lpath
(i.e. stuff exists in different places in the system). E.g. :

% ls A/
libfoo.so libbar.so
% ls B/
libfoo.so libbar.so
% gcc ... -LA/ -lfoo -LB/ -lbar

i.e trying to pick-up foo from A and bar from B where both A and B both contain
foo and bar.

My personal opinion on this is that trying to solve this is bound to fail, no
matter what you try to make it apparently work. I personnaly always avoid this
situation. If I need different versions of something, I install everything
into different prefixes, that do not know each other and do not interfere.

For instance, if it happens that foo depends on bar, then the libfoo.so
will have a registered dependency on "libbar.so". And if it there is an ELF
RPATH flag as well in libfoo pointing on A/, then you are going to have
confliting versions of libbar.so (picked up both from A because of libfoo.so
and from B because of the ld flags). This is only the most trivial example of
what could go wrong. I'm pretty sure you can imagine many other kind of
failures.

So, I would personnaly say that the reordering issue is ... not an issue :)
And using only -L/-l in .pc files is the only supported, working, standard and
usual way to go :)

@wjwwood
Copy link
Member

wjwwood commented May 11, 2015

My personal opinion on this is that trying to solve this is bound to fail, no
matter what you try to make it apparently work.

Well, to be fair, it didn't fail when ld provided the mechanism for us to order it and keep it ordered. From my perspective (maybe I'm not forseeing yet another issue that we don't handle) if either ld left this mechanism in place (or one like it) or if pkg-config didn't reorder things, then everything would be working.

If I need different versions of something, I install everything
into different prefixes, that do not know each other and do not interfere.

That's great if it works for you, but for most of our users who don't have an intimate knowledge of linkers, it still leaves them with a subtly broken library exporting mechanism when used with rosbuild. So I think we should still try to resolve the issue, if possible.

Presumably I've broken something else in exchange; how I can demonstrate that other broken thing?

@gerkey I have an idea of how to setup a simple(ish) set of workspaces which reproduces the issue, but they'll be sort of contrived. If you'd like to get together in the near future maybe we can work through it on a white board. Unfortunately I don't remember the exact scenario where this caused us a problem before, so finding an existing package with this issue may not be the easiest way to reproduce the problem.

@dirk-thomas
Copy link
Member

@wjwwood You need to link against two libraries which work with one order but not the other. Usually you do that by having one library require symbols from the other library. While the symbols required for the executable will always be found - independent of the library order - the symbols used in one library provided by the other will only work for one of the the cases. Based on that you can construct the two workspaces.

@gerkey
Copy link
Contributor

gerkey commented May 13, 2015

@tho- I appreciate that using something other than -L/-l flags in Libs: is unusual, but I can't find anything in the documentation that advises against it. E.g., from the man page:

Libs:  This line should give the link flags specific to your package.  Don't add any flags for required packages; pkg-config will add  those  automatically

@gerkey
Copy link
Contributor

gerkey commented May 13, 2015

Hypothesis: pkg-config isn't reordering libraries, but rosbuild is. To get build flags, rosbuild calls out to rospack, which in turn calls out to pkg-config (via rosdep2 running inside an embedded Python interpreter, but for own sanity, let's pretend that that last step is not happening).

For libs, rosbuild calls out three times, once for libs-only-L, once for libs-only-l, and once more for libs-only-other. The resulting lists are then used separately with different CMake calls: link_directories, target_link_libraries, and set_target_properties LINK_FLAGS.

When we specify in Libs: an absolute library without a -l prefix, it ends up in the libs-only-other bucket, which is given to CMake separately from the other libs, without preserving ordering between -lfoo args and /path/to/libbar.so args. I would expect ordering to be preserved within the -lfoo args and within the /path/to/libbar.so args (I haven't verified this), but it's up to CMake to decide where on the command line the link flags go with respect to the libs (link flags probably go first?).

So, a potential fix is to modify rosbuild to call rospack differently, or to use the results differently. We already have an example of parsing the contents of the return from libs-only-other to treat absolute library path names differently (only static libs in that case).

Before I implement this fix, any comments or alternative suggestions?

@gerkey
Copy link
Contributor

gerkey commented May 13, 2015

To clarify, my proposal is:

  • Change catkin to produce .pc files that don't use any -l prefix on absolute library path names in Libs:.
  • Change rosbuild to give absolute library path names to CMake in target_link_libraries (along with non-absolute library names, preserving order among them all), rather than set_target_properties LINK FLAGS.

@wjwwood
Copy link
Member

wjwwood commented May 13, 2015

@gerkey That sounds reasonable, and I trust your diagnosis of the issue, but do we have a test case yet that will tell us if it is resolved? Put another way, how will we know it works after your change?

I started working on a test setup for this yesterday, but I didn't finish it yet.

It seems likely that what you've described may be the case. When we were fixing the compatibility between catkin and rosbuild when I first joined WG, I remember that we always approached the issues with the mind set that we would not change rosbuild if at all possible. So we may have originally arrived at this solution we have by avoiding the consideration that we could change rosbuild to fix the issue.

@gerkey
Copy link
Contributor

gerkey commented May 14, 2015

It turns out that I couldn't fix the behavior in rosbuild, because rospack doesn't offer a subcommand to give the entire libs string. The fix, which is pretty minor, needs to go in rospack: ros/rospack#48. I'm working on a test case for it.

@dirk-thomas
Copy link
Member

While the patch from ros/ros#87 fixes building with rosbuild it does not address the problem that the generated pkg-config files are not usable with newer version of ld. Since older versions of pkg-config do not keep the order intact this requires a conditional generation of the .pc files. If pkg-config is new enough that it does not break the library order the .pc file should be generated without the :.

If someone would like to work on a PR for that it would be highly appreciated.

@dirk-thomas dirk-thomas reopened this Sep 16, 2015
timn added a commit to fawkesrobotics/fawkes that referenced this issue Oct 21, 2015
ROS Jade creates library link commands of the form
"-l:/full/path/to/lib.so" in the generated pkg-config files. This used
to work with older ld version, but does not longer with recent versions.
Therefore, adopt a workaround discussed for rosbuild by removing "-l:",
as ld allows to simply name shared libraries on the command line.

Confer:
ros/catkin#694
ros/ros#87
@dirk-thomas
Copy link
Member

I will set the milestone to untargeted for this since the maintainers will likely not work on this in the foreseeable future.

@gerkey
Copy link
Contributor

gerkey commented Feb 24, 2016

We ran into this problem again while writing some examples of building against ROS packages directly with pkg-config. So far, we've been working around it with the $(subst) function in Makefiles: https://github.com/gerkey/ros1_external_use/blob/master/ros1_comms/Makefile#L9-L14.

evenator pushed a commit to evenator/ros that referenced this issue Mar 2, 2016
@NikolausDemmel
Copy link
Contributor

I was also just surprised by this. We are using pkgconfig to depend on some ROS packages in a qmake project. The following works fine on trusty / indigo, but fails on xenial / kinetic:

CONFIG += link_pkgconfig

PKGCONFIG += \
    roscpp \
    tf \
    geometry_msgs

We might have to resort to manually calling pkg-config and using somehting like $(subst) as @gerkey does in the external use Makefile 😐

@dirk-thomas
Copy link
Member

@NikolausDemmel I agree with you that having to work around this by modifying the output of pkg-config is pretty bad. I suggested an alternative above which wouldn't require such hacks but @gerkey decided to implement the workaround in rosbuild since it was less effort.

Imo it would be better to actually fix what pkg-config outputs. Since time has moved forward it might be the case that we can rely on all platforms targeted by Kinetic to have new enough ld and libtool which don't have the reordering bug anymore. That would make implementing my proposal much easier. It would be great if you could try to make a PR for this since you have a use case for it.

@NikolausDemmel
Copy link
Contributor

Thanks for the update Dirk.

It would be great if you could try to make a PR for this since you have a use case for it.

I would love to, but I am not sure when I will have the time (which usually means never, at least until priorities shift, unfortunately...). We are still mostly using trusty / indigo, but I recently needed to release one package for xenial/kinetic for one xenial machine... For that we have simply modified the pc file after install as a quick and dirty workaround.

@mintar
Copy link

mintar commented Jul 7, 2017

I just ran into this problem again while trying to link to some ROS libraries directly via pkg-config from a non-ROS project. Can we assume that all platforms running Kinetic now have ld versions that don't understand the -l: syntax any more? For Ubuntu, that's everything from 14.10 Utopic upwards.

If so, couldn't we just merge @kartikmohta 's proposal (slightly modified) into the kinetic-devel branch?

diff --git a/cmake/catkin_package.cmake b/cmake/catkin_package.cmake
index 07613ce..f2a13ad 100644
--- a/cmake/catkin_package.cmake
+++ b/cmake/catkin_package.cmake
@@ -272,7 +272,8 @@ function(_catkin_package)
     if(IS_ABSOLUTE ${library})
       get_filename_component(suffix ${library} EXT)
       if(NOT "${suffix}" STREQUAL "${CMAKE_STATIC_LIBRARY_SUFFIX}")
-        set(library "-l:${library}")
+        get_filename_component(libname ${library} NAME)
+        set(library "-l${libname}")
       endif()
     else()
       set(library "-l${library}")

Look at it this way: On systems with the new version of ld, we cannot break anything (e.g., rosbuild) because it's already broken - any attempt to use the .pc files in question will result in linker errors.

I haven't tested the proposal above yet; my quick and dirty workaround was to simply replace all occurrences of -l: with -l in the installed .pc files, while keeping the absolute paths. It made the linker errors go away, but I still need to do some testing.

@mintar
Copy link

mintar commented Jul 7, 2017

Or, alternatively, follow @gerkey 's example and keep the absolute paths without any -l prefix:

diff --git a/cmake/catkin_package.cmake b/cmake/catkin_package.cmake
index 07613ce..f2a13ad 100644
--- a/cmake/catkin_package.cmake
+++ b/cmake/catkin_package.cmake
@@ -272,7 +272,8 @@ function(_catkin_package)
     if(IS_ABSOLUTE ${library})
       get_filename_component(suffix ${library} EXT)
       if(NOT "${suffix}" STREQUAL "${CMAKE_STATIC_LIBRARY_SUFFIX}")
-        set(library "-l:${library}")
+        set(library "${library}")
       endif()
     else()
       set(library "-l${library}")

@dirk-thomas
Copy link
Member

It could very well be that newer platforms won't be affected anymore and it would be safe to use either of the proposed changes. This will require some significant testing though (including reproduce the original problem and than confirm that it has been resolved in newer versions across all targeted platforms). I will likely not have the bandwidth to do that so if anybody would like to spend that effort that would be great.

@mintar
Copy link

mintar commented Jul 19, 2017

Ok, I've set up a small repo to demonstrate the original problem and to confirm that it affects all supported platforms from Xenial onwards:

https://github.com/mintar/catkin_pkgconfig_tests

Quick summary:

The catkin-generated *.pc files are broken on all platforms except trusty:

compiles? compiles with hack?
trusty/indigo
xenial/kinetic
yakkety/lunar
zesty/lunar
fedora23/kinetic
fedora24/kinetic
fedora26/lunar
jessie/kinetic
stretch/lunar
  • "compiles" means whether the test_project in this repo compiles and runs with the original *.pc files.
  • "compiles with hack" means whether the test_project compiles and runs after running the following command:
sudo find /opt/ros/$ROS_DISTRO -name '*.pc' -exec sed -i -e 's/-l://g' {} \;

If you have any suggestions what else I should test, please let me know or submit a PR to my test repo.

Is there a way to keep the current behavior for Trusty and implement a quick fix like the one from my previous comment for all other platforms? Remember, it's already broken on those platforms, so doing anything is better than doing nothing and hoping a future version of ld will restore the -l:/full/path syntax.

@dirk-thomas
Copy link
Member

If you have any suggestions what else I should test, please let me know

The interesting question is on which platforms the -l: prefix is needed in order to prevent the reordering to happen (since that was what the prefix was working around in the first place). If that workaround is not required anymore on newer platforms the prefix can just be removed as your proposed. But if the reordering still happens on some platforms I think a solutions which considers the exact versions of the tools involved might be necessary.

Is there a way to keep the current behavior for Trusty

A potential fix could only be applied to the kinetic-devel branch and would not affect Indigo and Jade.

@mintar
Copy link

mintar commented Jul 19, 2017

The interesting question is on which platforms the -l: prefix is needed in order to prevent the reordering to happen

Ok, the answer to that question is easy: none of the platforms except perhaps trusty need the -l: prefix. On the contrary: all of them require the prefix to be removed.

Merging the fix into kinetic-devel cannot hurt and will fix at least 99% of the use cases (right now, 100% are broken). If we later find that some reordering takes place, we'll have a concrete example that needs a more elaborate fix. As @gerkey wrote 2 years ago, he hasn't seen any reordering in pkg-config 0.28 (Utopic and later), so we should be good.

Note that we don't change Trusty, where the current version is tested and tried and we know it works. We change the other platforms, where we know that the current version is broken.

@dirk-thomas
Copy link
Member

That sounds indeed easy. If we only target Kinetic and newer we only have to care for Ubuntu Xenial (and higher), Debian Jessie and Fedora 23 (and higher). All of those ship with at least version 0.28 of pkg-config. Under the assumption that neither of them has the reordering bug anymore it should be safe to remove the workaround. I created #879 with the simple patch. It would be great if you could confirm that it works for your use case.

@dirk-thomas dirk-thomas removed this from the untargeted milestone Jul 19, 2017
@mintar
Copy link

mintar commented Jul 19, 2017

I've just tested this and it works for my use case!

@dirk-thomas
Copy link
Member

Thank you for your effort on this long standing ticket.

@NikolausDemmel
Copy link
Contributor

Thanks @mintar and @dirk-thomas !

@mintar
Copy link

mintar commented Oct 10, 2017

Any ETA when this will land in kinetic? If I see it correctly, there was a tag containing this fix created in July (0.7.7), but ros-kinetic-catkin is still on 0.7.6.

@dirk-thomas
Copy link
Member

The new version was released only into Lunar during the last release round. This will certainly be released into Kinetic when the next round of ROS core packages are released into Kinetic. I would estimate that this happens in the next 2-3 weeks - but don't quote me on this 😉

cwecht pushed a commit to cwecht/catkin that referenced this issue Mar 20, 2018
Allow same topics to be used in different bags (a previously supported use case).
Remove unused variable `current_bag_sensor_topics`.
Touch up flag descriptions.

Fixes ros#693.

pair=@gaschler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants