Gazebo ROS plugins failing to load on MacOS

[sonelu]

Environment:

MacOS Catalina 10.15.7
Conda: 4.9.2
Python: 3.8.8

Installed ros-noetic-desktop-full in it's own environment.

While now Gazebo starts, no ROS topics and services are published:

(robostack) Alex3-2:~ Alex$ rostopic list
/rosout
/rosout_agg
(robostack) Alex3-2:~ Alex$ rosservice list
/rosout/get_loggers
/rosout/set_logger_level

There is no error message issued even if Gazebo is started with --verbose.

Fortunately there is an error message that is produced if I start Gazebo with rosrun, instead:

(robostack) Alex3-2:~ Alex$ rosrun gazebo_ros gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Err] [Plugin.hh:178] Failed to load plugin /Users/Alex/anaconda3/envs/robostack/lib/libgazebo_ros_api_plugin.dylib: dlopen(/Users/Alex/anaconda3/envs/robostack/lib/libgazebo_ros_api_plugin.dylib, 9): Library not loaded: @rpath/libprotobuf.24.dylib
  Referenced from: /Users/Alex/anaconda3/envs/robostack/lib/libgazebo_ros_api_plugin.dylib
  Reason: image not found
[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.0.30
[Msg] Loading world file [/Users/Alex/anaconda3/envs/robostack/share/gazebo-11/worlds/empty.world]

It looks like libgazebo_ros_api_plugin.dylib is linked against the version 24 of the libprotobuf:

(robostack) Alex3-2:~ Alex$ otool -L /Users/Alex/anaconda3/envs/robostack/lib/libgazebo_ros_api_plugin.dylib
/Users/Alex/anaconda3/envs/robostack/lib/libgazebo_ros_api_plugin.dylib:
	@rpath/libgazebo_ros_api_plugin.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libgazebo_physics.11.dylib (compatibility version 11.0.0, current version 11.3.0)
	@rpath/libgazebo_transport.11.dylib (compatibility version 11.0.0, current version 11.3.0)
	@rpath/libgazebo_msgs.11.dylib (compatibility version 11.0.0, current version 11.3.0)
	@rpath/libgazebo_common.11.dylib (compatibility version 11.0.0, current version 11.3.0)
	@rpath/libprotobuf.24.dylib (compatibility version 25.0.0, current version 25.0.0)
	@rpath/libsdformat9.9.dylib (compatibility version 9.0.0, current version 9.3.0)
	@rpath/libOgreBites.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreHLMS.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreMeshLodGenerator.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreOverlay.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgrePaging.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreProperty.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreRTShaderSystem.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreTerrain.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreVolume.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libOgreMain.1.10.12.dylib (compatibility version 1.10.12, current version 1.10.12)
	@rpath/libroslib.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libroscpp.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/librosconsole.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libdynamic_reconfigure_config_init_mutex.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libroscpp_serialization.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/librostime.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libboost_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libtinyxml.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

While the version that is deployed with the conda installation is 26:

(robostack) Alex3-2:~ Alex$ ls -al /Users/Alex/anaconda3/envs/robostack/lib/libprotobuf*
-rwxrwxr-x  2 Alex  staff   712528 26 Feb 16:47 /Users/Alex/anaconda3/envs/robostack/lib/libprotobuf-lite.26.dylib
lrwxr-xr-x  1 Alex  staff       25  1 Mar 16:17 /Users/Alex/anaconda3/envs/robostack/lib/libprotobuf-lite.dylib -> libprotobuf-lite.26.dylib
-rwxrwxr-x  2 Alex  staff  2953352 26 Feb 16:47 /Users/Alex/anaconda3/envs/robostack/lib/libprotobuf.26.dylib
lrwxr-xr-x  1 Alex  staff       20  1 Mar 16:17 /Users/Alex/anaconda3/envs/robostack/lib/libprotobuf.dylib -> libprotobuf.26.dylib

[Tobias-Fischer]

Could you please check conda list | grep gazebo?

[sonelu]

(robostack) Alex3-2:cmake Alex$ conda list | grep gazebo
gazebo                    11.3.0               hc02bdab_8    conda-forge
ros-noetic-gazebo-dev     2.9.1            py38hba8b470_3    robostack
ros-noetic-gazebo-msgs    2.9.1            py38h143cb38_3    robostack
ros-noetic-gazebo-plugins 2.9.1            py38h143cb38_3    robostack
ros-noetic-gazebo-ros     2.9.1            py38h143cb38_3    robostack
ros-noetic-gazebo-ros-control 2.9.1            py38h143cb38_3    robostack
ros-noetic-gazebo-ros-pkgs 2.9.1            py38h143cb38_3    robostack

[traversaro]

Really Short term comment: @sonelu Probably a workaround would be to just get a version of Gazebo built against protobuf 3.13 (or 3.14, not sure to which version the .24 soname corresponds). However, it does not exist at the moment a build of Gazebo that links protobuf 3.13 but also solves #55, so unfortunatly this workaround is of no use in macOS.

Short term comment: @Tobias-Fischer @wolfv Probably we need to re-build everything to use protobuf 3.15 ?

Medium term comment: @wolfv @Tobias-Fischer The main problem here is that for Gazebo uses Protobuf in its public headers, so any ABI constraint between Gazebo and protobuf is also propagated to the software that uses Gazebo (including Gazebo plugins). Should we add a - {{ pin_subpackage('libprotobuf', max_pin='x.x') }} (or similar, not sure if it works) in gazebo-feedstock meta.yaml ?

[wolfv]

I think what we might have to sync with the conda-forge pinnings... I've been meaning to write a script for that. I cannot get to this today but I can try tomorrow morning, and then we can rebuild all of ROS with the latest pinnings. What do you think @Tobias-Fischer @traversaro ?

[wolfv]

also thanks @sonelu for continuing to test. This feedback is very valuable!

[traversaro]

I think what we might have to sync with the conda-forge pinnings... I've been meaning to write a script for that. I cannot get to this today but I can try tomorrow morning, and then we can rebuild all of ROS with the latest pinnings. What do you think @Tobias-Fischer @traversaro ?

Sounds great!

[sonelu]

also thanks @sonelu for continuing to test. This feedback is very valuable!

@wolfv you're more than welcome. I really appreciate the work you guys are doing. As a Mac user I've been struggling for a while with ROS support and mostly I ended up using VMs in VirtualBox. But it's not ideal and I'd rather use a native environment.

[Tobias-Fischer]

Ha, a full rebuild will be painful but it's the way to go :)

[Tobias-Fischer]

Hi @sonelu - this should be fixed now as everything was rebuilt. Could you please check and let us know?

[sonelu]

I conform that now Gazebo starts and the ROS plugins and loaded and running (ex. topics and services are advertised).

I've created a separate issue (#86) as I'm getting a "Segmentation fault: 11" when spawning a URDF but that seems to be traced down to tf2_py package.