Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User dependent gazebo installation #2782

Closed
stefanbuettner opened this issue Jul 9, 2020 · 17 comments · Fixed by #2879
Closed

User dependent gazebo installation #2782

stefanbuettner opened this issue Jul 9, 2020 · 17 comments · Fixed by #2879

Comments

@stefanbuettner
Copy link
Contributor

Hey there,

I want to provide Gazebo via the conan package manager to some colleagues. Building all the components worked like a charm using conan's cmake build helpers and git tool. Also gazebo starts without problems for my user and I can compile plugins against the Gazebo C++ API using cmake.

However, if I upload the built package to our Artifactory server, and another user installs the pre-built binaries, Gazebo doesn't start and find_package cannot find Gazebo anymore.

We were able to start Gazebo under the other user by providing the environment variable GAZEBO_RESOURCE_PATH (see PX4/PX4-Avoidance#177).

But in general it seems that user-specific paths are compiled into the generated packages, both in cmake config files and so-files.
Is this the case? And if so, can Gazebo and its dependencies be compiled in a user-independent manner?

I could not find cmake options in any of the packages. I also didn't find any pointers in the documentation or READMEs.

Cheers,
Stefan

@traversaro
Copy link
Collaborator

But in general it seems that user-specific paths are compiled into the generated packages, both in cmake config files and so-files.
Is this the case? And if so, can Gazebo and its dependencies be compiled in a user-independent manner?

At least for the CMake aspect, this problem is also tracked in #2755, even if I do not think that anyone is actively working on it.

For the paths that contained the hardcoded installation prefix in them, in general the suggestion is always to source the <prefix>/share/gazebo/setup.sh script before running Gazebo, see http://gazebosim.org/tutorials?tut=components&cat=get_started#EnvironmentVariables . The main problem is that the <prefix>/share/gazebo/setup.sh script itself at the moment is not relocatable. However, fixing that is relatively simple, it is necessary to refactor it to compute the installation prefix relatively to the location of the setup.sh path. I have a similar script working in this way in https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/blob/master/scripts/setup-gazebo.sh .

@stefanbuettner
Copy link
Contributor Author

Cool, thanks. I'll have a look. Would such a change be merge request worthy?

@traversaro
Copy link
Collaborator

Would such a change be merge request worthy?

In my opinion definitely, and I would be happy to provide feedback as I would be happy to not have to mantain my own scripts for that. However let's see what the Gazebo maintainers think about this.

Unrelated comment: do you have any plan to open source your Gazebo conan recipes?

@stefanbuettner
Copy link
Contributor Author

No plans so far. They're also rather rudimentary at this point. Just to compile stuff. No options. But could serve as a starting point for the community.

Pro added a commit to Pro/gazebo that referenced this issue Nov 17, 2020
The CMake config used absolute paths which doesn't allow to move the installed cmake files. This is the case, e.g., if you create a conan package and distribute the binaries to other users.

Fixes: gazebosim#2755
Fixes: gazebosim#2782
Pro added a commit to Pro/gazebo that referenced this issue Nov 17, 2020
The CMake config used absolute paths which doesn't allow to move the installed cmake files. This is the case, e.g., if you create a conan package and distribute the binaries to other users.

Fixes: gazebosim#2755
Fixes: gazebosim#2782
Pro added a commit to Pro/gazebo that referenced this issue Nov 17, 2020
The CMake config used absolute paths which doesn't allow to move the installed cmake files. This is the case, e.g., if you create a conan package and distribute the binaries to other users.

Fixes: gazebosim#2755
Fixes: gazebosim#2782
@traversaro
Copy link
Collaborator

I had forgot about this issue, but as it is related to conan I guess @joxoby could also be interested in this.

Pro added a commit to Pro/gazebo that referenced this issue Nov 17, 2020
The CMake config used absolute paths which doesn't allow to move the installed cmake files. This is the case, e.g., if you create a conan package and distribute the binaries to other users.

Fixes: gazebosim#2755
Fixes: gazebosim#2782
@joxoby
Copy link

joxoby commented Nov 17, 2020

However, if I upload the built package to our Artifactory server, and another user installs the pre-built binaries, Gazebo doesn't start and find_package cannot find Gazebo anymore.

This sounds confusing to me. Conan itself will generate the required Find<package>.cmake files that will point to the local Conan packages. This step will normally happen when you call conan_cmake_run inside your CMake scripts that want to include Gazebo, so it shouldn't really matter whether the Conan package was built locally or downloaded from Artifactory.

@stefanbuettner
Copy link
Contributor Author

This is only true, if you use the cmake_find_package generator which generates the FindXXX.config scripts. But I don't see a reason to use them when the package itself provides CMake config scripts.
So for all packages which provide CMake config scripts we use the cmake_paths generator which only points the CMAKE_PREFIX_PATH to the appropriate locations.

@joxoby
Copy link

joxoby commented Nov 18, 2020

@traversaro
Copy link
Collaborator

traversaro commented Nov 18, 2020

Normally, you wouldn't package the *.cmake scripts: https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#kb-h016-cmake-modules-config-files

More info: https://blog.conan.io/2018/06/11/Transparent-CMake-Integration.html

I am probably missing something, but it seems that cmake_find_package generator, can't work for all libraries, i.e. as the say in the blog post:

If you are consuming packages and you have high restrictions to change your CMakeLists.txt, probably the cmake_paths is the best choice.

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

Furthermore, how can this work with complex libraries such as Qt that also need to install CMake macros?

@joxoby
Copy link

joxoby commented Nov 18, 2020

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

Furthermore, how can this work with complex libraries such as Qt that also need to install CMake macros?

Great question. To be clear, the hook restriction is trying to avoid that you don't package the pacakge's own Find<package>.cmake scripts. If the package includes cmake scripts to find other packages, like ignition-cmake (and you want to upload it to conan-center-index), you have to add an exception to that hook check. Example. Anyways, this is somewhat tangential to the main issue.

@stefanbuettner, if you'd like, you can try making a PR to upload your recipes to conan-center-index. Besides making them available to other people, the main Conan reviewers will provide you with great feedback on best practices.

@traversaro
Copy link
Collaborator

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

@joxoby
Copy link

joxoby commented Nov 19, 2020

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

The blogpost can be misleading in that specific example. For any Conan package, you can set the name that the generator will use to create the Find<name>.cmake script. https://github.com/conan-io/conan-center-index/blob/7d73c4e97052080a712028c082d6da4a78dd7179/recipes/libcurl/all/conanfile.py#L536

@traversaro
Copy link
Collaborator

traversaro commented Nov 19, 2020

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

The blogpost can be misleading in that specific example. For any Conan package, you can set the name that the generator will use to create the Find<name>.cmake script. https://github.com/conan-io/conan-center-index/blob/7d73c4e97052080a712028c082d6da4a78dd7179/recipes/libcurl/all/conanfile.py#L536

Cool, that is much more clear, I just have two more doubts:

  • Gazebo's CMake config files do not use imported targets, but just CMake variables, can you generate also those with the cmake_find_package generator? This is similar to the problem of packages that install CMake macros (that are not CMake's Find**.cmake scripts, but rather CMake functions such as qt5_generate_moc.
  • I wonder why conan generates Find<package>.cmake files, that in CMake are meant to inspect the system to find a dependency, instead of generating <package>-config.cmake files, that are the ones that are generated by packages to make sure that that they are found by downstream projects, but I suspect the reason is that if you call find_package without specific options Find<package>.cmake have the precedence over <package>-config.cmake .

In any case, a personal note: even if the approach of trying to avoid build system-specific formats is quite interesting, replicating all the information of the imported target in manually maintained (and package manager-specif) python scripts (see for example https://github.com/conan-io/conan-center-index/pull/3215/files#diff-68a42dbaa991102b8ab0fa45fa4ed20dae5b48e127822865644e071974f6be07R126) may be error-prone in some cases. A possible solution for this would be a real cross-build-system format to express C++ library information, instead of relying on CMake config files and .pc files, and on that there is an interesting discussion between CMake and meson developers in https://gitlab.kitware.com/cmake/cmake/-/issues/20106 and in the issues of https://github.com/mwoehlke/cps .

@traversaro
Copy link
Collaborator

A possible solution for this would be a real cross-build-system format to express C++ library information, instead of relying on CMake config files and .pc files, and on that there is an interesting discussion between CMake and meson developers in https://gitlab.kitware.com/cmake/cmake/-/issues/20106 and in the issues of https://github.com/mwoehlke/cps .

Indeed, I found a related conan discussion in conan-io/conan#4878 .

j-rivero pushed a commit that referenced this issue Mar 15, 2021
* fix: Use proper cmake package config for relocatable CMake config
* Fix relative path in pkg-config files

The CMake config used absolute paths which doesn't allow to move the installed cmake files. This is the case, e.g., if you create a conan package and distribute the binaries to other users.

Fixes: #2755
Fixes: #2782

Co-authored-by: Steve Peters <scpeters@openrobotics.org>
@traversaro
Copy link
Collaborator

#2879 makes CMake config packages relocatable, but some other files in Gazebo (in particular the setup.sh script) is not relocatable, not sure if it make sense to open a new issue for tracking that or it make sense to re-open this one.

@traversaro
Copy link
Collaborator

fyi @mboisson

@scpeters
Copy link
Member

scpeters commented Aug 6, 2021

#2879 makes CMake config packages relocatable, but some other files in Gazebo (in particular the setup.sh script) is not relocatable, not sure if it make sense to open a new issue for tracking that or it make sense to re-open this one.

I've opened #3056 to track the relocatability of setup.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants