How to disable --only-pkg-with-deps?

[mkoval]

I used --only-pkg-with-deps to rebuild a single package in a Catkin workspace. After doing so, future invocations of catkin_make (with no options) do not force a configure. As a result, they build the subset of packages that were selected by the flag.

The only solution I've found to work is to delete the entire build directory. Surprisingly, the --force-cmake flag also doesn't force a reconfigure (this seems like a bug?).

This definitely happens in version 0.6.11 on Ubuntu 12.04 and 14.04.

[dirk-thomas]

--only-pkg-with-deps equates to set the CMake variable CATKIN_WHITELIST_PACKAGES. Once that is set it will be in the CMakeCache.txt file until being redefined.

Passing --force-make does invoke the CMake configuration step again. But it will keep using the variable from the cache.

The script has no knowledge if the user had passed -CATKIN_WHITELIST_PACKAGES=... before (and it should keep using it) or if the user passed --only-pkg-with-deps before and now intends it to be "reset". Therefore I don't see a good approach to change the current behavior.

[wjwwood]

I can confirm the behavior you describe is happening, but it is not because CMake is not running. I did this:

$ mkdir -p /tmp/test_catkin_ws/src

$ cd /tmp/test_catkin_ws/src

$ git clone https://github.com/ros/catkin.git
...

$ cat catkin/package.xml | grep version
<?xml version="1.0"?>
  <version>0.6.11</version>
  <depend version_gte="0.2.2">python-catkin-pkg</depend>

$ catkin_create_pkg --rosdistro indigo foo
...

$ catkin_create_pkg --rosdistro indigo bar foo
...

$ cd ..

$ ./src/catkin/bin/catkin_make
Base path: /tmp/test_catkin_ws
Source space: /tmp/test_catkin_ws/src
Build space: /tmp/test_catkin_ws/build
Devel space: /tmp/test_catkin_ws/devel
Install space: /tmp/test_catkin_ws/install
Creating symlink "/tmp/test_catkin_ws/src/CMakeLists.txt" pointing to "catkin/cmake/toplevel.cmake"
####
#### Running command: "cmake /tmp/test_catkin_ws/src -DCATKIN_DEVEL_PREFIX=/tmp/test_catkin_ws/devel -DCMAKE_INSTALL_PREFIX=/tmp/test_catkin_ws/install -G Unix Makefiles" in "/tmp/test_catkin_ws/build"
####
-- The C compiler identification is AppleClang 6.0.0.6000056
-- The CXX compiler identification is AppleClang 6.0.0.6000056
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Using CATKIN_DEVEL_PREFIX: /tmp/test_catkin_ws/devel
-- Using CMAKE_PREFIX_PATH:
-- Found PythonInterp: /usr/bin/python (found version "2.7.5")
-- Using PYTHON_EXECUTABLE: /usr/bin/python
-- Using default Python package layout
-- Found PY_em: /Library/Python/2.7/site-packages/em.pyc
-- Using empy: /Library/Python/2.7/site-packages/em.pyc
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /tmp/test_catkin_ws/build/test_results
-- Found gtest: gtests will be built
-- Using Python nosetests: /usr/local/bin/nosetests-2.7
-- catkin 0.6.11
CMake Warning (dev) at catkin/CMakeLists.txt:5 (project):
  Policy CMP0048 is not set: project() command manages VERSION variables.
  Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The following variable(s) would be set to empty:

    catkin_VERSION
This warning is for project developers.  Use -Wno-dev to suppress it.

-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 2 packages in topological order:
-- ~~  - foo
-- ~~  - bar
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'foo'
-- ==> add_subdirectory(foo)
-- +++ processing catkin package: 'bar'
-- ==> add_subdirectory(bar)
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test_catkin_ws/build
####
#### Running command: "make -j4 -l4" in "/tmp/test_catkin_ws/build"
####

$ ./src/catkin/bin/catkin_make --only-pkg-with-deps foo
Base path: /tmp/test_catkin_ws
Source space: /tmp/test_catkin_ws/src
Build space: /tmp/test_catkin_ws/build
Devel space: /tmp/test_catkin_ws/devel
Install space: /tmp/test_catkin_ws/install
Whitelisted packages: catkin, foo
####
#### Running command: "cmake /tmp/test_catkin_ws/src -DCATKIN_DEVEL_PREFIX=/tmp/test_catkin_ws/devel -DCMAKE_INSTALL_PREFIX=/tmp/test_catkin_ws/install -DCATKIN_WHITELIST_PACKAGES=catkin;foo -G Unix Makefiles" in "/tmp/test_catkin_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /tmp/test_catkin_ws/devel
-- Using CMAKE_PREFIX_PATH:
-- Using PYTHON_EXECUTABLE: /usr/bin/python
-- Using default Python package layout
-- Using empy: /Library/Python/2.7/site-packages/em.pyc
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /tmp/test_catkin_ws/build/test_results
-- Found gtest: gtests will be built
-- Using Python nosetests: /usr/local/bin/nosetests-2.7
-- catkin 0.6.11
CMake Warning (dev) at catkin/CMakeLists.txt:5 (project):
  Policy CMP0048 is not set: project() command manages VERSION variables.
  Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The following variable(s) would be set to empty:

    catkin_VERSION
This warning is for project developers.  Use -Wno-dev to suppress it.

-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 1 packages in topological order:
-- ~~  - foo
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'foo'
-- ==> add_subdirectory(foo)
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test_catkin_ws/build
####
#### Running command: "make -j4 -l4" in "/tmp/test_catkin_ws/build"
####

At this point I can see why you would expect just catkin_make to build foo and bar, but it doesn't even though it does run CMake:

./src/catkin/bin/catkin_make
Base path: /tmp/test_catkin_ws
Source space: /tmp/test_catkin_ws/src
Build space: /tmp/test_catkin_ws/build
Devel space: /tmp/test_catkin_ws/devel
Install space: /tmp/test_catkin_ws/install
####
#### Running command: "cmake /tmp/test_catkin_ws/src -DCATKIN_DEVEL_PREFIX=/tmp/test_catkin_ws/devel -DCMAKE_INSTALL_PREFIX=/tmp/test_catkin_ws/install -G Unix Makefiles" in "/tmp/test_catkin_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /tmp/test_catkin_ws/devel
-- Using CMAKE_PREFIX_PATH:
-- Using PYTHON_EXECUTABLE: /usr/bin/python
-- Using default Python package layout
-- Using empy: /Library/Python/2.7/site-packages/em.pyc
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /tmp/test_catkin_ws/build/test_results
-- Found gtest: gtests will be built
-- Using Python nosetests: /usr/local/bin/nosetests-2.7
-- catkin 0.6.11
CMake Warning (dev) at catkin/CMakeLists.txt:5 (project):
  Policy CMP0048 is not set: project() command manages VERSION variables.
  Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The following variable(s) would be set to empty:

    catkin_VERSION
This warning is for project developers.  Use -Wno-dev to suppress it.

-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 1 packages in topological order:
-- ~~  - foo
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'foo'
-- ==> add_subdirectory(foo)
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test_catkin_ws/build
####
#### Running command: "make -j4 -l4" in "/tmp/test_catkin_ws/build"
####

A second invocation shows that CMake doesn't run everytime:

$ ./src/catkin/bin/catkin_make
Base path: /tmp/test_catkin_ws
Source space: /tmp/test_catkin_ws/src
Build space: /tmp/test_catkin_ws/build
Devel space: /tmp/test_catkin_ws/devel
Install space: /tmp/test_catkin_ws/install
####
#### Running command: "make cmake_check_build_system" in "/tmp/test_catkin_ws/build"
####
####
#### Running command: "make -j4 -l4" in "/tmp/test_catkin_ws/build"
####

And though --force-cmake runs CMake again, it doesn't get it back in the list:

$ ./src/catkin/bin/catkin_make --force-cmake
Base path: /tmp/test_catkin_ws
Source space: /tmp/test_catkin_ws/src
Build space: /tmp/test_catkin_ws/build
Devel space: /tmp/test_catkin_ws/devel
Install space: /tmp/test_catkin_ws/install
####
#### Running command: "cmake /tmp/test_catkin_ws/src -DCATKIN_DEVEL_PREFIX=/tmp/test_catkin_ws/devel -DCMAKE_INSTALL_PREFIX=/tmp/test_catkin_ws/install -G Unix Makefiles" in "/tmp/test_catkin_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /tmp/test_catkin_ws/devel
-- Using CMAKE_PREFIX_PATH:
-- Using PYTHON_EXECUTABLE: /usr/bin/python
-- Using default Python package layout
-- Using empy: /Library/Python/2.7/site-packages/em.pyc
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /tmp/test_catkin_ws/build/test_results
-- Found gtest: gtests will be built
-- Using Python nosetests: /usr/local/bin/nosetests-2.7
-- catkin 0.6.11
CMake Warning (dev) at catkin/CMakeLists.txt:5 (project):
  Policy CMP0048 is not set: project() command manages VERSION variables.
  Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The following variable(s) would be set to empty:

    catkin_VERSION
This warning is for project developers.  Use -Wno-dev to suppress it.

-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 1 packages in topological order:
-- ~~  - foo
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'foo'
-- ==> add_subdirectory(foo)
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test_catkin_ws/build
####
#### Running command: "make -j4 -l4" in "/tmp/test_catkin_ws/build"
####

This is because the "whitelisted" packages are stored in the CMake cache. I'm not sure how we could change the behavior to be more obvious.

Not a solution to this, but as a matter of workflow I would recommend you use one of the isolated tools if you want to selectively build things like this or use more than one workspace.

IIRC, we added the --build-pkg-with-deps out of popular demand, but this is just one if its drawbacks.

[wjwwood]

Doh, what @dirk-thomas said. I should check before posting...

[mkoval]

@dirk-thomas Thanks for the explanation! That makes sense. I also don't see a solution with the current architecture.

@wjwwood I am using --build-pkg-with-deps because Catkin's configure times are so atrocious. It's very tedious to debug issues in a package's CMakeLists.txt file because every edit forces the entire workspace to (very slowly) re-configure. What workflow do you suggest for this?

Regarding this issue: It would help to document this in the help text for --build-pkg-with-deps. Its current help text, which I've included below, does not suggest that the option is stateful:

Only consider the specific packages and their recursive dependencies and ignore all other packages in the workspace

[wjwwood]

@mkoval catkin_make_isolated is slower than catkin_make because it must run CMake and then make on each package in turn without parallelization amongst packages. However, for partial rebuilds it can be much faster with options like --only-pkg-with-deps or --from-pkg.

We've also been working on a new tool under the umbrella of https://github.com/catkin/catkin_tools called catkin build. This tool is more promising since it allows you to build parts of the package tree and later other parts and reuse parts of the package tree already built. It also has the ability to build packages which do not depend on each other in parallel. It's pretty sweet, but there is still a list of problems with it.

Here are the docs:

http://catkin-tools.readthedocs.org/en/latest/

And here is the issue which is tracking its outstanding issues, which we are addressing as we have free time, which lately has been slim:

catkin/catkin_tools#90

However, it may already work for you use case. I would say "Here be dragons" with respect to catkin build, but if you can avoid them it is the most powerful tool we have.

[wjwwood]

Also, w.r.t the documentation for catkin_make --build-pkg-with-deps I agree that the docs should be updated, I don't support there is a chance we could solicit a pull request from you? 😄

It seems adding -DCATKIN_WHITELIST_PACKAGES="" to the next catkin_make call will clear the condition and include all packages again, though I haven't considered all of the ramifications of this approach.

[dirk-thomas]

Addressed by #707.