Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace use of tbb::task with oneapi::tbb::task_group, where available #3146

Closed

Conversation

alexdewar
Copy link
Contributor

As of TBB 2021.01, tbb::task has been removed: #2867.

Accordingly, I have implemented a wrapper class, TaskGroup, which makes use of oneapi::tbb::task_group where it is available but falls back on tbb::task otherwise. Note that the tbb::task_group class has been around for a while now (predating the rebranding as oneTBB), so we could be less conservative and essentially use this code path for some older versions of TBB which still have tbb::task.

Resolves #2867.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
As tbb::task has been removed in oneTBB 2021.01, we need to replace its use with oneapi::tbb::task_group. Define a wrapper so that tbb::task_group is used for newer versions of oneTBB.

Fixes gazebosim#2867.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
@alexdewar
Copy link
Contributor Author

I forgot to note that I have introduced a small functional change: classes owning a TaskGroup will now block until its child tasks have finished execution in their destructors. We could presumably change this if desired though.

Copy link
Member

@scpeters scpeters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this! One requirement that we have for gazebo11 is to preserve API and ABI for the packages already released on 18.04 and 20.04. I think we can do this with #ifdef macros, which hopefully won't be too painful. libtbb-dev has the following versions in these releases:

I think we should aim for keeping the abi-checker CI job happy while being able to change to the tbb dependency in the homebrew formula for gazebo11.

@@ -0,0 +1,84 @@
/*
* Copyright (C) 2021 Alex Dewar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to assign copyright to Open Source Robotics Foundation but add your name to the AUTHORS file and in a Changelog entry?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you ❤️

@scpeters
Copy link
Member

It looks like the INTEGRATION_transport test is seg-faulting:

[ RUN      ] TransportTest.Errors
517: [Msg] Waiting for master.
517: [Msg] Connected to gazebo master @ http://127.0.0.1:11345
517: [Msg] Publicized address: 172.17.0.2
517: [Msg] Loading world file [/var/lib/jenkins/workspace/gazebo-ci-pr_any-ubuntu_auto-amd64-gpu-nvidia/gazebo/worlds/empty.world]
517: [Dbg] [ServerFixture.cc:211] ServerFixture load in 1 seconds, timeout after 600 seconds
517: [Err] [TopicManager.cc:381] EXCEPTION: Attempting to advertise on an existing topic with a conflicting message type
517: 
517: 
517: 
1/2 Test #517: INTEGRATION_transport ............***Exception: SegFault 23.91 sec

scpeters added a commit to gazebo-tooling/release-tools that referenced this pull request Dec 14, 2021
Testing gazebosim/gazebo-classic#3146

Signed-off-by: Steve Peters <scpeters@openrobotics.org>
@scpeters
Copy link
Member

@osrf-jenkins run tests please

@scpeters
Copy link
Member

I updated our macOS CI to run a job with a very new version of tbb, but there are linking errors when building some tests

Build Status https://build.osrfoundation.org/job/gazebo-ci-pr_any-homebrew-amd64/2622/

In file included from /Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/test/integration/transport.cc:21:
In file included from /Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/test/ServerFixture.hh:44:
In file included from /Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/build/gazebo/transport/transport.hh:3:
In file included from /Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/Connection.hh:39:
In file included from /Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/TaskGroup.hh:26:
/usr/local/include/oneapi/tbb/task_group.h:135:9: error: no matching function for call to object of type 'const gazebo::transport::PublishTask'
        std::forward<F>(f)();
        ^~~~~~~~~~~~~~~~~~
/usr/local/include/oneapi/tbb/task_group.h:466:25: note: in instantiation of function template specialization 'tbb::detail::d2::(anonymous namespace)::task_ptr_or_nullptr<const gazebo::transport::PublishTask &>' requested here
        task* res = d2::task_ptr_or_nullptr(m_func);
                        ^
/usr/local/include/oneapi/tbb/task_group.h:480:5: note: in instantiation of member function 'tbb::detail::d1::function_task<gazebo::transport::PublishTask>::execute' requested here
    function_task(F&& f, wait_context& wo, small_object_allocator& alloc)
    ^
/usr/local/include/oneapi/tbb/detail/_small_object_pool.h:63:57: note: in instantiation of member function 'tbb::detail::d1::function_task<gazebo::transport::PublishTask>::function_task' requested here
        auto constructed_object = new(allocated_object) Type(std::forward<Args>(args)...);
                                                        ^
/usr/local/include/oneapi/tbb/task_group.h:531:22: note: in instantiation of function template specialization 'tbb::detail::d1::small_object_allocator::new_object<tbb::detail::d1::function_task<gazebo::transport::PublishTask>, gazebo::transport::PublishTask, tbb::detail::d1::wait_context &, tbb::detail::d1::small_object_allocator &>' requested here
        return alloc.new_object<function_task<typename std::decay<F>::type>>(std::forward<F>(f), m_wait_ctx, alloc);
                     ^
/usr/local/include/oneapi/tbb/task_group.h:607:16: note: in instantiation of function template specialization 'tbb::detail::d1::task_group_base::prepare_task<gazebo::transport::PublishTask>' requested here
        spawn(*prepare_task(std::forward<F>(f)), context());
               ^
/Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/TaskGroup.hh:41:25: note: in instantiation of function template specialization 'tbb::detail::d1::task_group::run<gazebo::transport::PublishTask>' requested here
        this->taskGroup.run(Functor(std::forward<Args>(args)...));
                        ^
/Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/Node.hh:160:33: note: in instantiation of function template specialization 'gazebo::transport::TaskGroup::run<gazebo::transport::PublishTask, boost::shared_ptr<gazebo::transport::Publisher> &, const google::protobuf::Message &>' requested here
                this->taskGroup.run<PublishTask>(pub, _message);
                                ^
/Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/TransportIface.hh:142:13: note: in instantiation of function template specialization 'gazebo::transport::Node::Publish<gazebo::msgs::Scene>' requested here
      node->Publish<M>(_topic, _message);
            ^
/Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/test/integration/transport.cc:172:14: note: in instantiation of function template specialization 'gazebo::transport::publish<gazebo::msgs::Scene>' requested here
  transport::publish<msgs::Scene>("~/scene", msg);
             ^
/Users/jenkins/workspace/gazebo-ci-pr_any-homebrew-amd64/gazebo/gazebo/transport/Node.hh:53:20: note: candidate function not viable: 'this' argument has type 'const gazebo::transport::PublishTask', but method is not marked const
      public: void operator()()
                   ^
1 error generated.
make[3]: *** [test/integration/CMakeFiles/INTEGRATION_transport.dir/transport.cc.o] Error 1
make[2]: *** [test/integration/CMakeFiles/INTEGRATION_transport.dir/all] Error 2

Suggested by @scpeters.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
Suggested by @scpeters.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
In any case, this check wasn't actually halting the build in the case that tbb>=2021. Instead, find_package() is used (as it is if tbb.pc isn't present).

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
@alexdewar
Copy link
Contributor Author

@scpeters Interesting... Could be a const-correctness issue. I'll see if I can replicate it by building with clang on Linux.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
@alexdewar
Copy link
Contributor Author

Thanks for your work on this! One requirement that we have for gazebo11 is to preserve API and ABI for the packages already released on 18.04 and 20.04. I think we can do this with #ifdef macros, which hopefully won't be too painful. libtbb-dev has the following versions in these releases:

I think we should aim for keeping the abi-checker CI job happy while being able to change to the tbb dependency in the homebrew formula for gazebo11.

Done in 49701f4.

The CI is currently failing and this could be a possible problem.
Copy link
Collaborator

@traversaro traversaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments.

set (USE_LEGACY_TBB_TASK OFF)
endif(${TBB_VERSION} VERSION_LESS 2021)
endif()
set(USE_LEGACY_TBB_TASK ${USE_LEGACY_TBB_TASK} CACHE BOOL "Whether to use the deprecated tbb::task class")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick, feel free to ignore: in my experience cache variable vs normal variables is one aspect of CMake that is most confusing for novice users. To improve readability, it may be worth to use a different name for the normal variable name, like USE_LEGACY_TASK_DEFAULT_VALUE.

endif()
set(USE_LEGACY_TBB_TASK ${USE_LEGACY_TBB_TASK} CACHE BOOL "Whether to use the deprecated tbb::task class")
if (USE_LEGACY_TBB_TASK)
add_definitions(-DUSE_LEGACY_TBB_TASK)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This define will be used in public headers, hence it will need to be define also in downstream compilation units. For these reason, it could make sense to use a gazebo-specific name here (like GAZEBO_USE_LEGACY_TBB_TASK), to avoid confusion and the risk of collisions.

endif()
set(USE_LEGACY_TBB_TASK ${USE_LEGACY_TBB_TASK} CACHE BOOL "Whether to use the deprecated tbb::task class")
if (USE_LEGACY_TBB_TASK)
add_definitions(-DUSE_LEGACY_TBB_TASK)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definition needs to be propagated to downstream compilation units as it is used in public headers, but unfortunately gazebo does not use modern style CMake targets so there is no straightforward way to do so. I can check if there is any facility currently use to expose definitions to downstream projects, like a GAZEBO_DEFINITIONS CMake variable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the docs for building downstream projects that use Gazebo (see http://gazebosim.org/tutorials?tut=plugins_hello_world) and there is no GAZEBO_DEFINITIONS or similar. We could try to encode this definition in Gazebo's GAZEBO_CXX_FLAGS, but this is not straightforward. Thinking about it, a possible alternative solution is to remove all the CMake logic related to USE_LEGACY_TBB_TASK, and just define this macro at the C++ preprocessor level, depending on the tbb version. This is the strategy that we used for supporting Ogre 1.12, see bc30cd9 . In this case we would lose the possibility of compiling with tbb task when using tbb <= 2020, but I am not sure this is actually of interest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like a more sensible approach. I'll have a go 😄

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
As tbb/version.h is seemingly not present before v2021.01, it cannot be
used to get the definition for TBB_VERSION_MAJOR, so let's use the main
tbb.h header which is present on all versions.

Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk>
@alexdewar
Copy link
Contributor Author

alexdewar commented Dec 17, 2021

Arg, so the CI checks are still failing for various reasons.

Annoyingly, tbb/tbb.h emits a warning about deprecated functionality and I can't see how else I can get at the TBB_VERSION_MAJOR macro. It's a harmless warning though -- I don't know how you feel about having additional warnings in the codebase. I'd rather avoid having to go back to using CMake to determine the version at compile time for reasons previously discussed.

The Windows build is now complaining about std::binary_function not being available, presumably because of a missing #include in TBB's headers. I couldn't help noticing that the TBB version seems to be really old though (from 2014!) so maybe updating to a more recent version might be a good idea in any case?

@traversaro
Copy link
Collaborator

Thanks a lot @alexdewar for working on this! Note that given the requirement listed by @scpeters on #3146 (review), if it is necessary to preserve the ABI of the library when compiled with tbb < 2021, I guess that we can't:

  • Remove an inheritance from tbb::task
  • Remove a public method
  • Add a private attribute (as it changes the ABI anyhow)

I guess we need to also put all those changes under the TBB_VERSION_MAJOR ifdef.

@scpeters
Copy link
Member

the work from this branch was merged in #3174

thanks for your contribution!

@scpeters scpeters closed this May 24, 2022
@alexdewar
Copy link
Contributor Author

Nw. Thanks for fixing this 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TBB task is deprecated and removed in oneTBB 2021.01
3 participants