Clang sanitizer error with tbb

[SteveBronder]

Description

Adding the following to CXXFLAGS in make/local and running the profiling test (though I'm pretty sure this would fail with any test related to the tbb) gives an undefined behavior warning from the sanitizer

CXXFLAGS+=-fsanitize=undefined,address,leak -g -fno-omit-frame-pointer 

./runTests.py -j32 test/unit/math/rev/core/profiling_threading_test

Full err output here with the big bits being

lib/tbb_2019_U8/include/tbb/parallel_reduce.h:173:19: runtime error: member call on address 0x7f44453d3cc0 which does not point to an object of type 'tbb::task'
0x7f44453d3cc0: note: object has invalid vptr
 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lib/tbb_2019_U8/include/tbb/parallel_reduce.h:173:19 in 
lib/tbb_2019_U8/include/tbb/task.h:735:9: runtime error: member call on address 0x7f44453d3cc0 which does not point to an object of type 'tbb::task'
0x7f44453d3cc0: note: object has invalid vptr
 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lib/tbb_2019_U8/include/tbb/task.h:735:9 in 
lib/tbb_2019_U8/include/tbb/parallel_reduce.h:174:31: runtime error: member call on address 0x7f44453d3cc0 which does not point to an object of type 'tbb::task'
0x7f44453d3cc0: note: object has invalid vptr
 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lib/tbb_2019_U8/include/tbb/parallel_reduce.h:174:31 in 

and

Direct leak of 2592 byte(s) in 9 object(s) allocated from:
    #0 0x4cff8d in operator new(unsigned long) (/home/steve/stan/origin/math/test/unit/math/rev/core/profiling_threading_test+0x4cff8d)
    #1 0x4de385 in stan::math::AutodiffStackSingleton<stan::math::vari_base, stan::math::chainable_alloc>::init() /home/steve/stan/origin/math/./stan/math/rev/core/autodiffstackstorage.hpp:125:19
    #2 0x4de2c2 in stan::math::AutodiffStackSingleton<stan::math::vari_base, stan::math::chainable_alloc>::AutodiffStackSingleton() /home/steve/stan/origin/math/./stan/math/rev/core/autodiffstackstorage.hpp:93:44
    #3 0x4d9446 in std::_MakeUniq<stan::math::AutodiffStackSingleton<stan::math::vari_base, stan::math::chainable_alloc> >::__single_object std::make_unique<stan::math::AutodiffStackSingleton<stan::math::vari_base, stan::math::chainable_alloc> >() /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:962:34
    #4 0x4d7850 in stan::math::ad_tape_observer::on_scheduler_entry(bool) /home/steve/stan/origin/math/./stan/math/rev/core/init_chainablestack.hpp:46:29
    #5 0x7f4448f37508 in tbb::internal::observer_list::do_notify_entry_observers(tbb::internal::observer_proxy*&, bool) (/home/steve/stan/origin/math/lib/tbb/libtbb.so.2+0x29508)

This looks like it might be a tbb or sanitizer issue?

uxlfoundation/oneTBB#140

RcppCore/RcppParallel#36 (comment)

Expected Output

No UB or leaks

Current Version:

v4.0.1

[SteveBronder]

If this is a dependency issue then maybe we shouldn't run the sanitizer with tests that need to use STAN_THREADS

[SteveBronder]

Also ugh looking at this it looks one oneTBB uses cmake now so if updating tbb is the answer to this we have a whole swarm of stuff to do

[bbbales2]

Like all future TBBs are cmake? Like the next time we upgrade we gotta deal with this somehow?

[SteveBronder]

If onetbb is where we get new versions from then I think so :-/

https://github.com/oneapi-src/oneTBB

[wds15]

We should probably not upgrade for a while given that RcppParallel is using an outdated version... unless you move the PR you started. Is that PR dead by now or what is missing there? Updating the TBB is not so urgent, but it would be good.

As a compromise @adamhaber made our sources compatible with an external TBB.

[rok-cesnovar]

I dont think we will require cmake, I recently tried 2011.1.1 on Linux and worked fine without requiring any additional steps.

made our sources compatible with an external TBB.

That was @hsbadr :)

[rok-cesnovar]

But I agree, we should not upgrade until Rcppparallel upgrades.

[wds15]

@hsbadr sorry for confusing credits here. Thanks @rok-cesnovar for correcting.

[hsbadr]

The development version of RcppParallel uses the same version of TBB source code (RcppCore/RcppParallel#151) and supports oneTBB as an external library (RcppCore/RcppParallel#144 & RcppCore/RcppParallel#141). Though, the CRAN version uses incompatible older TBB source code (users run into issues with reduce_sum due to conflicting header). I use oneTBB for both.

[bbbales2]

There was also this: #2203

And then there is this: #1749

Not sure if either thing is relevant but seems like they could be.

[SteveBronder]

^^I think those are relevant. imo I think we can update to 2021.3 and keep backwards compatibility with the version on RcppParallel.

[SteveBronder]

I think this should be fine with the tbb version bump