Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] (coredump) Heap use after free in the bitmap functions results in be core #27409

Closed
3 tasks done
xy720 opened this issue Nov 22, 2023 · 0 comments
Closed
3 tasks done

[Bug] (coredump) Heap use after free in the bitmap functions results in be core #27409

xy720 opened this issue Nov 22, 2023 · 0 comments

Comments

@xy720
Copy link
Member

xy720 commented Nov 22, 2023 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar issues.

Version

trunk branch commit id:39663119ca435d4f1b8fe64c75def2e61484594f
Also in branch-1.2-lts

What's Wrong?

Heap use after free

What You Expected?

All is well

How to Reproduce?

1、create table with bitmap function

CREATE TABLE `behavior_bitmap_base` (
  `statistic_data` int(11) NULL,
  `one_id` bitmap BITMAP_UNION NULL,
  INDEX index_statistic_data (`statistic_data`) USING BITMAP COMMENT 'statistic_data'
) ENGINE=OLAP
AGGREGATE KEY(`statistic_data`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`statistic_data`) BUCKETS 5
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
);

2、load some data(skip details)

3、vim select_1.sql

select a.statistic_data,bitmap_or(a.one_id, b.one_id) as one_id from  
(
    select statistic_data,one_id from behavior_bitmap_base where identity_type_desc = 'MemberID'
) a join 
(
    select one_id from behavior_bitmap_base where system_type = 'm_oneid'
) b
union all 
select a.statistic_data,bitmap_or(a.one_id, b.one_id) as one_id from 
(
    select statistic_data,one_id from behavior_bitmap_base where identity_type_desc = 'CustomerID'
) a join 
(
    select one_id from orga302af5ab5e84d5e3c06900a6c9e65df.behavior_bitmap_base where system_type = 'c_oneid'
) b;

4、pressure test

mysqlslap -h127.0.0.1 -uroot -P9030 --debug-info --iterations=1 --concurrency=50 --number-of-queries=1000 --create-schema=xxx --query="./select_1" --delimiter=";"

Anything Else?

It was coredump both in jemalloc and tcmalloc:

1、jemalloc coredump:

0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /data/doris-2.x/be/src/common/signal_handler.h:
417
 1# 0x00007F71E866C400 in /lib64/libc.so.6
 2# __GI_raise in /lib64/libc.so.6
 3# abort in /lib64/libc.so.6
 4# __assert_fail_base in /lib64/libc.so.6
 5# 0x00007F71E8665252 in /lib64/libc.so.6
 6# 0x0000560455A1106D in /usr/local/service/doris/lib/be/doris_be
 7# ra_overwrite in /usr/local/service/doris/lib/be/doris_be
 8# roaring::Roaring::Roaring(roaring::Roaring const&) at /var/local/thirdparty/installed/include/roaring/roaring.hh:68
 9# void phmap::priv::btree_node<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::p
air<unsigned int const, roaring::Roaring> >, 256, false> >::emplace_value<std::pair<unsigned int const, roaring::Roaring> const&>(unsigne
d long, std::allocator<std::pair<unsigned int const, roaring::Roaring> >*, std::pair<unsigned int const, roaring::Roaring> const&) at /va
r/local/thirdparty/installed/include/parallel_hashmap/btree.h:2151
10# phmap::priv::btree_iterator<phmap::priv::btree_node<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>
, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >, std::pair<unsigned int const, roaring::Roaring>&, std:
:pair<unsigned int const, roaring::Roaring>*> phmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsi
gned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >::internal_emplace<std::pair<unsigned int const
, roaring::Roaring> const&>(phmap::priv::btree_iterator<phmap::priv::btree_node<phmap::priv::map_params<unsigned int, roaring::Roaring, p
hmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >, std::pair<unsigned int const, 
roaring::Roaring>&, std::pair<unsigned int const, roaring::Roaring>*>, std::pair<unsigned int const, roaring::Roaring> const&) at /var/lo
cal/thirdparty/installed/include/parallel_hashmap/btree.h:3135
11# void phmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<u
nsigned int const, roaring::Roaring> >, 256, false> >::copy_or_move_values_in_order<phmap::priv::btree<phmap::priv::map_params<unsigned i
nt, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> > const>(p
hmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned in
t const, roaring::Roaring> >, 256, false> > const*) at /var/local/thirdparty/installed/include/parallel_hashmap/btree.h:2487
12# doris::BitmapValue::_prepare_bitmap_for_write() at /data/doris-2.x/be/src/util/bitmap_value.h:2700
13# doris::BitmapValue::operator^=(doris::BitmapValue const&) at /data/doris-2.x/be/src/util/bitmap_value.h:1876
14# doris::vectorized::BitmapXor::vector_vector(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*, unsigned lon
g, unsigned long, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >&, doris::vectorized::IColumn*) at /data/doris-2.x/
be/src/vec/functions/function_bitmap_variadic.cpp:155
15# doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl_internal(doris::FunctionContext*, doris::vector
ized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) at /data/doris-2.x/be/src/vec/functions/function_bitmap_variadic.cpp:254
16# doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) at /data/doris-2.x/be/src/vec/functions/function_bitmap_variadic.cpp:220
17# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) at /data/doris-2.x/be/src/vec/functions/function.h:506
18# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-2.x/be/src/vec/functions/function.cpp:153
19# doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool, bool*) at /data/doris-2.x/be/src/vec/functions/function.cpp:239
20# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-2.x/be/src/vec/functions/function.cpp:262
21# doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-2.x/be/src/vec/functions/function.cpp:268
22# doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-2.x/be/src/vec/functions/function.h:177
23# doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) at /data/doris-2.x/be/src/vec/exprs/vectorized_fn_call.cpp:159
24# doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /data/doris-2.x/be/src/vec/exprs/vexpr_context.cpp:60
25# doris::vectorized::VUnionNode::materialize_block(doris::vectorized::Block*, int, doris::vectorized::Block*) at /data/doris-2.x/be/src/vec/exec/vunion_node.cpp:325
26# doris::vectorized::VUnionNode::get_next_materialized(doris::RuntimeState*, doris::vectorized::Block*) at /data/doris-2.x/be/src/vec/exec/vunion_node.cpp:192

...

2、tcmalloc coredump

0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /data/doris-1.x/be/src/common/signal_handler.h:
420
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
 4# 0x00007F25B632C400 in /lib64/libc.so.6
 5# memcpy at /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 6# run_container_clone in /usr/local/service/doris/lib/be/doris_be
 7# ra_overwrite in /usr/local/service/doris/lib/be/doris_be
 8# roaring::Roaring::Roaring(roaring::Roaring const&) at /var/local/thirdparty/installed/include/roaring/roaring.hh:68
 9# phmap::priv::btree_iterator<phmap::priv::btree_node<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>
, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >, std::pair<unsigned int const, roaring::Roaring>&, std:
:pair<unsigned int const, roaring::Roaring>*> phmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsi
gned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >::internal_emplace<std::pair<unsigned int const
, roaring::Roaring> const&>(phmap::priv::btree_iterator<phmap::priv::btree_node<phmap::priv::map_params<unsigned int, roaring::Roaring, p
hmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> >, std::pair<unsigned int const, 
roaring::Roaring>&, std::pair<unsigned int const, roaring::Roaring>*>, std::pair<unsigned int const, roaring::Roaring> const&) at /var/lo
cal/thirdparty/installed/include/parallel_hashmap/btree.h:3133
10# void phmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<u
nsigned int const, roaring::Roaring> >, 256, false> >::copy_or_move_values_in_order<phmap::priv::btree<phmap::priv::map_params<unsigned i
nt, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> >, 256, false> > const>(p
hmap::priv::btree<phmap::priv::map_params<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned in
t const, roaring::Roaring> >, 256, false> > const*) at /var/local/thirdparty/installed/include/parallel_hashmap/btree.h:2487
11# doris::BitmapValue::_prepare_bitmap_for_write() at /data/doris-1.x/be/src/util/bitmap_value.h:1955
12# doris::BitmapValue::operator^=(doris::BitmapValue const&) [clone .isra.0] at /data/doris-1.x/be/src/util/bitmap_value.h:1467
13# doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl_internal(doris_udf::FunctionContext*, doris::ve
ctorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) at /data/doris-1.x/be/
src/vec/functions/function_bitmap_variadic.cpp:230
14# doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl(doris_udf::FunctionContext*, doris::vectorized:
:Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) at /data/doris-1.x/be/src/vec/f
unctions/function_bitmap_variadic.cpp:198
15# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris_udf::FunctionContext*, doris::vectorized::Bloc
k&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-1.x/be/src/vec/
functions/function.cpp:244
16# doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool, bool*) at /data/doris-1.x/be/src/vec/functions/function.cpp:214
17# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-1.x/be/src/vec/functions/function.cpp:235
18# doris::vectorized::PreparedFunctionImpl::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-1.x/be/src/vec/functions/function.cpp:267
19# doris::vectorized::IFunctionBase::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) at /data/doris-1.x/be/src/vec/functions/function.h:154
20# doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) at /data/doris-1.x/be/src/vec/exprs/vectorized_fn_call.cpp:109
21# doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) at /data/doris-1.x/be/src/vec/exprs/vexpr_context.cpp:47
22# doris::vectorized::VUnionNode::materialize_block(doris::vectorized::Block*, doris::vectorized::Block*) at /data/doris-1.x/be/src/vec/exec/vunion_node.cpp:285
23# doris::vectorized::VUnionNode::get_next_materialized(doris::RuntimeState*, doris::vectorized::Block*) at /data/doris-1.x/be/src/vec/exec/vunion_node.cpp:157

...

3、asan output:

==87116==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200160a118 at pc 0x5560d96fe0f3 bp 0x7f47783b7f90 sp 0x7f47783b7f80
READ of size 2 at 0x60200160a118 thread T391 (FragmentMgrThre)
    #0 0x5560d96fe0f2 in inline_memcpy /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:132
    #1 0x5560d96feea5 in memcpy /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
    #2 0x5560ed490b67 in ra_overwrite (/usr/local/service/doris/lib/be/doris_be+0x25b72b67)
    #3 0x5560d9d077f5 in roaring::Roaring::Roaring(roaring::Roaring const&) /var/local/thirdparty/installed/include/roaring/roaring.hh:68

    ...too much output.

    #21 0x5560d9d08b26 in phmap::btree_map<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> > >::operator=(phmap::btree_map<unsigned int, roaring::Roaring, phmap::Less<unsigned int>, std::allocator<std::pair<unsigned int const, roaring::Roaring> > > const&) /var/local/thirdparty/installed/include/parallel_hashmap/btree.h:3963
    #22 0x5560d9d08b50 in doris::detail::Roaring64Map::operator=(doris::detail::Roaring64Map const&) /data/doris-1.x/be/src/util/bitmap_value.h:140
    #23 0x5560d9d12665 in doris::BitmapValue::_prepare_bitmap_for_write() /data/doris-1.x/be/src/util/bitmap_value.h:1954
    #24 0x5560d9d0f258 in doris::BitmapValue::operator^=(doris::BitmapValue const&) /data/doris-1.x/be/src/util/bitmap_value.h:1466
    #25 0x5560e31dac6c in doris::vectorized::BitmapXor::vector_vector(COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>*, unsigned long, unsigned long, std::vector<doris::BitmapValue, std::allocator<doris::BitmapValue> >&, doris::vectorized::IColumn*) /data/doris-1.x/be/src/vec/functions/function_bitmap_variadic.cpp:130
    #26 0x5560e31ed0a7 in doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl_internal(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) /data/doris-1.x/be/src/vec/functions/function_bitmap_variadic.cpp:230
    #27 0x5560e31e8a1c in doris::vectorized::FunctionBitMapVariadic<doris::vectorized::BitmapXor>::execute_impl(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) /data/doris-1.x/be/src/vec/functions/function_bitmap_variadic.cpp:196
    #28 0x5560e25fdaf5 in doris::vectorized::DefaultExecutable::execute_impl(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) /data/doris-1.x/be/src/vec/functions/function.h:484
    #29 0x5560e3f01c98 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) /data/doris-1.x/be/src/vec/functions/function.cpp:244
    #30 0x5560e3f01325 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool, bool*) /data/doris-1.x/be/src/vec/functions/function.cpp:214
    #31 0x5560e3f01a26 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) /data/doris-1.x/be/src/vec/functions/function.cpp:235
    #32 0x5560e3f01d98 in doris::vectorized::PreparedFunctionImpl::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) /data/doris-1.x/be/src/vec/functions/function.cpp:266
    #33 0x5560e25fa730 in doris::vectorized::IFunctionBase::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) /data/doris-1.x/be/src/vec/functions/function.h:155
    #34 0x5560e2513fef in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /data/doris-1.x/be/src/vec/exprs/vectorized_fn_call.cpp:109
#35 0x5560e25233be in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /data/doris-1.x/be/src/vec/exprs/vexpr_context.cpp:46
    #36 0x5560df029a77 in doris::vectorized::VUnionNode::materialize_block(doris::vectorized::Block*, doris::vectorized::Block*) /data/doris-1.x/be/src/vec/exec/vunion_node.cpp:285
    #37 0x5560df025478 in doris::vectorized::VUnionNode::get_next_materialized(doris::RuntimeState*, doris::vectorized::Block*) /data/doris-1.x/be/src/vec/exec/vunion_node.cpp:157

...

0x60200160a118 is located 8 bytes inside of 11-byte region [0x60200160a110,0x60200160a11b)
freed by thread T373 (FragmentMgrThre) here:
    #0 0x5560d96b8a6f in free (/usr/local/service/doris/lib/be/doris_be+0x11d9aa6f)
    #1 0x5560ed490883 in ra_shrink_to_fit (/usr/local/service/doris/lib/be/doris_be+0x25b72883)

previously allocated by thread T373 (FragmentMgrThre) here:
    #0 0x5560d96b8dc7 in __interceptor_malloc (/usr/local/service/doris/lib/be/doris_be+0x11d9adc7)
    #1 0x5560ed490822 in ra_shrink_to_fit (/usr/local/service/doris/lib/be/doris_be+0x25b72822)

...

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xy720