Fixed MemoryError when decoding large definite strings #204

agronholm · 2023-12-30T00:25:18Z

Relates to #198.

…trings

coveralls · 2023-12-30T00:27:31Z

coverage: 93.169% (-0.07%) from 93.237%
when pulling cbc78c5 on fix-memoryerror
into 0d54000 on master.

mschwager · 2024-01-02T18:19:17Z

I fuzzed this branch a bit, and it produced the following crash:

Output

==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x51600068f190 at pc 0xffff9530ccb8 bp 0xffffc175bb00 sp 0xffffc175baf8
READ of size 8 at 0x51600068f190 thread T0
    #0 0xffff9530ccb4 in Py_SIZE /usr/include/python3.11/object.h:142:20
    #1 0xffff9530ccb4 in PyBytes_GET_SIZE /usr/include/python3.11/cpython/bytesobject.h:45:12
    #2 0xffff9530ccb4 in fp_read_object /app/cbor2/source/decoder.c:367:47
    #3 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21
    #4 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15
    #5 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27
    #6 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33
    #7 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #8 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29
    #9 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #10 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27
    #11 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #12 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28
    #13 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16
    #14 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27
    #15 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15
    #16 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31
    #17 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #18 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #19 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #20 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #21 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #22 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13
    #23 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31
    #24 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #25 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19
    #26 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19
    #27 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #28 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #29 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #30 0x4e2ce8 in _PyFunction_Vectorcall (/usr/bin/python3.11+0x4e2ce8) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #31 0xffff955db6bc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47
    #32 0xffff955db6bc in pybind11::object pybind11::detail::object_api<pybind11::handle>::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95
    #33 0xffff955db6bc in pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82
    #34 0xffff955db6bc in void std::__invoke_impl<void, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36
    #35 0xffff955db6bc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33
    #36 0xffff955db6bc in std::_Function_handler<void (pybind11::bytes), pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30
    #37 0xffff955c270c in std::function<void (pybind11::bytes)>::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9
    #38 0xffff955c270c in void pybind11::detail::argument_loader<pybind11::bytes>::call_impl<void, std::function<void (pybind11::bytes)>&, 0ul, pybind11::detail::void_type>(std::function<void (pybind11::bytes)>&, std::integer_sequence<unsigned long, 0ul>, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37
    #39 0xffff955c270c in _ZNO8pybind116detail15argument_loaderIJNS_5bytesEEE4callIvNS0_9void_typeERSt8functionIFvS2_EEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES5_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65
    #40 0xffff955c270c in void pybind11::cpp_function::initialize<std::function<void (pybind11::bytes)>&, void, pybind11::bytes, pybind11::return_value_policy>(std::function<void (pybind11::bytes)>&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75
    #41 0xffff955c270c in void pybind11::cpp_function::initialize<std::function<void (pybind11::bytes)>&, void, pybind11::bytes, pybind11::return_value_policy>(std::function<void (pybind11::bytes)>&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21
    #42 0xffff955d1ee4 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35
    #43 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #44 0x5987d4 in PyObject_CallObject (/usr/bin/python3.11+0x5987d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #45 0xffff9509c0dc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47
    #46 0xffff9509c0dc in pybind11::object pybind11::detail::object_api<pybind11::handle>::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95
    #47 0xffff9509c0dc in pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82
    #48 0xffff9509c0dc in void std::__invoke_impl<void, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36
    #49 0xffff9509c0dc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33
    #50 0xffff9509c0dc in std::_Function_handler<void (pybind11::bytes), pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30
    #51 0xffff95089d38 in std::function<void (pybind11::bytes)>::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9
    #52 0xffff95089d38 in atheris::TestOneInput(unsigned char const*, unsigned long) /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/core.cc:146:26
    #53 0xffff98985308 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
    #54 0xffff98984bec in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:516:7
    #55 0xffff989862d0 in fuzzer::Fuzzer::MutateAndTestOne() /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19
    #56 0xffff9898712c in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5
    #57 0xffff98976d00 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:911:6
    #58 0xffff9508ac28 in atheris::start_fuzzing(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&) /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/core.cc:226:15
    #59 0xffff9509b9f8 in void pybind11::detail::argument_loader<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&>::call_impl<void, void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), 0ul, 1ul, pybind11::detail::void_type>(void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), std::integer_sequence<unsigned long, 0ul, 1ul>, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37
    #60 0xffff9509b9f8 in _ZNO8pybind116detail15argument_loaderIJRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EERKSt8functionIFvNS_5bytesEEEEE4callIvNS0_9void_typeERPFvSC_SI_EEENSt9enable_ifIXsrSt7is_voidIT_E5valueESL_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65
    #61 0xffff9509b9f8 in void pybind11::cpp_function::initialize<void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), void, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), void (*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75
    #62 0xffff9509b9f8 in void pybind11::cpp_function::initialize<void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), void, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), void (*)(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::function<void (pybind11::bytes)> const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21
    #63 0xffff950988b8 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35
    #64 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #65 0x5987d4 in PyObject_CallObject (/usr/bin/python3.11+0x5987d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #66 0xffff955bbdac in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47
    #67 0xffff955bbdac in pybind11::object pybind11::detail::object_api<pybind11::detail::accessor<pybind11::detail::accessor_policies::str_attr>>::operator()<(pybind11::return_value_policy)1, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>&, std::function<void (pybind11::bytes)>&>(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>&, std::function<void (pybind11::bytes)>&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95
    #68 0xffff955bbdac in atheris::Fuzz() /tmp/pip-install-ssq6l7v4/atheris_afc2a48c09d548c399b1f66614c10d64/src/native/atheris.cc:249:29
    #69 0xffff955c1540 in void pybind11::detail::argument_loader<>::call_impl<void, void (*&)(), pybind11::detail::void_type>(void (*&)(), std::integer_sequence<unsigned long>, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37
    #70 0xffff955c1540 in _ZNO8pybind116detail15argument_loaderIJEE4callIvNS0_9void_typeERPFvvEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES4_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65
    #71 0xffff955c1540 in void pybind11::cpp_function::initialize<void (*&)(), void, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(), void (*)(), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75
    #72 0xffff955c1540 in void pybind11::cpp_function::initialize<void (*&)(), void, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(), void (*)(), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21
    #73 0xffff955d1ee4 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:946:35
    #74 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #75 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #76 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #77 0x4a0b5c in PyEval_EvalCode (/usr/bin/python3.11+0x4a0b5c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #78 0x5fafa4  (/usr/bin/python3.11+0x5fafa4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #79 0x5f7bcc  (/usr/bin/python3.11+0x5f7bcc) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #80 0x60875c  (/usr/bin/python3.11+0x60875c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #81 0x608304 in _PyRun_SimpleFileObject (/usr/bin/python3.11+0x608304) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #82 0x60806c in _PyRun_AnyFileObject (/usr/bin/python3.11+0x60806c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #83 0x606318 in Py_RunMain (/usr/bin/python3.11+0x606318) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #84 0x5d0150 in Py_BytesMain (/usr/bin/python3.11+0x5d0150) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #85 0xffff9866777c  (/lib/aarch64-linux-gnu/libc.so.6+0x2777c) (BuildId: 122e8b69a986ce5b1fde3a0fa5d5c4fd522c701f)
    #86 0xffff98667854 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x27854) (BuildId: 122e8b69a986ce5b1fde3a0fa5d5c4fd522c701f)
    #87 0x5cffec in _start (/usr/bin/python3.11+0x5cffec) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)

0x51600068f190 is located 16 bytes inside of 518-byte region [0x51600068f180,0x51600068f386)
freed by thread T0 here:
    #0 0xffff98a99a9c in free /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
    #1 0x4c00f4  (/usr/bin/python3.11+0x4c00f4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #2 0xffff9530cac4 in Py_DECREF /usr/include/python3.11/object.h:538:9
    #3 0xffff9530cac4 in fp_read_object /app/cbor2/source/decoder.c:363:17
    #4 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21
    #5 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15
    #6 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27
    #7 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33
    #8 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #9 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29
    #10 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #11 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27
    #12 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #13 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28
    #14 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16
    #15 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27
    #16 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15
    #17 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31
    #18 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #19 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #20 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #21 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #22 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #23 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13
    #24 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31
    #25 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #26 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19
    #27 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19
    #28 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #29 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #30 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #31 0x4e2ce8 in _PyFunction_Vectorcall (/usr/bin/python3.11+0x4e2ce8) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #32 0xffff955db6bc in pybind11::detail::simple_collector<(pybind11::return_value_policy)1>::call(_object*) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1502:47
    #33 0xffff955db6bc in pybind11::object pybind11::detail::object_api<pybind11::handle>::operator()<(pybind11::return_value_policy)1, pybind11::bytes>(pybind11::bytes&&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1672:95
    #34 0xffff955db6bc in pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper::operator()(pybind11::bytes) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/functional.h:109:82
    #35 0xffff955db6bc in void std::__invoke_impl<void, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes>(std::__invoke_other, pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper&, pybind11::bytes&&) /usr/include/c++/12/bits/invoke.h:61:36
    #36 0xffff955db6bc in _ZSt10__invoke_rIvRZN8pybind116detail11type_casterISt8functionIFvNS0_5bytesEEEvE4loadENS0_6handleEbE12func_wrapperJS4_EENSt9enable_ifIXsrSt6__and_IJSt7is_voidIT_ESt14__is_invocableIT0_JDpT1_EEEE5valueESE_E4typeEOSH_DpOSI_ /usr/include/c++/12/bits/invoke.h:154:33
    #37 0xffff955db6bc in std::_Function_handler<void (pybind11::bytes), pybind11::detail::type_caster<std::function<void (pybind11::bytes)>, void>::load(pybind11::handle, bool)::func_wrapper>::_M_invoke(std::_Any_data const&, pybind11::bytes&&) /usr/include/c++/12/bits/std_function.h:290:30
    #38 0xffff955c270c in std::function<void (pybind11::bytes)>::operator()(pybind11::bytes) const /usr/include/c++/12/bits/std_function.h:591:9
    #39 0xffff955c270c in void pybind11::detail::argument_loader<pybind11::bytes>::call_impl<void, std::function<void (pybind11::bytes)>&, 0ul, pybind11::detail::void_type>(std::function<void (pybind11::bytes)>&, std::integer_sequence<unsigned long, 0ul>, pybind11::detail::void_type&&) && /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1480:37
    #40 0xffff955c270c in _ZNO8pybind116detail15argument_loaderIJNS_5bytesEEE4callIvNS0_9void_typeERSt8functionIFvS2_EEEENSt9enable_ifIXsrSt7is_voidIT_E5valueES5_E4typeEOT1_ /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/detail/../cast.h:1454:65
    #41 0xffff955c270c in void pybind11::cpp_function::initialize<std::function<void (pybind11::bytes)>&, void, pybind11::bytes, pybind11::return_value_policy>(std::function<void (pybind11::bytes)>&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:254:75
    #42 0xffff955c270c in void pybind11::cpp_function::initialize<std::function<void (pybind11::bytes)>&, void, pybind11::bytes, pybind11::return_value_policy>(std::function<void (pybind11::bytes)>&, void (*)(pybind11::bytes), pybind11::return_value_policy const&)::'lambda1'(pybind11::detail::function_call&)::_FUN(pybind11::detail::function_call&) /tmp/pip-build-env-dfdgednk/normal/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:224:21

previously allocated by thread T0 here:
    #0 0xffff98a99d30 in malloc /home/tcwg-buildslave/workspace/tcwg-llvm-release/tcwg-jade-03/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x5a25d4  (/usr/bin/python3.11+0x5a25d4) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #2 0x4d10ec  (/usr/bin/python3.11+0x4d10ec) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #3 0x4c800c  (/usr/bin/python3.11+0x4c800c) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #4 0x4c7988 in PyObject_CallFunctionObjArgs (/usr/bin/python3.11+0x4c7988) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #5 0xffff9530c920 in fp_read_object /app/cbor2/source/decoder.c:356:15
    #6 0xffff95308d8c in decode_definite_bytestring /app/cbor2/source/decoder.c:554:21
    #7 0xffff95308d8c in decode_bytestring /app/cbor2/source/decoder.c:625:15
    #8 0xffff95306834 in decode /app/cbor2/source/decoder.c:1736:27
    #9 0xffff9530b1d4 in decode_map /app/cbor2/source/decoder.c:914:33
    #10 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #11 0xffff9530bea0 in decode_semantic /app/cbor2/source/decoder.c:988:29
    #12 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #13 0xffff9530b460 in decode_map /app/cbor2/source/decoder.c:894:27
    #14 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #15 0xffff9530a634 in decode_definite_array /app/cbor2/source/decoder.c:818:28
    #16 0xffff9530a634 in decode_array /app/cbor2/source/decoder.c:875:16
    #17 0xffff95306814 in decode /app/cbor2/source/decoder.c:1738:27
    #18 0xffff95312550 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1458:15
    #19 0xffff9530be40 in decode_semantic /app/cbor2/source/decoder.c:977:31
    #20 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #21 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #22 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #23 0xffff9530b47c in decode_map /app/cbor2/source/decoder.c:899:33
    #24 0xffff953068b4 in decode /app/cbor2/source/decoder.c:1739:27
    #25 0xffff953107b8 in CBORDecoder_decode_bigfloat /app/cbor2/source/decoder.c:1246:13
    #26 0xffff9530bff8 in decode_semantic /app/cbor2/source/decoder.c:969:31
    #27 0xffff953068d4 in decode /app/cbor2/source/decoder.c:1740:27
    #28 0xffff953358f0 in CBOR2_load /app/cbor2/source/module.c:318:19
    #29 0xffff953358f0 in CBOR2_loads /app/cbor2/source/module.c:367:19
    #30 0x4c9d58  (/usr/bin/python3.11+0x4c9d58) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #31 0x494544 in _PyObject_MakeTpCall (/usr/bin/python3.11+0x494544) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)
    #32 0x4aa238 in _PyEval_EvalFrameDefault (/usr/bin/python3.11+0x4aa238) (BuildId: 15a1b7b17a3e246ca60bac3646ced99af27ca711)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/python3.11/object.h:142:20 in Py_SIZE
Shadow bytes around the buggy address:
  0x51600068ef00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51600068ef80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51600068f000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51600068f080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51600068f100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x51600068f180: fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51600068f200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51600068f280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51600068f300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x51600068f380: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51600068f400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1==ABORTING
MS: 4 InsertRepeatedBytes-InsertRepeatedBytes-InsertByte-CMP- DE: "~\377\377\377\377\377\377\336"-; base unit: b4d12705e8bb56c5481ecec9ebfda713fc9676b1
artifact_prefix='/tmp/output/'; Test unit written to /tmp/output/crash-54a4c03551e10f514542bdf35c00142e617b1ff1

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

mschwager · 2024-01-02T19:43:40Z

Would you be interested in having your project be a part of OSS-Fuzz? That could help with automatically finding crashes and bugs for you. I could take care of adding it as a new project - all I'd need is a primary contact email address.

agronholm · 2024-01-02T21:50:23Z

Sounds okay. Is it comparable with Hypothesis?

agronholm · 2024-01-02T21:50:51Z

And my primary contact address is alex.gronholm@nextday.fi.

mschwager · 2024-01-02T21:58:16Z

And my primary contact address is alex.gronholm@nextday.fi.

Hmm, the adding new projects docs say that a Google account is required. Is that email address connected to a Google account by chance?

Fuzzing testing is similar in some ways to Hypothesis. OSS-Fuzz is a project by Google to provide free compute cycles to fuzz OSS software. For more information on fuzzing, I'd recommend starting here.

agronholm · 2024-01-02T21:58:43Z

Yes, that's associated with a Google account.

agronholm · 2024-01-03T09:36:32Z

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

What do I do with this if I want to reproduce the crash?

mschwager · 2024-01-03T13:56:18Z

I've attached the crash here: crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt

What do I do with this if I want to reproduce the crash?

I tested it out like this:

$ python -m cbor2.tool -p crash-54a4c03551e10f514542bdf35c00142e617b1ff1.txt 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "lib/python3.11/site-packages/cbor2/tool.py", line 225, in <module>
    main()
  File "lib/python3.11/site-packages/cbor2/tool.py", line 208, in main
    objs = (load(infile, tag_hook=my_hook),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError

agronholm · 2024-01-04T08:52:35Z

Alright, I was able to reproduce the crash here. I think I need to start fuzzing the code before I push anything to master.

agronholm · 2024-01-04T22:10:23Z

Looks like this is just another case of allocating a huge amount of memory (4629771061636907009) bytes, but this triggers a MemoryError with the Python implementation too.

agronholm · 2024-01-06T12:37:35Z

Turns out that the problem wasn't triggered with a BytesIO instance, but it is triggered with a real open file.

agronholm · 2024-01-06T12:39:47Z

All other Python CBOR implementations I tested with also raised a MemoryError when trying to decode input like that.

agronholm · 2024-01-08T09:15:18Z

I have a fix that avoids MemoryError in both implementations, but it introduced a new bug on the C side that I'm still tracking down.

agronholm · 2024-01-09T10:27:34Z

Alright, what's left now is to fuzz this branch before I merge.

agronholm · 2024-01-09T10:37:19Z

I'm not getting any MemoryError anymore with fuzzing. Would you like to verify?

mschwager · 2024-01-10T17:03:06Z

I'm not getting any MemoryError anymore with fuzzing. Would you like to verify?

I found another crash with the following file: crash-c528afcec87be909de91322a14693702fd1d44a0.txt

I think I'm correctly fuzzing this branch, but I'm not sure. Are you able to reproduce the crash?

agronholm · 2024-01-14T12:01:14Z

Yeah, I can reproduce it. Looking into it now.

agronholm · 2024-01-14T12:04:35Z

No, wait, I forgot to recompile after switching branches. I'm getting this instead now: _cbor2.CBORDecodeEOF: premature end of stream (expected to read 65536 bytes, got 509 instead). This is the exception we should be getting, yes?

agronholm · 2024-01-14T12:12:19Z

I'm not seeing anything bad in this branch after the commit I just pushed a little while ago, so I'm merging it.

mschwager · 2024-01-30T16:51:45Z

Looks like the OSS-Fuzz PR went through: google/oss-fuzz#11444 🎉

Fixed MemoryError when decoding large definite byte-strings or text s…

42f75c9

…trings

mschwager mentioned this pull request Jan 3, 2024

Proposing new project: Python cbor2 google/oss-fuzz#11444

Merged

Merge branch 'master' into fix-memoryerror

fee08d9

agronholm added 4 commits January 9, 2024 12:10

Read long definite byte/text strings in 65536 byte chunks at most

68f74e1

Merge branch 'master' into fix-memoryerror

7670657

Merge branch 'master' into fix-memoryerror

a00f345

Fixed test error on Windows

8384772

Fixed return value check in CBORTag_hash()

cbc78c5

agronholm merged commit 387755e into master Jan 14, 2024
13 checks passed

agronholm deleted the fix-memoryerror branch January 14, 2024 12:12

kpwelsh mentioned this pull request Jun 14, 2024

N^2 Performance on decoding large definite bytestrings #240

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed MemoryError when decoding large definite strings #204

Fixed MemoryError when decoding large definite strings #204

agronholm commented Dec 30, 2023

coveralls commented Dec 30, 2023 •

edited

Loading

mschwager commented Jan 2, 2024

mschwager commented Jan 2, 2024

agronholm commented Jan 2, 2024

agronholm commented Jan 2, 2024

mschwager commented Jan 2, 2024

agronholm commented Jan 2, 2024

agronholm commented Jan 3, 2024

mschwager commented Jan 3, 2024

agronholm commented Jan 4, 2024

agronholm commented Jan 4, 2024

agronholm commented Jan 6, 2024

agronholm commented Jan 6, 2024

agronholm commented Jan 8, 2024

agronholm commented Jan 9, 2024

agronholm commented Jan 9, 2024

mschwager commented Jan 10, 2024

agronholm commented Jan 14, 2024

agronholm commented Jan 14, 2024

agronholm commented Jan 14, 2024

mschwager commented Jan 30, 2024

Fixed MemoryError when decoding large definite strings #204

Fixed MemoryError when decoding large definite strings #204

Conversation

agronholm commented Dec 30, 2023

coveralls commented Dec 30, 2023 • edited Loading

mschwager commented Jan 2, 2024

mschwager commented Jan 2, 2024

agronholm commented Jan 2, 2024

agronholm commented Jan 2, 2024

mschwager commented Jan 2, 2024

agronholm commented Jan 2, 2024

agronholm commented Jan 3, 2024

mschwager commented Jan 3, 2024

agronholm commented Jan 4, 2024

agronholm commented Jan 4, 2024

agronholm commented Jan 6, 2024

agronholm commented Jan 6, 2024

agronholm commented Jan 8, 2024

agronholm commented Jan 9, 2024

agronholm commented Jan 9, 2024

mschwager commented Jan 10, 2024

agronholm commented Jan 14, 2024

agronholm commented Jan 14, 2024

agronholm commented Jan 14, 2024

mschwager commented Jan 30, 2024

coveralls commented Dec 30, 2023 •

edited

Loading