heap-buffer-overflow when using nlohmann/json, ASAN, and gtest

[hamchapman]

• What is the issue you have?

I'll preface this by saying that I've also created an issue on the googletest repo. I'm not sure what the root cause of the issue is and so I figured I'd also create an issue here, in case you can shed some light.

I have a project uses CMake to build a library, FooBug, and a tests executable, which uses of googletest.

There appears to be an issue with an interaction between:

• making use of nlohmann/json, specifically creating a nlohmann::json object
  and then trying to extract the value using the get<std::vector<int>>()
  syntax, for example:

  json ints = json{1, 2, 3};
  std::vector<int> ints_vec = ints.get<std::vector<int>>();

• compiling the tests executable, which links gtest_main with the following
  ASAN-related flag: -fsanitize=address

• defining at least 5 tests

At the linking stage I get a heap-buffer-overflow reported by ASAN.

Error output

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/ham/Desktop/cpp/build/googletest-download
[ 11%] Performing update step for 'googletest'
[ 22%] No configure step for 'googletest'
[ 33%] No build step for 'googletest'
[ 44%] No install step for 'googletest'
[ 55%] No test step for 'googletest'
[ 66%] Completed 'googletest'
[100%] Built target googletest
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/ham/Desktop/cpp/build
[ 50%] Built target FooBug
[ 50%] Built target gtest
[ 75%] Built target gtest_main
Scanning dependencies of target tests
[ 87%] Building CXX object CMakeFiles/tests.dir/test/FooTest.cpp.o
[100%] Linking CXX executable tests
=================================================================
==28828==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000002bc at pc 0x0001085d2ba3 bp 0x7ffee7893e40 sp 0x7ffee7893608
READ of size 16 at 0x6020000002bc thread T0
    #0 0x1085d2ba2 in __asan_memcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x43ba2)
    #1 0x10838efdf in std::__1::enable_if<((std::__1::integral_constant::value) || (!(__has_construct, bool*, bool>::value))) && (is_trivially_move_constructible::value), void>::type std::__1::allocator_traits >::__construct_backward(std::__1::allocator&, bool*, bool*, bool*&) memory:1699
    #2 0x10838e570 in std::__1::vector >::__swap_out_circular_buffer(std::__1::__split_buffer&>&) vector:937
    #3 0x1083e3acb in void std::__1::vector >::__push_back_slow_path(int&&) vector:1621
    #4 0x1083aacd9 in std::__1::vector >::push_back(int&&) vector:1658
    #5 0x1083aab5e in testing::TestSuite::AddTestInfo(testing::TestInfo*) gtest.cc:2994
    #6 0x1083a95d3 in testing::internal::UnitTestImpl::AddTestInfo(void (*)(), void (*)(), testing::TestInfo*) gtest-internal-inl.h:700
    #7 0x1083a93c3 in testing::internal::MakeAndRegisterTestInfo(char const*, char const*, char const*, char const*, testing::internal::CodeLocation, void const*, void (*)(), void (*)(), testing::internal::TestFactoryBase*) gtest.cc:2769
    #8 0x10839985f in __cxx_global_var_init.9 FooTest.cpp:18
    #9 0x108399a58 in _GLOBAL__sub_I_FooTest.cpp FooTest.cpp
    #10 0x1139491e2 in ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) (dyld:x86_64+0x1b1e2)
    #11 0x1139495ed in ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) (dyld:x86_64+0x1b5ed)
    #12 0x11394400a in ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (dyld:x86_64+0x1600a)
    #13 0x113942013 in ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (dyld:x86_64+0x14013)
    #14 0x1139420b3 in ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) (dyld:x86_64+0x140b3)
    #15 0x1139305e5 in dyld::initializeMainExecutable() (dyld:x86_64+0x25e5)
    #16 0x113935af7 in dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) (dyld:x86_64+0x7af7)
    #17 0x11392f226 in dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) (dyld:x86_64+0x1226)
    #18 0x11392f024 in _dyld_start (dyld:x86_64+0x1024)
0x6020000002c0 is located 0 bytes to the right of 16-byte region [0x6020000002b0,0x6020000002c0)

allocated by thread T0 here:

#0 0x1085e0fdd in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x51fdd)

#1 0x108376648 in std::__1::__libcpp_allocate(unsigned long, unsigned long) new:253

#2 0x10838ed67 in std::__1::allocator::allocate(unsigned long, void const*) memory:1813

#3 0x10838ebc0 in std::__1::allocator_traits<std::__1::allocator >::allocate(std::__1::allocator&, unsigned long) memory:1546

#4 0x10838e96f in std::__1::__split_buffer<int, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) __split_buffer:311

#5 0x10838e49c in std::__1::__split_buffer<int, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) __split_buffer:310

#6 0x1083e3a7c in void std::__1::vector<int, std::__1::allocator >::__push_back_slow_path(int&&) vector:1617

#7 0x1083aacd9 in std::__1::vector<int, std::__1::allocator >::push_back(int&&) vector:1658

#8 0x1083aab5e in testing::TestSuite::AddTestInfo(testing::TestInfo*) gtest.cc:2994

#9 0x1083a95d3 in testing::internal::UnitTestImpl::AddTestInfo(void ()(), void ()(), testing::TestInfo*) gtest-internal-inl.h:700

#10 0x1083a93c3 in testing::internal::MakeAndRegisterTestInfo(char const*, char const*, char const*, char const*, testing::internal::CodeLocation, void const*, void ()(), void ()(), testing::internal::TestFactoryBase*) gtest.cc:2769

#11 0x1083990ff in __cxx_global_var_init.5 FooTest.cpp:13

#12 0x108399a4e in _GLOBAL__sub_I_FooTest.cpp FooTest.cpp

#13 0x1139491e2 in ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) (dyld:x86_64+0x1b1e2)

#14 0x1139495ed in ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) (dyld:x86_64+0x1b5ed)

#15 0x11394400a in ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (dyld:x86_64+0x1600a)

#16 0x113942013 in ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) (dyld:x86_64+0x14013)

#17 0x1139420b3 in ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) (dyld:x86_64+0x140b3)

#18 0x1139305e5 in dyld::initializeMainExecutable() (dyld:x86_64+0x25e5)

#19 0x113935af7 in dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) (dyld:x86_64+0x7af7)

#20 0x11392f226 in dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) (dyld:x86_64+0x1226)

#21 0x11392f024 in _dyld_start (dyld:x86_64+0x1024)
SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x43ba2) in __asan_memcpy

Shadow bytes around the buggy address:

0x1c0400000000: fa fa fd fd fa fa 00 00 fa fa 00 03 fa fa 00 02

0x1c0400000010: fa fa 00 04 fa fa 00 00 fa fa 00 06 fa fa 00 fa

0x1c0400000020: fa fa 00 00 fa fa 00 fa fa fa 00 fa fa fa 00 fa

0x1c0400000030: fa fa 00 fa fa fa 00 fa fa fa 04 fa fa fa fd fa

0x1c0400000040: fa fa fd fa fa fa 00 fa fa fa fd fd fa fa fd fa

=>0x1c0400000050: fa fa 00 fa fa fa 00[04]fa fa 00 fa fa fa 00 fa

0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

0x1c0400000070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

0x1c0400000080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

0x1c0400000090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

0x1c04000000a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa

Shadow byte legend (one shadow byte represents 8 application bytes):

Addressable:           00

Partially addressable: 01 02 03 04 05 06 07

Heap left redzone:       fa

Freed heap region:       fd

Stack left redzone:      f1

Stack mid redzone:       f2

Stack right redzone:     f3

Stack after return:      f5

Stack use after scope:   f8

Global redzone:          f9

Global init order:       f6

Poisoned by user:        f7

Container overflow:      fc

Array cookie:            ac

Intra object redzone:    bb

ASan internal:           fe

Left alloca redzone:     ca

Right alloca redzone:    cb

Shadow gap:              cc

==28828==ABORTING

CMake Error at /usr/local/Cellar/cmake/3.17.2/share/cmake/Modules/GoogleTestAddTests.cmake:40 (message):

Error running test executable.
Path: '/Users/ham/Desktop/cpp/build/tests'
Result: Child aborted
Output:

make[2]: *** [tests] Error 1

make[2]: *** Deleting file `tests'

make[1]: *** [CMakeFiles/tests.dir/all] Error 2

make: *** [all] Error 2

---

• Please describe the steps to reproduce the issue. Can you provide a small but working code example?

I've created a repo that reproduces the issue.

The relevant code is in FooTest.cpp:

#include "gtest/gtest.h"
#include <nlohmann/json.hpp>
#include "../include/foobar.hpp"
#include <vector>

using nlohmann::json;

class FooTests : public ::testing::Test {};

TEST_F(FooTests, Foo1) {
  ASSERT_EQ(foobar::add(1, 2), 3);

  // Everything is fine if you use a vector of doubles.
  json doubles_json = {1.1, 2.2, 3.3};
  std::vector<double> fetched_doubles = doubles_json.get<std::vector<double>>();

  // But you get the heap buffer overflow when using a vector of ints
  json ints_json = {1, 2, 3};

  // If you uncomment the line below then the buffer overflow won't occur
  std::vector<int> fetched_ints = ints_json.get<std::vector<int>>();
}

TEST_F(FooTests, Foo2) {}
TEST_F(FooTests, Foo3) {}
TEST_F(FooTests, Foo4) {}

// If you uncomment the line below then the buffer overflow won't occur
TEST_F(FooTests, Foo5) {}

• What is the expected behavior?

I can compile and link the executable and run it successfully without a heap-buffer-overflow.

• And what is the actual behavior instead?

A heap-buffer-overflow reported by ASAN.

• Which compiler and operating system are you using? Is it a supported compiler?

AppleClang 11.0.3.11030032
macOS 10.15.4 (19E287)

• Did you use a released version of the library or the version from the develop branch?

The version with tag v3.7.3 from the fetchcontent fork/mirror repo

The same issue occurs with the latest commit on develop though.

• If you experience a compilation error: can you compile and run the unit tests?

N/A

[nlohmann]

Hm. This seems unrelated to the library. At least the stack trace does not contain library code.

[hamchapman]

I think you're correct. I find it strange that it seems to be (at least in part) specifically tied to the get<std::vector<int>>() call from this library. The trace points to this line in the googletest repo:

https://github.com/google/googletest/blob/48bf552cbf014daf9149f8ee38b49e171ba4208f/googletest/src/gtest.cc#L2992

And you can see the definition of test_indices_ here: https://github.com/google/googletest/blob/48bf552cbf014daf9149f8ee38b49e171ba4208f/googletest/include/gtest/gtest.h#L1025-L1028

If I'm being honest that's all I really have to go on in terms of anything related to the issue pointing to this library.

Thanks for taking a look anyway, and feel free to close this if you feel confident that it's not something in this library that could be the root cause.

[nlohmann]

I'm not using Google Test myself, so I am not that familiar in setting up everything. Would be interesting to step through the code.

[hamchapman]

Just in case you are interested in stepping through the code then the repo I created that reproduces the issue is self-contained. You can get it up and running and reproduce with the following:

git clone https://github.com/hamchapman/googletest-json-asan-issue.git \
&& cd googletest-json-asan-issue && mkdir -p build && cd build && cmake .. \
&& make -j8 && ./tests

[hamchapman]

I think I've isolated this down to the line that's causing the issue for me.

If I comment out this line in the single include json.hpp file then I no longer get a buffer overflow:

json/single_include/nlohmann/json.hpp

Line 3404 in bcf4f3c

ret.reserve(j.size());

I really don't understand why that's problematic, especially seeing as if I comment out that line and instead print out j.size() then it prints out 3, as I'd expect, given my test code:

json ints_json = {1, 2, 3};
std::vector<int> fetched_ints = ints_json.get<std::vector<int>>();

If you've got any ideas as to what might be going on that would be great!

[hamchapman]

I now believe this to be an ASAN bug and have therefore created an issue there: google/sanitizers#1251

Feel free to close this if you're happy with that theory!

[hamchapman]

Update: It was a known false positive that can occur if not compiling everything (in my case gtest wasn't) with the -fsanitize=address flag.

google/sanitizers#1251

Sorry for the noise and thanks for the library 😊

[nlohmann]

Thanks for checking back! And deep down in my memory, I think I also had that problem once, but totally forgot about it. Good that this could be closed!