HELP! There is a memory leak in the code?!

[HuberyQ]

Discussed in https://github.com/nlohmann/json/discussions/2836

^{Originally posted by HuberyQ June 23, 2021}
Sorry to attract you with such a scary title. First of all, I would like to thank you for this work!

I found that after parsing the. MSG file, I got the JSON object successfully, but the CPU memory was not released until the end of the program! Is there a mistake in the way I use it? Can you manually free up the consumed memory?

The size of the. MSG file is about 24MB, but the memory consumption of the parsing process is about 0.4G! Here is the test source code:

void test_json(string path)
{
    std::ifstream ifs(path, std::ios::in | std::ios::binary);
    if (!ifs.is_open())
    {
        throw std::runtime_error("cannot load the file at " + path);
    }

    std::vector<uint8_t> msgpack;
    while (true)
    {
        uint8_t buffer;
        ifs.read(reinterpret_cast<char*>(&buffer), sizeof(uint8_t));
        if (ifs.eof())
        {
            break;
        }
        msgpack.push_back(buffer);
    }
    ifs.close();

    const auto json = nlohmann::json::from_msgpack(msgpack);
    cout << sizeof(json) / 1024.0 / 1024.0 << endl;
}

[HuberyQ]

I have two questions: (1) How to release the memory consumption that has not been released? (2) Why is memory consumption so huge?
Hope to get help from you or other people, thank you!

[HuberyQ]

This is the .msg file: 1.msg

[HuberyQ]

I didn't test with Valgrind or ASAN...I just monitoring CPU through my terminal.
Ubuntu 20.04 / C++11

[nlohmann]

• There is no memory leak in the library - you can compile your code with ASAN or run it with Valgrind. All memory acquired for json is release once it goes out of scope.

• The memory consumption of the library is due to the overhead described in the README:

  Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte).

  Your JSON value contains of a lot of numbers, and hence this overhead adds up.

Note the overhead is much smaller if you compile your binary in release mode.

[HuberyQ]

* There is no memory leak in the library - you can compile your code with ASAN or run it with Valgrind. All memory acquired for `json` is release once it goes out of scope.

* The memory consumption of the library is due to the overhead described in the README:
  > Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte).
  
  
  Your JSON value contains of a lot of numbers, and hence this overhead adds up.

Note the overhead is much smaller if you compile your binary in release mode.

Thank you for your reply! But when the test_json function exits, the CPU memory growth caused by it is not released! You can use the data and code above to reproduce it. We need your help!

[nlohmann]

I cannot reproduce this:

The library releases all memory, there are no memory leaks. Can you please try to run your code with Valgrind or ASAN?

[gregmarr]

For an explanation of why memory freed by the application isn't returned to the OS: https://stackoverflow.com/questions/45538993/why-dont-memory-allocators-actively-return-freed-memory-to-the-os

[HuberyQ]

For an explanation of why memory freed by the application isn't returned to the OS: https://stackoverflow.com/questions/45538993/why-dont-memory-allocators-actively-return-freed-memory-to-the-os

Thanks! It really helps a lot!

[HuberyQ]

I cannot reproduce this:

The library releases all memory, there are no memory leaks. Can you please try to run your code with Valgrind or ASAN?

I tested on different systems：
ubuntu16：It's normal and ok!
ubuntu18：The memory was not released!

Then I refered to #1924 , and it worked! I guess that different versions of the glibc lead to different memory recovery mechanisms.

Thanks a lot!
This library is really good to use!

[nlohmann]

Do you need further assistance with this issue?

[HuberyQ]

Do you need further assistance with this issue?

Nope.Thanks!