ASan needs to keep track of all the libraries loaded during the process lifetime #89
Comments
|
Reported by
|
|
Any chance for this getting fixed some day? |
|
@obfuscated we are not working on this. What exactly do you need? |
|
@kcc I'm getting unknown modules lines in the callstacks printed by the leak report from asan. Our application uses plugins (dlopened .so files on linux) quite extensively, so the leak reports aren't too useful. Also there are some leaks that I want to suppress, but I'm not sure I can, because of the lines. I'm using clang-3.9.1, centos 6, linux. |
|
Understood. I think there could also be a simple workaround on your side: don't dlclose anything when testing under asan/lsan |
|
OK, going with the no-dlclose workaround. I remembered that I've done this for valgrind, but there is another place in our code that does dlclose calls, so patching these resolved the problem. I've stopped seeing the leaks for the global variables in dlopened shared libraries, but I guess this is expected. |
Depends on what exactly you mean here |
|
I'm still investigating, so I'm not sure if it is a problem of the tool, my setup or real bug in the application. |
|
Is this still unlikely to be fixed? Given the prevalence of shard libraries in most production system it seems very likely for people to trip over this. We're running into this problem working with the CUDA libraries for example. They have a leak in an internal driver library that they dlopen and dlclose themselves. |
|
We are not planing any work in this space, my previous comment (Aug 10 2017) still holds. If you have some CUDA-specific problem I suggest you open a separate issue -- we may be able to find a specialized solution. |
|
I just ran into this too. It's a common problem for different leak detectors. Our solution (like @kcc suggests and @obfuscated implemented) was to simply not For valgrind the solution was: #ifdef HAVE_VALGRIND_H
# include <valgrind.h>
#else
# define RUNNING_ON_VALGRIND 0
#endif
static int fr_dlfree(dl_t *module)
{
...
/*
* Only dlclose() handle if we're *NOT* running under valgrind
* as it unloads the symbols valgrind needs.
*/
if (!RUNNING_ON_VALGRIND) dlclose(module->handle);
...
}There's a couple of approaches for LSAN detection in this stack overflow post. It would be nice if there were a similar function/macro available to the one valgrind provides, so that we could do this in a way that didn't rely on implementation details. |
|
@kcc: If we still don't plan to fix this, can we close this bug? |
|
Not going to work on this any time soon. Closing for now, will reopen if there is |
|
I have this same problem in multiple projects and a good example of how to reproduce this problem was provided here https://stackoverflow.com/questions/44627258/addresssanitizer-and-loading-of-dynamic-libraries-at-runtime-unknown-module Valgrind over the same code appears to be working fine for me and tracking down these leaks with ASAN isn't straightforward so a fix for this dynamic linking issue to give better symbols would be appreciated. |
|
Just in case it's helpful:
|
|
Just ran into this, mentioning it here in case resources ever become available for a fix |
|
An option to get a reliable stack trace on Linux is to use |
|
@bungow Thanks that worked, but you might want to Note the |
|
Also, since there is a decent workaround, perhaps at the point where it prints |
Oof. Writing down the details on this one, as I doubt I'll remember...
The start was turning on both the AddressSanitizer and Qt. When doing
so, most programs suddenly began reporting memory leaks. However, the
report was rather cryptic, being four entries similar to this one:
==1262202==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 21 byte(s) in 1 object(s) allocated from:
#0 0x498087 in posix_memalign (build/src/libdm/tests/dm_test+0x498087)
#1 0x7f4e90e53675 in alloc brlcad/src/libbu/malloc.c:129:10
#2 0x7f4e90e53400 in bu_malloc brlcad/src/libbu/malloc.c:167:12
#3 0x7f4e90ee1c02 in bu_strdupm brlcad/src/libbu/str.c:165:17
#4 0x7f4e87dcd3e9 (<unknown module>)
#5 0x7f4e87dcdea5 (<unknown module>)
#6 0x7f4e9650eb89 (/lib64/ld-linux-x86-64.so.2+0x11b89)
I wasn't immediately sure if the unknown module error was related to the
plugin loading, and spent a lot of time trying to find a string memory
issue in dm_init.cpp. After that proved fruitless, other than to
confirm that the error disappeared if i removed the bu_dlclose-ing of
the handles saved at initialization, I began to look for ways to decode
the "unknown module" entries. That led to the following issue:
google/sanitizers#89 which describes the
problem ASan has with dynamic libraries. Comments indicated that it
would see real issues, but won't report which dynamic library they come
from (and there are no plans to fix this anytime soon... grr...). In
fairness, valgrind can see the error too but also has the same reporting
problem; it appears to be ubiquitous.
Fortunately, we have an alternative due to the way our plugin system and
test apps work - we can simply add and remove .so files to the directory
and see how the error reporting changes to zero in on which file(s) are
triggering the problem. Doing so quickly made apparent what should have
been obvious in retrospect - two of the errors each were coming from the
Qt and swrast plugins, which had been off in earlier testing.
Since there was no backtrace beyond the bu_strdupm call itself, and
there were two errors per file, the suspect was the bu_strdup calls
initializing the "char *" names for the fb structures. The C files use
static strings (which is why the non-Qt plugins didn't show the issue),
but C++ doesn't tolerate the type mismatch. The original hack
workaround was just to bu_strdup and create a (char *) string, but as
the leak detectors correctly note this also means there's no way to
clean up the allocated memory.
As far as I can tell there is no reason for these strings to be editable
(char *) strings - the dm container's equivalents are static. This
commit removes any logic assuming if_name is dynamic, and also removes
the bu_strdup hack from Qt and swrast.
|
Valgrind supports this feature since https://bugs.kde.org/show_bug.cgi?id=79362 with the parameter --keep-debuginfo=yes |
https://jira.percona.com/browse/PXC-3907 Do not unload galera for ASan builds. google/sanitizers#89
…nitizers#89) - build for debug w/UBSAN to avoid ub being optimized away
…nitizers#89) - build for debug w/UBSAN to avoid ub being optimized away (cherry picked from commit c7868e8)
…nitizers#89) - build for debug w/UBSAN to avoid ub being optimized away (cherry picked from commit c7868e8)
…nitizers#89) - build for debug w/UBSAN to avoid ub being optimized away
|
I ran into this issue, and the work-arounds of not unloading the library didn't work for me, FYI: here's links to the patches in Valgrind which support the equivalent feature in Valgrind: |
133: Fix use-after-free and 3 memory leaks; enforce AddressSanitizer in CI r=Bromeon a=Bromeon
Changes implementation of the `Gd` smart pointer to cache the instance ID in each object. To my knowledge, instance IDs are the only reliable way to check for object validity in Godot, as raw object pointers may become dangling.
In addition, this PR fixes 3 memory leaks around arrays and dictionaries. Those occurred due to `from_sys_init()` using `T::default()` in too many places. This essentially lead to allocating a default-constructed object, which is then immediately overwritten by the `init` function, which _also_ allocates a new object. I refactored the `GodotFfi::from_sys_init()` to **never** call `default()`, and add an explicit `from_sys_init_default()` for cases where this is desired. This also cuts down on some of the boilerplate.
---
Both use-after-free and memory leaks were discovered using AddressSanitizer/LeakSanitizer. Great tooling from the C++ world, which also proves useful for us, as miri can't be used in FFI contexts. From now on, UB or leaks detected by ASan/LSan will cause a hard error in CI. The tools are not 100% bullet-proof; they didn't detect the following UAF case in a short test, but they are still of great value as a more systematic counter to memory errors.
<details><summary>use-after-free, false negative</summary>
```rs
let mut boks = Box::new(44);
let ptr = std::ptr::addr_of_mut!(*boks);
println!("deref={}", unsafe { *ptr }); // output: deref=44
drop(boks);
println!("deref={}", unsafe { *ptr }); // output: deref=1719666059
```
</details>
On a side note, getting these working correctly in CI was a bit of a marathon because ASan/LSan don't have stacktraces for dynamically loaded libraries (a known wontfix problem, see google/sanitizers#89). Additionally, false positives for memory leaks were reported: a simple `println!` would cause 1024 bytes of non-reclaimable memory. Therefore, I had to compile a special version of nightly Godot that disables dynamic library unloading via `dlclose`, to keep the stacktrace around, and this seemed to fix the false-positive issue as well. Although likely unrelated, what I also found during research was rust-lang/rust#19776.
Fixes #89.
Co-authored-by: Jan Haller <bromeon@gmail.com>
|
I have a specific case of this problem where all workarounds mentioned here fail: There's a Java connector for a native library
Ideally, I'd like to suppress all leaks which don't involve libfoo, including those coming from "unknown modules" only. I haven't found a way to do that, though - leak suppressions don't offer something like negation. The whole thing is running in a Ubuntu 18.04 docker container. Full command is |
Another option I found is to override #define _GNU_SOURCE
#include <dlfcn.h>
#include <link.h>
#include <stdio.h>
#include <string.h>
// Override dlopen() function and inject RTLD_NODELETE so the library
// doesn't get deleted on close().
// This helps with asan traces with <unknown module>
void* dlopen(const char* filename, int flags){
typedef void* (*dlopen_t)(const char*, int);
dlopen_t original_dlopen = (dlopen_t)dlsym(RTLD_NEXT, "dlopen");
printf("Intercepted a dlopen call, injecting RTLD_NODELETE\n");
flags |= RTLD_NODELETE;
return original_dlopen(filename, flags);
}Build with Then pre-load it after asan. Asan must be loaded first. You'll have to adjust the following paths. |
|
@ericriff that may work for some people, but unfortunately making The situation is unchanged. For us it still a pain because in our software we have 200+ dynamicaly loaded modules (.so) and when a (small) memory leak is reported, we have no idea which one is leaking. |
|
Update on my previous comment: I was wrongly pointing at liblcms2, there's a simpler Java example where many |
Originally reported on Google Code with ID 89
Reported by
ramosian.glideron 2012-07-18 09:21:08The text was updated successfully, but these errors were encountered: