allocator_may_return_null=1 does not seem to work, hindering fuzzing for security issues

[vanhauser-thc]

allocator_may_return_null=1 is not working for me, which is quite an issue when fuzzing for security vulnerabilities.
It is an important security check to verify that if an allocator returns NULL that the program does not crash because of this, which is not possible if allocator_may_return_null does not work. We basically need ASAN, but not this specific check, when fuzzing.

The issue has been raised a few times, e.g. in #740, #779 and #1377, some fixes were done as it seems, however for me this is not working in either LLVM 16, 17 or 18.

Simple example:

#include <iostream>
#include <new>

int main() {
    try {
        char* arr = new char[3ul << 40];
        delete[] arr;
    } catch (const std::bad_alloc& e) {
        std::cerr << "Allocation failed: " << e.what() << '\n';
    }
    return 0;
}

LLVM result with allocator_may_return_null=1:

$ clang++ -fsanitize=address -o test test.cpp
$ ASAN_OPTIONS=allocator_may_return_null=1 ./test
==3453407==WARNING: AddressSanitizer failed to allocate 0x30000000000 bytes
=================================================================
==3453407==ERROR: AddressSanitizer: allocator is out of memory trying to allocate 0x30000000000 bytes
    #0 0x7ffff74b00c7 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:102
    #1 0x555555555263 in main (/tmp/x+0x1263)
    #2 0x7ffff72461c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

==3453407==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: out-of-memory ../../../../src/libsanitizer/asan/asan_new_delete.cpp:102 in operator new[](unsigned long)
==3453407==ABORTING

also pinging @kcc

[JarLob]

From the comment it sounds the sanitizer itself may not work in low memory conditions. This is probably why it prefers to abort early on OOM. Which is still unfortunate for fuzzing. Having the allocator_may_return_null=1 flag introduces confusion.