Chaos mode can fail with glibc <= 2.27 and executables with large unmapped regions

[rocallahan]

This can happen in glibc 2.27:

• An executable is mapped into memory with a large unmapped hole in it
• The dynamic linker loads libdl.so into that hole (due to rr chaos mode)
• Something calls dlopen()
• dlopen() calls dlopen_doit()
• dlopen_doit() calls _dl_check_caller, where the caller pointer is some code in libdl.so
• The first object encountered in the link map is the executable, whose address range contains libdl.so and therefore the caller pointer
• That executable is not on the list of allowed caller libraries, so _dl_check_caller returns 1 to indicate that the caller is invalid.
• dlopen() fails.

Fortunately this _dl_check_caller logic was removed in glibc 2.28. Unfortunately there are long-lived systems depending on glibc 2.27 that may not be upgraded for a long time.

[rocallahan]

I'm not sure yet if it's worth doing anything in rr to prevent this.