retdec-unpacker crashes with "LLVM ERROR: out of memory" when unpacking an ELF binary

[s3rvac]

retdec-unpacker crashes with "LLVM ERROR: out of memory" when trying the unpack the given ELF binary file.

Input

Run

$ retdec-unpacker FILE

where FILE is:

• 194DFCAD928AAB20EFBE8B70695A560191CAEEC09425C7CDFA5E74C250989E34

Output

[UPX] Detected NRV2E unpacking stub based on signature.
[UPX] Started unpacking of file '194DFCAD928AAB20EFBE8B70695A560191CAEEC09425C7CDFA5E74C250989E34.dat'.
[UPX] Unfiltering filter 0x0 with parameter 0.
[UPX] Unpacking block at file offset 0x15f.
[UPX] Unfiltering filter 0x49 with parameter 88.
[UPX] Unpacking block at file offset 0x8fdc9.
[UPX] Unfiltering filter 0x0 with parameter 0.
[UPX] Additional packed data detected at the end of the file.
[UPX] Additional data are at file offset 0x9fa80 and have size of 0xfffffffffffffffd.
LLVM ERROR: out of memory
Aborted

Expected output

retdec-unpacker does not crash.

Output from valgrind

Argument 'size' of function __builtin_new has a fishy (possibly negative) value: -3
   at 0x4838DEF: operator new(unsigned long) (vg_replace_malloc.c:334)
   by 0x1BB707: __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) (new_allocator.h:111)
   by 0x1BB658: std::allocator_traits<std::allocator<unsigned char> >::allocate(...) (alloc_traits.h:436)
   by 0x1BB5BD: std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) (stl_vector.h:296)
   by 0x1FBAE4: std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) (vector.tcc:604)
   by 0x1FAAEC: std::vector<unsigned char, std::allocator<unsigned char> >::resize(unsigned long) (stl_vector.h:827)
   by 0x215C81: bool retdec::utils::(anonymous namespace)::readBytes<unsigned char>(...) (file_io.h:30)
   by 0x2194D5: bool retdec::utils::readFile<unsigned char>(...) (file_io.h:113)
   by 0x228EC1: retdec::unpackertool::upx::ElfUpxStub<64>::unpack(...) (elf_upx_stub.cpp:197)
   by 0x20FDCE: retdec::unpackertool::upx::UpxPlugin::unpack() (upx.cpp:95)
   by 0x184EA5: retdec::unpackertool::Plugin::run(retdec::unpackertool::Plugin::Arguments const&) (plugin.h:132)
   by 0x183891: unpackFile(...) (unpacker.cpp:97)

Notes

I suggest verifying the line below:

[UPX] Additional data are at file offset 0x9fa80 and have size of 0xfffffffffffffffd.

Value 0xfffffffffffffffd is probably the one that causes the error (-3 reported by valgrind above).

Configuration

• Commit: cdf863c (current master)
• 64b Arch Linux, GCC 8.3.0, Debug build of RetDec

[PeterMatula]

Fixed in a2cf4b4.