bin2llvmir crashes in llvm::DataLayout::getAlignmentInfo

[bansan85]

bin2llvmir fails in llvm::DataLayout::getAlignmentInfo

input

/home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir -provider-init -decoder -verify -x87-fpu -main-detection -idioms-libgcc -inst-opt -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -x86-addr-spaces -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path /home/legarrec/info/programmation/lib2lgc/build/src/lib2lgc.so.json -max-memory-half-ram -o /home/legarrec/info/programmation/lib2lgc/build/src/lib2lgc.so.bc

lib2lgc.so.gz

lib2lgc.so.json.zip

Output

Backtrace:

#0  0x00005555562c8b0a in llvm::DataLayout::getAlignmentInfo(llvm::AlignTypeEnum, unsigned int, bool, llvm::Type*) const ()
#1  0x0000555576bdf7c0 in ?? ()
#2  0x00005555560f97f9 in llvm::canConstantFoldCallTo(llvm::Function const*) ()
#3  0x00005555560fc04c in llvm::ConstantFoldInstOperands(llvm::Instruction*, llvm::ArrayRef<llvm::Constant*>, llvm::DataLayout const&, llvm::TargetLibraryInfo const*) ()
#4  0x0000555591f396c0 in ?? ()
#5  0x00007fffffffc580 in ?? ()
#6  0x0000555576bdf7c0 in ?? ()
#7  0x000055558f560fc8 in ?? ()
#8  0x0000555555c8d791 in llvm::InstCombiner::visitStoreInst(llvm::StoreInst&) ()
#9  0x0000555555c3be20 in llvm::InstCombiner::run() ()
#10 0x0000555555c3ca0c in combineInstructionsOverFunction(llvm::Function&, llvm::InstCombineWorklist&, llvm::AAResults*, llvm::AssumptionCache&, llvm::TargetLibraryInfo&, llvm::DominatorTree&, bool, llvm::LoopInfo*) ()
#11 0x0000555555c3cd02 in llvm::InstructionCombiningPass::runOnFunction(llvm::Function&) ()
#12 0x0000555556326251 in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
#13 0x000055555632672b in llvm::FPPassManager::runOnModule(llvm::Module&) ()
#14 0x0000555556325a14 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
#15 0x0000555555796866 in _main (argc=<optimized out>, argv=<optimized out>) at /home/legarrec/info/programmation/retdec/src/bin2llvmirtool/bin2llvmir.cpp:508
#16 0x000055555563baf6 in main (argc=<optimized out>, argv=<optimized out>) at /home/legarrec/info/programmation/retdec/src/bin2llvmirtool/bin2llvmir.cpp:526

I don't know why I don't have debug info. I compile with -g.

Valgrind:

==18289== Invalid read of size 1
==18289==    at 0xE7CB0A: llvm::DataLayout::getAlignmentInfo(llvm::AlignTypeEnum, unsigned int, bool, llvm::Type*) const (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x1BE48CDF: ???
==18289==    by 0xCAD7F8: llvm::canConstantFoldCallTo(llvm::Function const*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0xCB004B: llvm::ConstantFoldInstOperands(llvm::Instruction*, llvm::ArrayRef<llvm::Constant*>, llvm::DataLayout const&, llvm::TargetLibraryInfo const*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x56B78B0F: ???
==18289==    by 0x1FFEFFE4BF: ???
==18289==    by 0x1BE48CDF: ???
==18289==    by 0x521F7737: ???
==18289==    by 0x841790: llvm::InstCombiner::visitStoreInst(llvm::StoreInst&) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7EFE1F: llvm::InstCombiner::run() (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7F0A0B: combineInstructionsOverFunction(llvm::Function&, llvm::InstCombineWorklist&, llvm::AAResults*, llvm::AssumptionCache&, llvm::TargetLibraryInfo&, llvm::DominatorTree&, bool, llvm::LoopInfo*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7F0D01: llvm::InstructionCombiningPass::runOnFunction(llvm::Function&) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==  Address 0xffdf542cffdf58c7 is not stack'd, malloc'd or (recently) free'd
==18289== 
==18289== 
==18289== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==18289==  General Protection Fault
==18289==    at 0xE7CB0A: llvm::DataLayout::getAlignmentInfo(llvm::AlignTypeEnum, unsigned int, bool, llvm::Type*) const (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x1BE48CDF: ???
==18289==    by 0xCAD7F8: llvm::canConstantFoldCallTo(llvm::Function const*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0xCB004B: llvm::ConstantFoldInstOperands(llvm::Instruction*, llvm::ArrayRef<llvm::Constant*>, llvm::DataLayout const&, llvm::TargetLibraryInfo const*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x56B78B0F: ???
==18289==    by 0x1FFEFFE4BF: ???
==18289==    by 0x1BE48CDF: ???
==18289==    by 0x521F7737: ???
==18289==    by 0x841790: llvm::InstCombiner::visitStoreInst(llvm::StoreInst&) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7EFE1F: llvm::InstCombiner::run() (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7F0A0B: combineInstructionsOverFunction(llvm::Function&, llvm::InstCombineWorklist&, llvm::AAResults*, llvm::AssumptionCache&, llvm::TargetLibraryInfo&, llvm::DominatorTree&, bool, llvm::LoopInfo*) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289==    by 0x7F0D01: llvm::InstructionCombiningPass::runOnFunction(llvm::Function&) (in /home/legarrec/info/programmation/retdec/install/bin/retdec-bin2llvmir)
==18289== 
==18289== HEAP SUMMARY:
==18289==     in use at exit: 917,391,321 bytes in 8,731,282 blocks
==18289==   total heap usage: 75,609,058 allocs, 66,877,776 frees, 6,806,465,852 bytes allocated
==18289== 
==18289== LEAK SUMMARY:
==18289==    definitely lost: 53,962,168 bytes in 887,332 blocks
==18289==    indirectly lost: 4,443,847 bytes in 71,617 blocks
==18289==      possibly lost: 111,856,466 bytes in 1,066,048 blocks
==18289==    still reachable: 747,128,840 bytes in 6,706,285 blocks
==18289==                       of which reachable via heuristic:
==18289==                         multipleinheritance: 1,368 bytes in 3 blocks
==18289==         suppressed: 0 bytes in 0 blocks
==18289== Rerun with --leak-check=full to see details of leaked memory
==18289== 
==18289== For counts of detected and suppressed errors, rerun with: -v
==18289== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 0 from 0)

From master (91915c6682bd322193106801c62dfefeb702ef1d)

[s3rvac]

Thank you for the report. When I try to decompile the input file via retdec-decompiler.py lib2lgc.so, retdec-bin2llvmir fails with the following assertion:

Running phase: Value protection optimization ( 112.41s )
retdec-bin2llvmir: retdec/build/external/src/llvm-project/lib/IR/Type.cpp:275: llvm::FunctionType::FunctionType(llvm::Type*, llvm::ArrayRef<llvm::Type*>, bool): Assertion `isValidReturnType(Result) && "invalid return type for function"' failed.

@PeterMatula Can you please analyze why is it failing?

I don't know why I don't have debug info. I compile with -g.

We always compile LLVM in the Release mode, which is why you do not see any line information in stacktraces from LLVM.

[PeterMatula]

The reported issue was fixed, but the decompilation (at least on my machine) does not end successfully - ends on out of memory in the backend's SimpleCopyPropagationOptimizer. This is a known issue (#13).