General issue tracking memory usage of llvmir2hll

[mewmew]

This is a general issue tracking the memory usage required to run llvmir2hll.

Background: I tried to run ./decompile.sh on a 1 MB sample executable (x86, 32-bit PE). All steps succeeded smoothly, until the llvmir2hll step, which managed to hang my laptop as it depleted the available RAM. The llvmir2hll process took roughly 7.1 GB at this point (and my hardware has 8 GB of RAM).

Have the memory usage of llvmir2hll been profiled before? If so, what are the low-hanging fruit to decrease memory usage for "regular" binaries, those below say 5 MB.

My wish is that as retdec develops, it should be possible to decompile an arbitrarily large executable, never requiring more than 1 GB of RAM memory at any given time during the decompile pipeline. However, this should be considered a long-term goal; as that requires intermediate representations of decompiled functions to be stored to disk, so that not all functions are kept in memory at the same time.

For now, locating low hanging fruit so that a decompile of binaries smaller than 5 MB (or so) should succeed using at most 3 GB of RAM. This seems like a worthy intermediate goal for memory usage optimization.

[mewmew]

This issue could be considered a duplicate of #13. Close it if you wish, so we can keep the discussion focused in one place.

[caioc2]

trying to decompile a 1.2MB program and llvmir2hll is using 46GB+ of memory untill the screen gets black.

If you need 128GB of RAM to decompile a 1.2mb program, makes little sense to even try to use retdec...

[s3rvac]

@caioc2 Mindless RetDec bashing isn't going to help anyone. What could help:

• Create a proper bug report, including attaching the input binary, or
• Try to analyze the problem and write your findings, or even
• Submit a PR that fixes some of the memory issues.

[PeterMatula]

This should be much better after this commit e6f47cf,
which was already merged to master. Also, there were some very helpful PRs optimizing llvmir2hll. We did not make a new release just yet. If you want, you can try to compile it yourself, or try the latest master builds.

Closing this, but keeping #13 open because there is more work to be done.