Add experimental -hash-threshold option to hash very long symbol names.

[JohanEngelen]

This adds MD5 hashing of symbol names that are larger than threshold set by -hashthres.

What is very unfortunate is that std.traits depends on the mangled name, doing string parsing of the mangled name of symbols to obtain symbol traits. This means that mangling cannot be changed (dramatically, like hashing) at a high level, and the hashing has to be done on a lower level.

Hashed symbols look like this:
_D3one3two5three3L3433_46a82aac733d8a4b3588d7fa8937aad66Result3fooZ
ddemangle gives:
one.two.three.L34._46a82aac733d8a4b3588d7fa8937aad6.Result.foo
Meaning: this symbol is defined in module one.two.three on line 34. The identifier is foo and is contained in the struct or class Result.

Symbols that may be hashed:

• functions
• struct/class initializer
• vtable
• typeinfo (needed surgery inside FE code)

The feature is experimental, and has been tested on Weka.io's codebase. Compilation with -hashthres=1000 results in a binary that is half the size of the original (201MB vs. 461MB). I did not observe a significant difference in total build times. Hash threshold of 8000 gives 229MB, 800 gives 195MB binary size: there is not much gain after a certain hash threshold.
Linking Weka's code fails with a threshold of 500: phobos contains a few large symbols (one larger than 8kb!) and this PR currently does not disable hashing of symbols that are inside phobos, hence "experimental". Future work could try to figure out whether a symbol is inside phobos or not.

[JohanEngelen]

phobos contains a few large symbols (one larger than 8kb!)

Correction: it is ~5322 bytes large

[wilzbach]

-hashthres

I was just confused by this, maybe you don't mind the extra 4 chars and make it -hashthreshold, s.t. it's name is clearer?

[JohanEngelen]

Hm, reading through our other options, I guess -hash-threshold would be best.

[JohanEngelen]

ping @klickverbot @redstar

[dnadlinger]

Why didn't you implement it in D? That way, there would be a chance of submitting it to upstream… ;)

[dnadlinger]

This seems rather unfortunate. Perhaps we should ditch IN_LLVM for such cases, or replace it by an enum set from the version (so you can do if (IN_LLVM && …). It doesn't seem like we would ever want to try using LDC's front end sources to build DMD…

[JohanEngelen]

OK. We already have the enum IN_LLVM, so I'll use that. I also think the copying is ugly/stupid.
I will indent the DDMD source, so that we are notified of (perhaps relevant) changes by merge errors.

[dnadlinger]

Personally, I'd probably keep the indentation as it is, so that the diff is kept tidy and the LDC-specific part is made painfully obvious when browsing the source. But I guess one could always use diff -w for the former…

[JohanEngelen]

Many (aggravating) reasons for it.

[dnadlinger]

If the switch is marked as experimental and it is made clear that it might disappear/behave differently in the future (release notes, …), it should probably be fine to add it as-is. I would very much hope that in the long term this remains a quick band-aid fix for Weka, though, until we get a proper upstream solution.

[JohanEngelen]

Merging when green on testers.

Ideas for future work:

• use faster hasher
• make a hashing OutBuffer that immediately hashes incoming characters when the length passes a certain threshold, and pass that outbuffer to the Mangler (see dmangle.d). This would prevent the need for allocating a lot of memory storage for large symbol names.

[dnadlinger]

Would it make sense to use DMD's approach of only hashing the names before emitting them to object files? This way, the Phobos code relying on .mangleof wouldn't be affected.

[JohanEngelen]

Would it make sense to use DMD's approach of only hashing the names before emitting them to object files? This way, the Phobos code relying on .mangleof wouldn't be affected.

Hm?
That is what this PR has been doing from day 1. ;)

[dnadlinger]

Ah, sorry, I had misremembered that (and the location of the diff in mtype.d).