[WIP] Fix issue 16431 - Get dependency output directly from the dmd run

[wilzbach]

As mentioned on #194 I also went ahead and implemented Andrei's idea:

apparently it is possible to run dmd in normal compilation mode and to simultaneously output dependencies. Then, for a single-file build the strategy would be to just compile the file to generate both dependencies and the .o file.

If no foreign dependencies found, great. Otherwise, there are two possibilities: hold on to that .o file and compile everything else in another .o file, then link both. Or (simpler with what we currently have), throw away the .o file and build everything in one shot.

Okay I tried both (I started with the simpler solution) and the runtimes with a single file are pretty good.

I didn't publish this yet, because I wanted to clean the code more, but well before @aG0aep6G and I create more duplicate work :P

Single file

Executable	runtime per loop
rdmd-master	646 ms
rdmd-regex	405 ms
rdmd-rerun	399 ms
rdmd-separate	399 ms
rdmd-aG0aep6G	400 ms

Multiple files

Tiny (one function file)

Executable	runtime per loop
rdmd-master	641 ms
rdmd-regex	643 ms
rdmd-rerun	608 ms
rdmd-separate	407ms
rdmd-aG0aep6G	408 ms

Medium (mir.random -unittest)

Executable	runtime per loop
rdmd-master	3.32s
rdmd-regex	3.15s
rdmd-rerun	3.56s
rdmd-separate	3.27s
rdmd-aG0aep6G	3.28s

I used the following to run the benchmarks (10 samples, 3 runs), so the results should be stable.

echo rdmd-master rdmd-regex rdmd-rerun rdmd-separate | tr ' ' '\n' | parallel -j1 "echo {}; python -m timeit -n 10 -r 3 -s 'import os' 'os.system(\"./{} --force bar2.d > /dev/null 2>&1\")'"

[dlang-bot]

@wilzbach, thanks for your PR! By analyzing the annotation information on this pull request, we identified @andralex, @CyberShadow and @leandro-lucarella-sociomantic to be potential reviewers. @andralex: The PR was automatically assigned to you, please reassign it if you were identified mistakenly.

(The DLang Bot is under development. If you experience any issues, please open an issue at its repo.)

[dlang-bot]

Fix	Bugzilla	Description
✓	16431	rdmd runs dmd twice for single-files with no dependencies

[aG0aep6G]

PR duel!

It's too late today for me to really look at rdmd.d anymore. Yours here seems to be more extensive than mine. So you may have caught details that I missed.

[wilzbach]

PR duel!

I posted a longer explanation on #194, but tl;dr this touches more lines than #194, so I am a happy looser.
If I find time, I will port the refactorings that make up most of the changes here to the future master branch.

[andralex]

@wilzbach thanks for this work and for graciously reviewing #194. I'll make separate comments about the refactorings.

[andralex]

Adding more structure has been looming over rdmd. When exactly the time has come depends on whom you ask. My guess is we can wait a bit more.