Performance regressions without LTO in LDC 1.3.0-beta[1|2]

[jondegenhardt]

I'm seeing performance regressions in several of my standard benchmarks (https://github.com/eBay/tsv-utils-dlang/blob/master/docs/Performance.md). Specifically the tsv-filter, tsv-summarize, and csv2tsv tests. Performance is off 20-30% from previous LDC releases.

So far I've only tested OSX (Xcode 8.3.3).

I won't have time to diagnose further for several days, but I'll see if I can narrow this further sometime during the week of June 19. I don't think it's likely to be due to changes in Phobos, but I can check that and if it is specific to OSX. If a fix is created for issue #2161 (performance degradation with boundscheck=off) I can check that as well. Testing with this option removed did show improvement on one benchmark, but overall benchmarks were worse.

[kinke]

According to your benchmark page, the numbers have been obtained using LDC 1.1, so have you compared 1.1 to the 1.3 betas, skipping 1.2? I'm asking since we switched to LLVM 4.0 (on non-Windows) starting with 1.2 and the regressions are most likely due to that, although we have only heard from general performance improvements (even > 10%) with LLVM 4.0 until now.

[jondegenhardt]

It's possible that I skipped 1.2, don't remember off-hand. I'll test with 1.2 and update this report as soon as I can. It may be a couple days, I have some schedule conflicts at the moment.

I normally test every LDC release, though without updating the benchmarks page. And I've been using 1.2 since it came out, so it's likely I tested it also. However, I don't have any record of it.

[jondegenhardt]

I've rerun the six benchmarks I use, there is definitely a performance regression in three tests between LDC 1.2 and LDC 1.3-beta2 on OSX (xcode 8.3.3). The main deltas:

• tsv-filter (numeric): 24% slower
• tsv-summarize: 38% slower
• csv2tsv: 27% slower

The other three tests are not materially changed. The regex version of the tsv-filter test has had a small performance decrease over several releases, this is likely due to changes in std.regex during these releases.

While running the tests, I discovered there is a delta when using the -flto=full switch. It generates speed gains on several tests.

I'll see what I can do to narrow this further. The tsv-filter (numeric) and tsv-summarize tests have numeric processing in common, may be a clue there. The csv2tsv test does not do numeric processing. I'll still have schedule conflicts for several days, so may be a bit before I have more.

Below is a table of performance numbers over several LDC releases. Each metric was run at least 5 times to ensure consistency. The "flto" versions are built with the -flto=full option. All source files are on included in one command line. Options used:

• Without flto: -release -O3 -boundscheck=off
• With flto: -release -O3 -boundscheck=off -singleobj -flto=full

Compilation	tsv-filter (numeric)	tsv-filter (regex)	tsv-select	tsv-summarize	tsv-join	csv2tsv
LDC 1.0	4.50	7.48	7.80	15.76	23.64	26.49
LDC 1.1	4.55	7.32	4.24	15.93	20.73	29.05
LDC 1.1 flto	4.53	7.16	4.16	16.25	20.84	22.50
LDC 1.2	4.47	7.95	4.26	15.37	20.75	29.70
LDC 1.2 flto	4.42	7.61	4.19	15.80	20.76	23.91
LDC 1.3	5.50	8.59	4.25	21.43	20.84	38.26
LDC 1.3 flto	5.47	7.57	4.17	21.78	20.77	30.37
DMD 2.074	5.71	11.34	9.71	18.58	40.59	57.38

LLVM and Phobos versions for each LDC release:

LDC version	Phobos version	LLVM version
LDC 1.0	2.070.2	3.8.0
LDC 1.1	2.071.2	3.9.1
LDC 1.2	2.072.2	4.0.0
LDC 1.3 beta2	2.073.2	4.0.0

[JohanEngelen]

@jondegenhardt just a quick note: those numbers are kindof cool to report in a blog post somewhere :) Did you also try with -flto=thin ?

[JohanEngelen]

Can you add the LLVM version to that table? Thanks.

[jondegenhardt]

@JohanEngelen Thanks! Thought the numbers might be of general interest. I did not try -flto=thin. I'll add the LLVM version to the table later today, can't right now.

[kinke]

I would have liked this to correlate with #2161 of course, but sadly it doesn't. It may be worth trying to recompile with LDC 1.3 and 2.072.2 druntime/Phobos (hopefully compatible) and see whether we have to blame the libs, the compiler or both. ;)

[kinke]

[Btw, I don't know how fair that'd be for your benchmark comparison, but compiling with -mcpu=native may give an additional boost.]

[jondegenhardt]

One specific possibility is that converting from char[] to double via std.conv.to could be slower. The "tsv-filter (numeric)" and "tsv-summarize" tests do this quite a bit. I ran some ad-hoc tests with same tools, but running operations that do not need to convert to double, and did not see a performance hit. I'll generate some more definitive tests when I get a chance.

The "csv2tsv" benchmark does not do any converting, or any numeric operations for that matter. It would be something different.

[jondegenhardt]

I tried the easy thing - Run DMD executables for 2.072.2 and 2.073.2 and see if there are similar regressions. There weren't. The "tsv-filter (numeric)" and "tsv-summarize" tests are actually faster. Still doesn't rule out Phobos changes as the cause.

Compilation	tsv-filter (numeric)	tsv-summarize	csv2tsv
DMD 2.072	6.15	25.32	55.38
DMD 2.073	5.54	18.31	55.14

[p0nce]

Tested it on some audio processing (30 measures, arch x86_64)

Results 1.3:
 * minimum time: 617 ms => 34.2472 x real-time
 * median  time: 620 ms => 34.0815 x real-time
 * average time: 625.767 ms => 33.7674 x real-time

Results 1.2:
 * minimum time: 613 ms => 34.4707 x real-time
 * median  time: 619.5 ms => 34.109 x real-time
 * average time: 624.667 ms => 33.8269 x real-time

Results 1.0:
 * minimum time: 633 ms => 33.3816 x real-time
 * median  time: 637.5 ms => 33.146 x real-time
 * average time: 646.567 ms => 32.6812 x real-time

Here it's a less than 1% regression so hard to tell.

[dnadlinger]

A 30% regression should be somewhat easy to localize by just comparing time profiles for the two programs.

[jondegenhardt]

@klickverbot Do you have suggestions for a profiling tool to use for this purpose? I tried looking at function call counts via the LDC -fprofile-instr-generate. Saw things I didn't expect, but no major differences between versions stood out. However, I'm not sure what tools are available to give insight into time allocation in an OSX LDC build.

[p0nce]

CodeXL is pretty great.

[dnadlinger]

On OS X, you can also simply use Xcode's Instruments for a rudimentary sampling profiler.