Tests take twice as much time to run with USE_ORCJIT

[nalimilan]

When USE_ORCJIT is defined, tests take about twice more time to run on Fedora and RHEL. This happens both with a vanilla Julia build (including all in-tree dependencies) and when building RPM packages (with USE_SYSTEM_*=1 and LLVM 3.7.1).

What's curious is that USE_ORCJIT reduces RAM usage a lot, but apparently this isn't enough to make it faster.

See the timings in the two logs, which were run on my laptop:
With USE_ORCJIT: https://gist.github.com/nalimilan/ed141ce0153b264958d6
Without USE_ORCJIT: https://gist.github.com/nalimilan/4e2e728cf7d8569dd64c
(Note that I had to kill the latter in the middle of the testsuite as I only have 6GB RAM and the system started swapping heavily. I could run everything again with a maxrss limit if it's useful.)

Am I the only one to observe this dramatic difference? It would be surprising that I see this on 4 different distribution versions on both 32- and 64-bit, but that nobody else would see it.

[Keno]

Twice is suspicious. When I tested this ORCJIT was definitely faster than MCJIT. Maybe somehow we're running everything twice now?

[Keno]

Oh, also are you doing a debug or a release build (of julia, not LLVM)?

[nalimilan]

When building the RPMs, I do both, but the logs shown above are just the result of make clean && make && make test.

[nalimilan]

Am I the only one to see this? I've noticed the Travis queue is particularly busy these days, could it be related?

[Keno]

I'll take a look at this in a bit, but I wouldn't be surprised if all that's happening is that somehow the code path that adds the LLVM optimizations got dropped from the MCJIT case, so what you're really seeing is that MCJIT isn't running any optimizations.

[nalimilan]

@Keno Have you had the chance to look into this?

[Keno]

No I haven't, but the codegen was also recently rewritten again so it would be good to see if the same phenomenon can still be observed.

[nalimilan]

The timings when building RPMs are still the same AFAICT.

[nalimilan]

Spoke too soon, there's been a noticeable improvement yesterday (right after I started the build I was talking about). Likely #15609.

That said, while some tests run even faster than without USE_ORCJIT before (timings above), some still take more time: e.g. the penalty associated with USE_ORCJIT seems to be half of what it was before for linalg/triangular. So overall, the tests run much faster, but still not as fast as before.

Unfortunately, I can't test master without USE_ORCJIT as the build fails (#15688).

[nalimilan]

Now that the build works on master without USE_ORCJIT, I've retried the comparison. And the difference seems to have vanished:

• With ORCJIT: https://gist.github.com/nalimilan/6817297213460e2dac08c29950b6e4d8
• Without ORCJIT: https://gist.github.com/nalimilan/92422879ee7854bd6b6be6652b7a3fe5

Overall it looks like the tests are a bit slower than before (as I noted above), but there's no consistent pattern. Anyway, doesn't look like it's related to USE_ORCJIT now.