20+ minute precompilation regression on v0.6.1 #24383

ChrisRackauckas · 2017-10-28T20:02:56Z

From here: SciML/DifferentialEquations.jl#209

We see a >20 minute compile time regression, along with post-compilation using statements taking as long as precompilation used to take. @mauro3 noted that it may not occur on Linux though, but latest Travis tests seem to show that this regression is widespread.

The text was updated successfully, but these errors were encountered:

mauro3 · 2017-10-28T21:29:19Z

Chris narrowed it down to BoundaryValueDiffEq.jl. I ran the pre-compilation of for both Julia 0.6 versions and see a 4x regression:

Here the comparison of v0.6.0 and v0.6.1 of precompilation of BoundaryValueDiffEq.jl (with the __precompile__() statement active here):

              _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0 (2017-06-19 13:05 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-pc-linux-gnu

julia> @time using BoundaryValueDiffEq
INFO: Recompiling stale cache file /home/mauro/.julia/lib/v0.6/BoundaryValueDiffEq.ji for module BoundaryValueDiffEq.
315.972758 seconds (9.44 M allocations: 690.993 MiB, 0.06% gc time)

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.1 (2017-10-24 22:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-pc-linux-gnu

julia> @time using BoundaryValueDiffEq                                                                                                                                   
INFO: Recompiling stale cache file /home/mauro/.julia/lib/v0.6/BoundaryValueDiffEq.ji for module BoundaryValueDiffEq.                                                    
1120.472693 seconds (17.77 M allocations: 1.298 GiB, 0.06% gc time)

JeffBezanson · 2017-10-28T22:50:02Z

Also on master. I notice that BandedMatrices.ji is about 120MB, apparently due to excessive inference. I suspect #21677, as much as I like that change.

ChrisRackauckas · 2017-10-29T00:06:53Z

BoundaryValueDiffEq is the only library in the DiffEq stack that has BandedMatrices.jl, so that seems like that's where it is. I'll remove BoundaryValueDiffEq.jl for now from DifferentialEquations.jl to isolate the problem.

JeffBezanson · 2017-10-29T02:21:25Z

I tried bisecting this. It's a little tricky since it seems to have regressed in two steps, where BandedMatrices.ji went from 20MB to 80MB to 120MB. Unfortunately the culprit from bisecting is #23012. That was definitely a bugfix, which means the bug was an accidental optimization. Frankly, even 20MB is way too big for this package, so I suspect we're doing too much inference, and the bug caused us to skip saving a bunch of stuff that actually didn't matter. We end up spending lots of time in jl_recache_types (loading lots of types takes longer than saving them).

ChrisRackauckas · 2017-10-29T02:23:32Z

Pinging @dlfivefifty

dlfivefifty · 2017-10-29T08:14:55Z

Commenting out precompile fixes this issue, though that's probably not the ideal solution.

mauro3 · 2017-10-29T08:30:03Z

But a package which has __precompile__() in it, will then still precompile BandedMatrices.jl, unless you set __precompile__(false), but then any packages depending on it can not have precompilation enabled.

dlfivefifty · 2017-10-29T09:12:51Z

The issue is not the __precompile__(), it's the precompile.jl file with a list of functions to precompile (from SnoopCompile.jl).

dlfivefifty · 2017-10-29T09:13:15Z

So I meant comment out the include("precompile.jl") line

JeffBezanson · 2017-10-29T18:18:46Z

This is related to the unions of matrix types like StridedMatrix. One thing that fixes it is to set MAX_UNION_SPLITTING in inference to 1 (disabling it). Long term, it would be nice to replace those type unions. For now, we might be able to hack around it in inference.

dlfivefifty · 2017-10-29T18:47:57Z

I think StridedMatrix would work better as a trait. This would allow other, non-Base, matrix types to dispatch to the right BLAS routines.

reduce O(n^2) behavior of `jl_recache_types` these help #24383

reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

…24399) reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

tknopp · 2018-01-21T22:41:47Z

Is here anything that can be done?

I tracked down why Images.jl loads so long and its

julia> @time using FileIO
 25.040638 seconds (19.17 M allocations: 1.065 GiB, 1.07% gc time)

This is pretty long for such a core package like FileIO which cannot be avoided (I need Images.jl)

Is there any workaround?

tknopp · 2018-01-22T18:27:51Z

Switching back to Julia 0.6.0 solves the performance issue. Is this fixed on master? I know that 1.0 is about feature completeness but 25 seconds and 1GB allocations for a tiny package like FileIO sound more like a bug to me.

KristofferC · 2018-02-05T15:49:38Z

Can you try deleting your FileIO.ji precompilation file and try again.

tknopp · 2018-02-05T16:13:33Z

currently not since I changed to 0.6.0 and need that environment right now. I missed to cross-ref JuliaIO/FileIO.jl#156

ViralBShah · 2018-06-11T22:01:32Z

Is this still an issue? Perhaps an update?

tknopp · 2018-06-11T22:20:26Z

under 0.6.3

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.3 (2018-05-28 20:20 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin14.5.0

julia> @time using FileIO
  8.100726 seconds (4.97 M allocations: 279.204 MiB, 0.75% gc time)

so still way too slow. (if you were interested in 0.6.3 timing)

ViralBShah · 2018-06-12T18:40:20Z

Sorry for not being clear - is it an issue in 0.7-alpha too? Good to see that at least its better than the 25 seconds.

ViralBShah · 2018-06-12T18:45:41Z

Well can't really do 0.7 testing yet, because of the deprecation warnings. With all the deprecations being printed, I got 15 seconds on a fairly newish server.

ViralBShah · 2018-06-12T18:52:29Z

On the original issue posted here (BoundaryValueDiffEq), I got the whole thing in 0.6.3 in 64 seconds. I did a Pkg.add(), which brought in a bunch of dependent packages for the first time as well.

I propose closing this and tracking compile-time regressions in a new/existing 0.7/1.0 related issue.

JeffBezanson added the regression Regression in behavior compared to a previous version label Oct 28, 2017

ChrisRackauckas mentioned this issue Oct 29, 2017

Hanging precompilation JuliaLinearAlgebra/BandedMatrices.jl#39

Closed

JeffBezanson added a commit that referenced this issue Oct 29, 2017

only infer concrete union-split signatures to cut down on inference

880ca90

reduce O(n^2) behavior of `jl_recache_types` these help #24383

JeffBezanson mentioned this issue Oct 29, 2017

[release-0.6] help #24383 inference and load time regression #24399

Merged

JeffBezanson added a commit that referenced this issue Oct 29, 2017

only infer concrete union-split signatures to cut down on inference

2b4cb95

reduce O(n^2) behavior of `jl_recache_types` these help #24383

JeffBezanson added a commit that referenced this issue Oct 30, 2017

only infer concrete union-split signatures to cut down on inference

c3f5f5f

reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

JeffBezanson added a commit that referenced this issue Oct 30, 2017

only infer concrete union-split signatures to cut down on inference

931b322

reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

SimonDanisch mentioned this issue Nov 2, 2017

Very slow using FileIO on Julia v0.6.1 JuliaIO/FileIO.jl#156

Closed

dlfivefifty mentioned this issue Nov 3, 2017

add special type for views of Band, macro to implement banded linear algebra JuliaLinearAlgebra/BandedMatrices.jl#37

Merged

JeffBezanson added a commit that referenced this issue Nov 14, 2017

only infer concrete union-split signatures to cut down on inference

9e84757

reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

ararslan pushed a commit that referenced this issue Nov 14, 2017

only infer concrete union-split signatures to cut down on inference (#…

cf2ed66

…24399) reduce O(n^2) behavior of `jl_recache_types` improve soundness of inferred typevar bounds in subtyping these help #24383

JeffBezanson mentioned this issue Nov 21, 2017

[release-0.6] revert part of cf2ed662 to fix JuliaDB test failure #24678

Closed

JeffBezanson mentioned this issue Dec 26, 2017

Gadfly load time regression from 0.5 to 0.6. #25254

Closed

tknopp mentioned this issue Feb 6, 2018

Precompilation does not produce a deterministic result #25900

Open

tknopp mentioned this issue May 12, 2018

Performance issue with Winston #27083

Closed

ViralBShah added the compiler:latency Compiler latency label Jun 11, 2018

ViralBShah closed this as completed Jun 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

20+ minute precompilation regression on v0.6.1 #24383

20+ minute precompilation regression on v0.6.1 #24383

ChrisRackauckas commented Oct 28, 2017

mauro3 commented Oct 28, 2017 •

edited

Loading

JeffBezanson commented Oct 28, 2017

ChrisRackauckas commented Oct 29, 2017

JeffBezanson commented Oct 29, 2017

ChrisRackauckas commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

mauro3 commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

JeffBezanson commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

tknopp commented Jan 21, 2018

tknopp commented Jan 22, 2018

KristofferC commented Feb 5, 2018 •

edited

Loading

tknopp commented Feb 5, 2018

ViralBShah commented Jun 11, 2018

tknopp commented Jun 11, 2018

ViralBShah commented Jun 12, 2018

ViralBShah commented Jun 12, 2018 •

edited

Loading

ViralBShah commented Jun 12, 2018

20+ minute precompilation regression on v0.6.1 #24383

20+ minute precompilation regression on v0.6.1 #24383

Comments

ChrisRackauckas commented Oct 28, 2017

mauro3 commented Oct 28, 2017 • edited Loading

JeffBezanson commented Oct 28, 2017

ChrisRackauckas commented Oct 29, 2017

JeffBezanson commented Oct 29, 2017

ChrisRackauckas commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

mauro3 commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

JeffBezanson commented Oct 29, 2017

dlfivefifty commented Oct 29, 2017

tknopp commented Jan 21, 2018

tknopp commented Jan 22, 2018

KristofferC commented Feb 5, 2018 • edited Loading

tknopp commented Feb 5, 2018

ViralBShah commented Jun 11, 2018

tknopp commented Jun 11, 2018

ViralBShah commented Jun 12, 2018

ViralBShah commented Jun 12, 2018 • edited Loading

ViralBShah commented Jun 12, 2018

mauro3 commented Oct 28, 2017 •

edited

Loading

KristofferC commented Feb 5, 2018 •

edited

Loading

ViralBShah commented Jun 12, 2018 •

edited

Loading