Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Massive load time for Julia 1.3+ #33

Closed
colinxs opened this issue Nov 15, 2019 · 5 comments
Closed

Massive load time for Julia 1.3+ #33

colinxs opened this issue Nov 15, 2019 · 5 comments

Comments

@colinxs
Copy link

colinxs commented Nov 15, 2019

First off, awesome package! I've really enjoyed it so far. Unfortunately I came across an issue with load time while trying out Julia 1.3.

In the following example:

f = tempname() * ".jlso"
JLSO.save(f, :x=>rand(1000), :y=>rand(1000)) 
x = JLSO.load(f)

The first call to JLSO.load(f) takes 1-2 minutes with Julia 1.3 (rc2, rc3, and rc4.1) on multiple machines, but only a few seconds with Julia 1.2. This is true for any level of compression or serializer. The difference appears to only be from compile time, as subsequent loads for other files are fast.

@oxinabox
Copy link
Member

Julia 1.3 has not been released yet, because of these kinds of issues.

@rofinn
Copy link
Member

rofinn commented Nov 17, 2019

Looks like this might be fixed on 1.3-rc5 JuliaLang/julia#33615

@colinxs
Copy link
Author

colinxs commented Nov 17, 2019

Looks like RC5 fixed it! In a fresh session:

julia> f = tempname() * ".jlso";
julia> JLSO.save(f, :x=>rand(1000), :y=>rand(1000));
julia> @time JLSO.load(f);
  2.822424 seconds (16.27 M allocations: 781.691 MiB, 9.60% gc time)

@colinxs colinxs closed this as completed Nov 17, 2019
@colinxs
Copy link
Author

colinxs commented Nov 17, 2019

I forgot to add: the slowdown came from _upgrade_env(pkgs::Dict) in deprecations.jl, and unfortunately there's still a relatively high penalty from it. Without it:

julia> @time JLSO.load(f);
1.372000 seconds (4.57 M allocations: 234.462 MiB, 2.11% gc time)

@rofinn
Copy link
Member

rofinn commented Nov 18, 2019

Is that also still a 1-time cost or is it twice as expensive every time you call JLSO.load?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants