Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: InitError: LoadError: Can't differentiate loopinfo expression #393

Closed
chriselrod opened this issue Nov 6, 2019 · 10 comments · Fixed by #453
Closed

ERROR: InitError: LoadError: Can't differentiate loopinfo expression #393

chriselrod opened this issue Nov 6, 2019 · 10 comments · Fixed by #453

Comments

@chriselrod
Copy link

chriselrod commented Nov 6, 2019

This is on a 3 day old master branch of Julia:

Julia Version 1.4.0-DEV.427
Commit c7e4b9929b* (2019-11-02 09:58 UTC)

And occurs when using Zygote:

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
ERROR: InitError: LoadError: Can't differentiate loopinfo expression
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion [inlined]
 [3] copyto! at ./broadcast.jl:908 [inlined]
 [4] copyto! at ./broadcast.jl:863 [inlined]
 [5] (::typeof((copyto!)))(::FillArrays.Fill{Float32,1,Tuple{Base.OneTo{Int64}}}) at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/compiler/interface2.jl:0
 [6] copy [inlined]
 [7] materialize at ./broadcast.jl:819 [inlined]
 [8] #1487 at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/precompile.jl:22 [inlined]
 [9] (::typeof((#1487)))(::Float32) at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/compiler/interface2.jl:0
 [10] (::Zygote.var"#28#29"{typeof((#1487))})(::Float32)
 [11] gradient(::Function, ::Array{Float64,1})
 [12] top-level scope at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/precompile.jl:22
 [13] include(::Module, ::String) at ./Base.jl:377
 [14] include [inlined]
 [15] precompile at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/Zygote.jl:43 [inlined]
 [16] (::Zygote.var"#1477#1478")() at /home/chriselrod/.julia/packages/Requires/9Jse8/src/init.jl:11
 [17] __init__()
 [18] _include_from_serialized(::String, ::Array{Any,1})
 [19] _require_from_serialized(::String)
 [20] _require(::Base.PkgId)
 [21] require(::Base.PkgId) (repeats 2 times)
in expression starting at /home/chriselrod/.julia/packages/Zygote/mMAdj/src/precompile.jl:22
during initialization of module Zygote

The error message from #229 sounds similar, but am guessing they're different problems. Feel free to close if this is a duplicate.

EDIT:
The error also shows up here: https://travis-ci.org/FluxML/Zygote.jl/jobs/608155890

@vchuravy
Copy link

vchuravy commented Nov 6, 2019

should be handled the same as the Expr(:simdloop before

@MikeInnes
Copy link
Member

We aren't handling simdloop at all currently. Something else must have changed on master to make us differentiate into the broadcast machinery, but not sure what.

@IanButterworth
Copy link
Contributor

If another report is helpful.. I'm also seeing this with Zygote v0.4.1 on nightly

Julia Version 1.4.0-DEV.527
Commit 6d26f14ede (2019-11-26 02:01 UTC)

https://travis-ci.com/r3tex/ObjectDetector.jl

@DilumAluthge
Copy link
Member

Bump @MikeInnes Is there any update on this?

This issue is causing all packages that depend on Zygote to fail the daily PkgEval builds.


Here is the log output that I get when I try to run import Zygote on Julia master and Zygote#master.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0-DEV.622 (2019-12-17)
 _/ |\__'_|_|_|\__'_|  |  Commit b73fc63796 (0 days old master)
|__/                   |

(@v1.4) pkg> add Zygote#master
   Cloning default registries into `~/.julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `~/.julia/registries/General`
   Cloning git-repo `https://github.com/FluxML/Zygote.jl.git`
  Updating git-repo `https://github.com/FluxML/Zygote.jl.git`
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed Reexport ───────────── v0.2.0
 Installed NaNMath ────────────── v0.3.3
 Installed AbstractFFTs ───────── v0.5.0
 Installed OrderedCollections ─── v1.1.0
 Installed FFTW ───────────────── v1.1.0
 Installed DataStructures ─────── v0.17.6
 Installed BinaryProvider ─────── v0.5.8
 Installed JSON ───────────────── v0.21.0
 Installed Requires ───────────── v0.5.2
 Installed TimerOutputs ───────── v0.5.3
 Installed Conda ──────────────── v1.3.0
 Installed DiffRules ──────────── v0.1.0
 Installed OpenSpecFun_jll ────── v0.5.3+1
 Installed ZygoteRules ────────── v0.2.0
 Installed StaticArrays ───────── v0.12.1
 Installed IRTools ────────────── v0.3.0
 Installed CommonSubexpressions ─ v0.2.0
 Installed DiffResults ────────── v0.0.4
 Installed Parsers ────────────── v0.3.10
 Installed FillArrays ─────────── v0.8.2
 Installed VersionParsing ─────── v1.2.0
 Installed NNlib ──────────────── v0.6.0
 Installed SpecialFunctions ───── v0.9.0
 Installed MacroTools ─────────── v0.5.3
 Installed ForwardDiff ────────── v0.10.7
 Installed Compat ─────────────── v2.2.0
  Updating `~/.julia/environments/v1.4/Project.toml`
  [e88e6eb3] + Zygote v0.4.1 #master (https://github.com/FluxML/Zygote.jl.git)
  Updating `~/.julia/environments/v1.4/Manifest.toml`
  [621f4979] + AbstractFFTs v0.5.0
  [b99e7846] + BinaryProvider v0.5.8
  [bbf7d656] + CommonSubexpressions v0.2.0
  [34da2185] + Compat v2.2.0
  [8f4d0f93] + Conda v1.3.0
  [864edb3b] + DataStructures v0.17.6
  [163ba53b] + DiffResults v0.0.4
  [b552c78f] + DiffRules v0.1.0
  [7a1cc6ca] + FFTW v1.1.0
  [1a297f60] + FillArrays v0.8.2
  [f6369f11] + ForwardDiff v0.10.7
  [7869d1d1] + IRTools v0.3.0
  [682c06a0] + JSON v0.21.0
  [1914dd2f] + MacroTools v0.5.3
  [872c559c] + NNlib v0.6.0
  [77ba4419] + NaNMath v0.3.3
  [efe28fd5] + OpenSpecFun_jll v0.5.3+1
  [bac558e1] + OrderedCollections v1.1.0
  [69de0a69] + Parsers v0.3.10
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [276daf66] + SpecialFunctions v0.9.0
  [90137ffa] + StaticArrays v0.12.1
  [a759f4b9] + TimerOutputs v0.5.3
  [81def892] + VersionParsing v1.2.0
  [e88e6eb3] + Zygote v0.4.1 #master (https://github.com/FluxML/Zygote.jl.git)
  [700de1a5] + ZygoteRules v0.2.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Building Conda → `~/.julia/packages/Conda/kLXeC/deps/build.log`
Path `/Users/dilum/.julia/packages/Conda/kLXeC` exists and looks like the correct package. Using existing path.
  Updating `/private/var/folders/48/n09p39y53bx6dnbnf244s2jc0000gp/T/jl_jHcKWS/Project.toml`
  [8f4d0f93] + Conda v1.3.0 [`~/.julia/packages/Conda/kLXeC`]
  Updating `/private/var/folders/48/n09p39y53bx6dnbnf244s2jc0000gp/T/jl_jHcKWS/Manifest.toml`
  [8f4d0f93] ~ Conda v1.3.0 ⇒ v1.3.0 [`~/.julia/packages/Conda/kLXeC`]
  Building FFTW ─→ `~/.julia/packages/FFTW/loJ3F/deps/build.log`
Path `/Users/dilum/.julia/packages/FFTW/loJ3F` exists and looks like the correct package. Using existing path.
  Updating `/private/var/folders/48/n09p39y53bx6dnbnf244s2jc0000gp/T/jl_Vhqlgz/Project.toml`
  [7a1cc6ca] + FFTW v1.1.0 [`~/.julia/packages/FFTW/loJ3F`]
  Updating `/private/var/folders/48/n09p39y53bx6dnbnf244s2jc0000gp/T/jl_Vhqlgz/Manifest.toml`
  [7a1cc6ca] ~ FFTW v1.1.0 ⇒ v1.1.0 [`~/.julia/packages/FFTW/loJ3F`]

julia> using Zygote
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
┌ Warning: Deprecated syntax `(;)` at /Users/dilum/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:148.
│ Use `begin end` instead.
└ @ ~/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:148
┌ Warning: Deprecated syntax `(;)` at /Users/dilum/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:157.
│ Use `begin end` instead.
└ @ ~/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:157
┌ Warning: Deprecated syntax `(;)` at /Users/dilum/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:167.
│ Use `begin end` instead.
└ @ ~/.julia/packages/IRTools/Fl3dY/src/passes/relooper.jl:167
┌ Warning: Deprecated syntax `(;)` at /Users/dilum/.julia/packages/Zygote/fUnhe/src/lib/grad.jl:3.
│ Use `begin end` instead.
└ @ ~/.julia/packages/Zygote/fUnhe/src/lib/grad.jl:3
ERROR: InitError: LoadError: Can't differentiate loopinfo expression
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./simdloop.jl:79 [inlined]
 [3] copyto! at ./broadcast.jl:908 [inlined]
 [4] copyto! at ./broadcast.jl:863 [inlined]
 [5] (::typeof(∂(copyto!)))(::FillArrays.Fill{Float32,1,Tuple{Base.OneTo{Int64}}}) at /Users/dilum/.julia/packages/Zygote/fUnhe/src/compiler/interface2.jl:0
 [6] copy at ./broadcast.jl:839 [inlined]
 [7] materialize at ./broadcast.jl:819 [inlined]
 [8] #1716 at /Users/dilum/.julia/packages/Zygote/fUnhe/src/precompile.jl:22 [inlined]
 [9] (::typeof(∂(#1716)))(::Float32) at /Users/dilum/.julia/packages/Zygote/fUnhe/src/compiler/interface2.jl:0
 [10] (::Zygote.var"#28#29"{typeof(∂(#1716))})(::Float32) at /Users/dilum/.julia/packages/Zygote/fUnhe/src/compiler/interface.jl:38
 [11] gradient(::Function, ::Array{Float64,1}) at /Users/dilum/.julia/packages/Zygote/fUnhe/src/compiler/interface.jl:47
 [12] top-level scope at /Users/dilum/.julia/packages/Zygote/fUnhe/src/precompile.jl:22
 [13] include(::Module, ::String) at ./Base.jl:377
 [14] include at /Users/dilum/.julia/packages/Zygote/fUnhe/src/Zygote.jl:1 [inlined]
 [15] precompile at /Users/dilum/.julia/packages/Zygote/fUnhe/src/Zygote.jl:43 [inlined]
 [16] (::Zygote.var"#1706#1707")() at /Users/dilum/.julia/packages/Requires/9Jse8/src/init.jl:11
 [17] __init__() at /Users/dilum/.julia/packages/Requires/9Jse8/src/init.jl:18
 [18] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:690
 [19] _require_from_serialized(::String) at ./loading.jl:741
 [20] _require(::Base.PkgId) at ./loading.jl:1032
 [21] require(::Base.PkgId) at ./loading.jl:920
 [22] require(::Module, ::Symbol) at ./loading.jl:915
in expression starting at /Users/dilum/.julia/packages/Zygote/fUnhe/src/precompile.jl:22
during initialization of module Zygote

Julia version info:

Julia Version 1.4.0-DEV.622
Commit b73fc63796 (2019-12-17 04:07 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.2.0)
  CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, haswell)

@MikeInnes
Copy link
Member

You can probably fix the immediate issue by adding :loopinfo here; I'm pretty sure this is just SIMD metadata that we can ignore.

It's a bit surprising that we're hitting such metadata for any of the functions we compile on startup, though. If something has changed in base to cause that, that might need looking into as well.

@andreasnoack
Copy link
Contributor

After adding :loopinfo to the list, as suggested, I get

julia> using Zygote
ERROR: InitError: LoadError: Mutating arrays is not supported
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] (::Zygote.var"#994#995")(::Nothing) at /Users/andreasnoack/.julia/dev/Zygote/src/lib/array.jl:44
 [3] (::Zygote.var"#2646#back#996"{Zygote.var"#994#995"})(::Nothing) at /Users/andreasnoack/.julia/packages/ZygoteRules/6nssF/src/adjoint.jl:49
 [4] macro expansion at ./broadcast.jl:909 [inlined]
 [5] macro expansion at ./simdloop.jl:77 [inlined]
 [6] copyto! at ./broadcast.jl:908 [inlined]
 [7] copyto! at ./broadcast.jl:863 [inlined]
 [8] (::typeof((copyto!)))(::FillArrays.Fill{Float32,1,Tuple{Base.OneTo{Int64}}}) at /Users/andreasnoack/.julia/dev/Zygote/src/compiler/interface2.jl:0
 [9] copy at ./broadcast.jl:839 [inlined]
 [10] materialize at ./broadcast.jl:819 [inlined]
 [11] #1734 at /Users/andreasnoack/.julia/dev/Zygote/src/precompile.jl:22 [inlined]
 [12] (::typeof((#1734)))(::Float32) at /Users/andreasnoack/.julia/dev/Zygote/src/compiler/interface2.jl:0
 [13] (::Zygote.var"#26#27"{typeof((#1734))})(::Float32) at /Users/andreasnoack/.julia/dev/Zygote/src/compiler/interface.jl:38
 [14] gradient(::Function, ::Array{Float64,1}) at /Users/andreasnoack/.julia/dev/Zygote/src/compiler/interface.jl:47
 [15] top-level scope at /Users/andreasnoack/.julia/dev/Zygote/src/precompile.jl:22
 [16] include(::Module, ::String) at ./Base.jl:377
 [17] include at /Users/andreasnoack/.julia/dev/Zygote/src/Zygote.jl:1 [inlined]
 [18] precompile at /Users/andreasnoack/.julia/dev/Zygote/src/Zygote.jl:43 [inlined]
 [19] (::Zygote.var"#1724#1725")() at /Users/andreasnoack/.julia/packages/Requires/9Jse8/src/init.jl:11
 [20] __init__() at /Users/andreasnoack/.julia/packages/Requires/9Jse8/src/init.jl:18
 [21] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
 [22] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781
 [23] _require(::Base.PkgId) at ./loading.jl:1006
 [24] require(::Base.PkgId) at ./loading.jl:927
 [25] require(::Module, ::Symbol) at ./loading.jl:922
in expression starting at /Users/andreasnoack/.julia/dev/Zygote/src/precompile.jl:22
during initialization of module Zygote

@MikeInnes
Copy link
Member

Great, that's the next error you expect if we're hitting some @simdd inner loop that we shouldn't be, and the stacktrace shows that this is coming from materialize.

The way our broadcast works is that it explicitly intercepts broadcasted calls to make them eager (where normally they would just build up a Broadcasted wrapper that gets copied by materialize). If that's working materialize should just be a no-op, so the problem here is that calls like broadcasted(*, a, b) aren't getting properly intercepted. An internal change to the way broadcasted dispatches is most likely to be the culprit.

@andreasnoack
Copy link
Contributor

It appears that this broke with

e39c4d8daa6cab46d699452db08f39f4792ce00f is the first bad commit
commit e39c4d8daa6cab46d699452db08f39f4792ce00f
Author: Keno Fischer <keno@juliacomputing.com>
Date:   Sat Oct 12 14:20:23 2019 -0400

    Add a builtin that allows specifying which iterate method to use (#33356)

    When using the Casette mechanism to intercept calls to _apply,
    a common strategy is to rewrite the function argument to properly
    consider the context and then falling back to regular _apply.
    However, as showin in https://github.com/jrevels/Cassette.jl/issues/146,
    this strategy is insufficient as the _apply itself may recurse into
    various `iterate` calls which are not properly tracked. This is an
    attempt to resolve this problem with a minimal performance penalty.
    Attempting to duplicate the _apply logic in julia, would lead to
    code that is very hard for inference (and nested Cassette passes to
    understand). In contrast, this simply adds a version of _apply that
    takes `iterate` as an explicit argument. Cassette and similar tools
    can override this argument and provide a function that properly
    allows the context to recurse through the iteration, while still
    allowing inference to take advantage of the special handling of _apply
    for simple cases.

    Also change the lowering of splatting to use this new intrinsic directly,
    thus fixing #26001.

@MikeInnes
Copy link
Member

Ok, interesting, so it sounds like a change to how splatting works has caused this, not a change to user code in broadcast. I'll take a look at that.

@vchuravy
Copy link

vchuravy commented Jan 7, 2020

Zygote should probably use Core.Compiler.is_meta_expr_head to check whether the expression is relevant.

MikeInnes added a commit that referenced this issue Jan 10, 2020
@MikeInnes MikeInnes mentioned this issue Jan 10, 2020
bors bot added a commit that referenced this issue Jan 10, 2020
453: 1.4 updates r=MikeInnes a=MikeInnes

fixes #393, partially addresses #449 (this doesn't fully support apply_iterate but should address the most common cases).

bors r+

Co-authored-by: Mike Innes <mike.j.innes@gmail.com>
bors bot added a commit that referenced this issue Jan 10, 2020
453: 1.4 updates r=MikeInnes a=MikeInnes

fixes #393, partially addresses #449 (this doesn't fully support apply_iterate but should address the most common cases).

bors r+

Co-authored-by: Mike Innes <mike.j.innes@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants