Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn about cycles in extension precompilation #53882

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Mar 28, 2024

Currently the reason for these errors isn't clear.
If we can't avoid the error, at least explain where the cycle is so the user has a chance to fix it.

#53112 is the original version that instead tried to succeed.

Helps #53081
May help #52511

@IanButterworth IanButterworth added the packages Package management and loading label Mar 28, 2024
@IanButterworth
Copy link
Member Author

Ok, I'm confused now. This is detecting and warning about the cycle, but it's not causing the @errors seen in #53081

Screenshot 2024-03-28 at 10 51 58 AM

Any idea what's changed here @KristofferC ?

@IanButterworth
Copy link
Member Author

Ok, I don't know why the FillArrays Ext cycle doesn't error, but the reason that this example didn't @error in the same way as #53081 is that Symbolics has since worked around the issue (as of v5.17.0)

(@v1.12) pkg> add Symbolics@v5.16.1
   Resolving package versions...
   Installed DomainSets ─ v0.6.7
   Installed Symbolics ── v5.16.1
    Updating `~/.julia/environments/v1.12/Project.toml`
⌃ [0c5d862f] ↓ Symbolics v5.26.0 ⇒ v5.16.1
    Updating `~/.julia/environments/v1.12/Manifest.toml`
⌅ [5b8099bc] ↓ DomainSets v0.7.10 ⇒ v0.6.7
  [19f23fe9] - SymbolicLimits v0.2.0
⌃ [0c5d862f] ↓ Symbolics v5.26.0 ⇒ v5.16.1
        Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`
Precompiling all packages...
  4 dependencies successfully precompiled in 21 seconds. 136 already precompiled.
  2 dependencies had output during precompilation:
┌ Symbolics → SymbolicsPreallocationToolsExt
│  ┌ Warning: Dependency cycle detected in extension precompilation: SymbolicsPreallocationToolsExt > SymbolicsForwardDiffExt > SymbolicsPreallocationToolsExt
│  │ Loading SymbolicsPreallocationToolsExt here will likely fail.
│  └ @ Base loading.jl:1488
│  ┌ Warning: Module SymbolicsPreallocationToolsExt with build ID ffffffff-ffff-ffff-000a-aab1fa523737 is missing from the cache.
│  │ This may mean SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2359
│  ┌ Error: Error during loading of extension SymbolicsPreallocationToolsExt of Symbolics, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2363
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2216
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2207 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2206 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1466
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1507
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1329
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2223
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2207
│  │     [14] macro expansion
│  │        @ ./loading.jl:99 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:17 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2104
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2097
│  │     [20] include
│  │        @ ./Base.jl:559 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2719
│  │     [22] top-level scope
│  │        @ stdin:4
│  │     [23] eval
│  │        @ ./boot.jl:432 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2541
│  │     [25] include_string
│  │        @ ./loading.jl:2551 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:255
│  │     [27] _start()
│  │        @ Base ./client.jl:533
│  └ @ Base loading.jl:1472
└  
┌ Symbolics → SymbolicsForwardDiffExt
│  ┌ Warning: Dependency cycle detected in extension precompilation: SymbolicsForwardDiffExt > SymbolicsPreallocationToolsExt > SymbolicsForwardDiffExt
│  │ Loading SymbolicsForwardDiffExt here will likely fail.
│  └ @ Base loading.jl:1488
│  ┌ Warning: Module SymbolicsForwardDiffExt with build ID ffffffff-ffff-ffff-000a-aab2f9fdcc32 is missing from the cache.
│  │ This may mean SymbolicsForwardDiffExt [4a213a23-c09c-5cde-9712-b631ad2c72df] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2359
│  ┌ Error: Error during loading of extension SymbolicsForwardDiffExt of Symbolics, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2363
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2216
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2207 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2206 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1466
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1507
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1329
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2223
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2207
│  │     [14] macro expansion
│  │        @ ./loading.jl:99 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:17 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2104
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2097
│  │     [20] include
│  │        @ ./Base.jl:559 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2719
│  │     [22] top-level scope
│  │        @ stdin:4
│  │     [23] eval
│  │        @ ./boot.jl:432 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2541
│  │     [25] include_string
│  │        @ ./loading.jl:2551 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:255
│  │     [27] _start()
│  │        @ Base ./client.jl:533
│  └ @ Base loading.jl:1472
└  

@IanButterworth IanButterworth added backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11 labels Mar 28, 2024
@IanButterworth
Copy link
Member Author

@KristofferC see a380337 (#53882) for why it was warning for FillArrays. I don't quite understand the increment decrement code around there, but it seemed to be in the wrong place.

Now seems to work

(@v1.12) pkg> add Symbolics@v5.16.1 SymbolicIndexingInterface@v0.3.12
Precompiling all packages...
  10 dependencies successfully precompiled in 31 seconds. 134 already precompiled.
  2 dependencies had output during precompilation:
┌ Symbolics → SymbolicsPreallocationToolsExt
│  ┌ Warning: Dependency cycle detected in extension precompilation: SymbolicsPreallocationToolsExt > SymbolicsForwardDiffExt > SymbolicsPreallocationToolsExt
│  │ Loading SymbolicsPreallocationToolsExt here will likely fail.
│  └ @ Base loading.jl:1502
│  ┌ Warning: Module SymbolicsPreallocationToolsExt with build ID ffffffff-ffff-ffff-0001-e62664c25e87 is missing from the cache.
│  │ This may mean SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2401
│  ┌ Error: Error during loading of extension SymbolicsPreallocationToolsExt of Symbolics, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2405
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2258
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2249 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2248 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1466
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1507
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1329
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2265
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2249
│  │     [14] macro expansion
│  │        @ ./loading.jl:141 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:17 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2146
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2139
│  │     [20] include
│  │        @ ./Base.jl:559 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2761
│  │     [22] top-level scope
│  │        @ stdin:4
│  │     [23] eval
│  │        @ ./boot.jl:432 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2583
│  │     [25] include_string
│  │        @ ./loading.jl:2593 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:255
│  │     [27] _start()
│  │        @ Base ./client.jl:533
│  └ @ Base loading.jl:1472
└  
┌ Symbolics → SymbolicsForwardDiffExt
│  ┌ Warning: Dependency cycle detected in extension precompilation: SymbolicsForwardDiffExt > SymbolicsPreallocationToolsExt > SymbolicsForwardDiffExt
│  │ Loading SymbolicsForwardDiffExt here will likely fail.
│  └ @ Base loading.jl:1502
│  ┌ Warning: Module SymbolicsForwardDiffExt with build ID ffffffff-ffff-ffff-0001-e6275bdee7f5 is missing from the cache.
│  │ This may mean SymbolicsForwardDiffExt [4a213a23-c09c-5cde-9712-b631ad2c72df] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2401
│  ┌ Error: Error during loading of extension SymbolicsForwardDiffExt of Symbolics, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2405
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2258
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2249 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2248 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1466
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1507
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1329
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2265
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2249
│  │     [14] macro expansion
│  │        @ ./loading.jl:141 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:17 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2146
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:1064 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:1061 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2139
│  │     [20] include
│  │        @ ./Base.jl:559 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2761
│  │     [22] top-level scope
│  │        @ stdin:4
│  │     [23] eval
│  │        @ ./boot.jl:432 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2583
│  │     [25] include_string
│  │        @ ./loading.jl:2593 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:255
│  │     [27] _start()
│  │        @ Base ./client.jl:533
│  └ @ Base loading.jl:1472
└  

This was referenced Apr 9, 2024
@KristofferC KristofferC mentioned this pull request Apr 17, 2024
59 tasks
@KristofferC KristofferC mentioned this pull request May 8, 2024
23 tasks
@IanButterworth
Copy link
Member Author

@KristofferC I believe we can close this thanks to #53972

@IanButterworth IanButterworth deleted the ib/ext_precomp_cycle_warn branch May 23, 2024 02:21
@fingolfin fingolfin removed backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11 labels May 23, 2024
KristofferC added a commit that referenced this pull request May 28, 2024
Backported PRs:
- [x] #53665 <!-- use afoldl instead of tail recursion for tuples -->
- [x] #53976 <!-- LinearAlgebra: LazyString in interpolated error
messages -->
- [x] #54005 <!-- make `view(::Memory, ::Colon)` produce a Vector -->
- [x] #54010 <!-- Overload `Base.literal_pow` for `AbstractQ` -->
- [x] #54069 <!-- Allow PrecompileTools to see MI's inferred by foreign
abstract interpreters -->
- [x] #53750 <!-- inference correctness: fields and globals can revert
to undef -->
- [x] #53984 <!-- Profile: fix heap snapshot is valid char check -->
- [x] #54102 <!-- Explicitly compute stride in unaliascopy for SubArray
-->
- [x] #54070 <!-- Fix integer overflow in `skip(s::IOBuffer,
typemax(Int64))` -->
- [x] #54013 <!-- Support case-changes to Annotated{String,Char}s -->
- [x] #53941 <!-- Fix writing of AnnotatedChars to AnnotatedIOBuffer -->
- [x] #54137 <!-- Fix typo in docs for `partialsortperm` -->
- [x] #54129 <!-- use correct size when creating output data from an
IOBuffer -->
- [x] #54153 <!-- Fixup IdSet docstring -->
- [x] #54143 <!-- Fix `make install` from tarballs -->
- [x] #54151 <!-- LinearAlgebra: Correct zero element in
`_generic_matvecmul!` for block adj/trans -->
- [x] #54213 <!-- Add `public` statement to `Base.GC` -->
- [x] #54222 <!-- Utilize correct tbaa when emitting stores of unions.
-->
- [x] #54233 <!-- set MAX_OS_WRITE on unix -->
- [x] #54255 <!-- fix `_checked_mul_dims` in the presence of 0s and
overflow. -->
- [x] #54259 <!-- Fix typo in `readuntil` -->
- [x] #54251 <!-- fix typo in gc_mark_memory8 when chunking a large
array -->
- [x] #54276 <!-- Fix solve for complex `Hermitian` with non-vanishing
imaginary part on diagonal -->
- [x] #54248 <!-- ensure package callbacks are invoked when no valid
precompile file exists for an "auto loaded" stdlib -->
- [x] #54308 <!-- Implement eval-able AnnotatedString 2-arg show -->
- [x] #54302 <!-- Specialised substring equality for annotated strs -->
- [x] #54243 <!-- prevent `package_callbacks` to run multiple time for a
single package -->
- [x] #54350 <!-- add a precompile signature to Artifacts code that is
used by JLLs -->
- [x] #54331 <!-- correctly track freed bytes in
jl_genericmemory_to_string -->
- [x] #53509 <!-- revert moving "creating packages" from Pkg.jl -->
- [x] #54335 <!-- When accessing the data pointer for an array, first
decay it to a Derived Pointer -->
- [x] #54239 <!-- Make sure `fieldcount` constant-folds for `Tuple{...}`
-->
- [x] #54288
- [x] #54067
- [x] #53715 <!-- Add read/write specialisation for IOContext{AnnIO} -->
- [x] #54289 <!-- Rework annotation ordering/optimisations -->
- [x] #53815 <!-- create phantom task for GC threads -->
- [x] #54130 <!-- inference: handle `LimitedAccuracy` in
`handle_global_assignment!` -->
- [x] #54428 <!-- Move ConsoleLogging.jl into Base -->
- [x] #54332 <!-- Revert "add unsetindex support to more copyto methods
(#51760)" -->
- [x] #53826 <!-- Make all command-line options documented in all
related files -->
- [x] #54465 <!-- typeintersect: conservative typevar subtitution during
`finish_unionall` -->
- [x] #54514 <!-- typeintersect: followup cleanup for the nothrow path
of type instantiation -->
- [x] #54499 <!-- make `@doc x` work without REPL loaded -->
- [x] #54210 <!-- attach finalizer in `mmap` to the correct object -->
- [x] #54359 <!-- Pkg REPL: cache `pkg_mode` lookup -->

Non-merged PRs with backport label:
- [ ] #54471 <!-- Actually setup jit targets when compiling
packageimages instead of targeting only one -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #54323 <!-- inference: fix too conservative effects for recursive
cycles -->
- [ ] #54322 <!-- effects: add new `@consistent_overlay` macro -->
- [ ] #54191 <!-- make `AbstractPipe` public -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #53882 <!-- Warn about cycles in extension precompilation -->
- [ ] #53707 <!-- Make ScopedValue public -->
- [ ] #53452 <!-- RFC: allow Tuple{Union{}}, returning Union{} -->
- [ ] #53402 <!-- Add `jl_getaffinity` and `jl_setaffinity` -->
- [ ] #53286 <!-- Raise an error when using `include_dependency` with
non-existent file or directory -->
- [ ] #52694 <!-- Reinstate similar for AbstractQ for backward
compatibility -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packages Package management and loading
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants