Compatibility for optional dependencies

[tkf]

Currently, [compat] information for packages listed in [extras] is ignored (see MWE below). It would be nice if they are handled so that:

• Packages using Requires.jl can add the optional dependencies in [extras] and then can impose compatibility.

  It resolves situations like Pyplot API update (fix #1932) JuliaPlots/Plots.jl#1956 where Plots devs cannot update PyPlot API without causing confusion https://discourse.julialang.org/t/23510.

  It also has a minor cosmetic effect that @require can read the UUID from Project.toml so that the UUID does not have to be in the source code.

• I also proposed a new framework for optional dependency management https://discourse.julialang.org/t/23526/41. Again, compatibility information for packages not in [deps] would help.

I just noticed that Julia 1.2 will ditch [extras] and go with test/Project.toml. So, any feature requests touching [extras] may be out of core devs' interest. However, it seems there is no solid plan for dealing with optional dependencies (IIUC #1085 is still in a "brainstorming" phase). Would it be useful to repurpose [extras] for run-time (in contrast to test-time) optional dependency constraint?

It is also somewhat related to #895 but I believe it is reasonably different to open a new issue.

MWE, just in case:

shell> cat upstream/Project.toml
name = "upstream"
uuid = "88ecca56-6ad2-11e9-2834-950dc93fed92"

[deps]
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

[compat]
JSON = "= 0.19"
VersionParsing = "= 1.1.1"

[extras]
VersionParsing = "81def892-9a0e-5fdd-b105-ffc91e053289"

shell> cat downstream/Project.toml
[deps]
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
VersionParsing = "81def892-9a0e-5fdd-b105-ffc91e053289"
upstream = "88ecca56-6ad2-11e9-2834-950dc93fed92"

shell> cat demo.jl
rm("downstream/Manifest.toml", force=true)

using Pkg
pkg"""
activate downstream
dev upstream
instantiate
"""

println()
println()
pkg"""
status JSON
status VersionParsing
"""

julia> include("demo.jl")
[ Info: resolving package identifier `upstream` as a directory at `~/repos/scratch/2019/julia/pkg/extras-compat/upstream`.
[ Info: activating environment at `~/repos/scratch/2019/julia/pkg/extras-compat/downstream/Project.toml`.
 Resolving package versions...
  Updating `~/repos/scratch/2019/julia/pkg/extras-compat/downstream/Project.toml`
  [682c06a0] + JSON v0.19.0
  [81def892] + VersionParsing v1.1.3
  [88ecca56] + upstream v0.0.0 [`../upstream`]
  Updating `~/repos/scratch/2019/julia/pkg/extras-compat/downstream/Manifest.toml`
  [34da2185] + Compat v2.1.0
  [682c06a0] + JSON v0.19.0
  [81def892] + VersionParsing v1.1.3
  [88ecca56] + upstream v0.0.0 [`../upstream`]
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`


    Status `~/repos/scratch/2019/julia/pkg/extras-compat/downstream/Project.toml`
  [682c06a0] JSON v0.19.0
    Status `~/repos/scratch/2019/julia/pkg/extras-compat/downstream/Project.toml`
  [81def892] VersionParsing v1.1.3

julia> VERSION
v"1.3.0-DEV.121"

I see VersionParsing v1.1.3 but I expected VersionParsing v1.1.1 as I put VersionParsing = "= 1.1.1" in the upstream.

[00vareladavid]

How is this different from #895?

[tkf]

Unlike #895, #1033 does not solve this issue. As I mentioned in the OP, JuliaPlots/Plots.jl#1956 is an example. Plots relies on PyPlot only via @require. It's not possible to declare compatibility in this case.

[dehann]

Hi, I just ran into the same issue: Have one optional dependency through @ require but don't know how to define its [compat] requirements?

[tkf]

I don't think there is a way to do it (that's why I opened this issue). Maybe it's obvious but I think what you can do is to error out or warn when an incompatible version is loaded. See, e.g., how Flux.jl does this: https://github.com/FluxML/Flux.jl/blob/8ee6af1bee39bd9f6ed8e6cdf854ea9aaccb14dd/src/cuda/cuda.jl#L7-L18

PS: Thanks for the comment. Now Pkg devs know there are at least more than two people want this feature :)

[KristofferC]

#1285 has discussion for compat with conditional dependencies. The conditional dependencies will simply need to be compatible with the rest of the project, otherwise they will not be loaded.