Enable REPL to offer to install missing packages if install hooks are provided #39026

IanButterworth · 2020-12-29T09:40:19Z

Just trying out the ideas discussed in #26469 and in #35062 and the suggestion here #26469 (comment) to introduce a specific error type to allow other handlers to decide what to do.

This goes for a more basic y/n prompt than the example in #26469, with the default being n, which is selected here first.
It also leaves it up to the user to decide if the active environment is appropriate to install into.
A ctrl-c acts the same as returning n or just pressing enter.

Here the default is selected first by just hitting enter

cc. @KristofferC @oxinabox

KristofferC · 2020-12-29T14:04:18Z

Nice! I do think that Pkg itself needs to manage some of this though (and not only REPL). Some comments:

I think the auto complete should only be done if the package exists in one of the current registries.
It would be good to be able to enable auto installation of future packages (without requiring a prompt).
Also see allow packages to hook into code loading #26469 (comment)
Regarding the implementation, I think having Pkg register a callback with the REPL (similar to allow packages to hook into code loading #26469) makes sense because providing the info above will likely require using Pkg internals.

oxinabox · 2020-12-29T15:37:03Z

I think the auto complete should only be done if the package exists in one of the current registries.

Strongly agreed. It would be annoying to get this if you had made a typo in the package name.
and so spent several seconds of it resolving only to get an error message..

It would be good to be able to enable auto installation of future packages (without requiring a prompt).

I think that can be a seperate PR?
This is useful without it, and the complexity of dealing with how to store persistient settings related to Pkg seems like it would bog this down.
Unless you just mean an eviroment variable that is set externally?

IanButterworth · 2020-12-29T21:35:21Z

Ok, I've made some changes:

Switched to a similar hooks method to allow packages to hook into code loading #26469 but the difference is it hooks into when ModuleNotFoundError is thrown during REPL eval, rather than at the Base.require level
Added the appropriate Pkg hook in Add REPL hook to prompt to install missing packages, if available in registry Pkg.jl#2307
Checks for whether the package is available in installed registries. If so, prompts to install, if not, throws the old error directly
Re-evaluating the ast after the hook is finished, if it returns true, so that the package is loaded at the end. I thought this makes sense given that if the prompted install process completes without error, it would be weird if the original instruction wasn't successful

Demo: https://asciinema.org/a/adqDA5jSp3fusjSdIPOtttvqK

julia> using CSVV
ERROR: ModuleNotFoundError: Package CSVV not found in current path:
- Run `import Pkg; Pkg.add("CSVV")` to install the mod package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:867

julia> using CSV
 └ Package "CSV" not found, but "CSV" is listed in registry.
   Active project: /Users/ian/Documents/GitHub/Pkg.jl/Project.toml
   Add "CSV" to the active project? (y/n) [n]: y
   Resolving package versions...
Updating `~/Documents/GitHub/Pkg.jl/Project.toml`
  [336ed68f] + CSV v0.8.2
Updating `~/Documents/GitHub/Pkg.jl/Manifest.toml`
  [336ed68f] + CSV v0.8.2
  [e2d170a0] + DataValueInterfaces v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [69de0a69] + Parsers v1.0.15
  [2dfb63ee] + PooledArrays v0.5.3
  [91c51154] + SentinelArrays v1.2.16
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v1.2.2
  Progress [========================================>]  9/9
9 dependencies successfully precompiled in 11 seconds (37 already precompiled)

julia> CSV
CSV

DilumAluthge · 2020-12-31T01:20:55Z

This is only in interactive usage, right? I.e. in scripts, you'd just throw the error.

IanButterworth · 2020-12-31T02:04:20Z

Yeah. That's the intention. This is what I've tested

ian@ian julia % ./julia -e "using CSV"
ERROR: ModuleNotFoundError: Package CSV not found in current path:
- Run `import Pkg; Pkg.add("CSV")` to install the mod package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:867

IanButterworth · 2021-01-02T05:36:50Z

Just pushed to the corresponding pkg PR to improve the info and formatting of the prompt:

Demo: https://asciinema.org/a/kO5Q4sZJp6qfI0a1Gtvo0xArx

carstenbauer · 2021-01-02T22:19:44Z

I like this! However, do we need to force the user into a yes/no dialog? These can be rather annoying. What about a more minimal approach, something like

julia> using CSV
Package "CSV" not found, but available from a registry.
Install the package with `] add CSV`  or by pressing `CTRL+I`.

julia>

Which would install the package upon pressing CTRL+I right after such a message has occured.

IanButterworth · 2021-01-02T22:57:54Z

One option is to make the default y, with a short timeout of a few seconds. I opened #39027 if that was desired.

That would feel less like forcing the user into a decision.

Providing a magic key combo to run the package add is also interesting

StefanKarpinski · 2021-01-04T19:05:24Z

The yes/no dialog is what most package managers do here, so it seems pretty standard and easy.

IanButterworth · 2021-01-15T04:31:59Z

Any further thoughts on this?

In summary I think the flow works well, and it seems like it would allow exotic front ends to do whatever it prefers when the ModuleNotFoundError is thrown.

However, I think there's one (hopefully small) con with this strategy of intercepting errors in the REPL evaluation..
Say the user does these examples, where Bar isn't installed

julia> using Foo, Bar

then Foo will be loaded twice, the 2nd time when the entire line is re-evaluated. I guess packages are usually safe to load twice though.

or

julia> x = 1;
julia> x += 1; using Bar

x would end up at 3.
That wouldn't be the case with

julia> x = 1;
julia> using Bar; x += 1

IanButterworth · 2021-03-13T08:16:40Z

Just rebased and made this prompt only trigger for lines starting with using or import to avoid the edge case of re-running code on the same line that precedes the load, shown in the above comment.

Also added "and load" to the end to make it clearer that the package is loaded after install

simeonschaub · 2021-03-13T11:31:02Z

stdlib/REPL/src/REPL.jl

+function handle_install_package_hooks(mod, ast, backend)
+    line = string(ast.args[2])
+    # only exercise hooks for lines starting with `using/import` to avoid re-running non-load code
+    if startswith(line, "using") || startswith(line, "import")


I am not sure this is the best way to check this, since there could also be leading whitespace or the using statement could be further nested but still run first. We could do some analysis on the parsed AST, but I am wondering if in practice just always reevaluating the whole input would really be a problem.

but I am wondering if in practice just always reevaluating the whole input would really be a problem.

Yeah, I think you might be right. I've removed the check for using/import

miguelraz · 2021-04-22T19:51:44Z

Feature request: if a user does

using Foo, Bar

It will prompt to install all of them. As Stefan says, if users want an in between, they can do it manually.

JeffBezanson · 2021-04-22T19:54:36Z

Suggestion from Stefan: After the repl input is parsed, search for using and import statements in the AST, collect all the names of packages not in the environment, and prompt to install all of them before doing eval.

JeffBezanson · 2021-04-22T20:39:33Z

The advantage of Stefan's parse-first method is it avoids all issues with when and how many times things are evaluated. Even if the input contains if false; using X; end, prompting to install X still makes sense, and the conditional using still works, since all we've done is modify the project, not the state of the running process.

A "disadvantage" (but depends on your point of view) is that meta-programmed using statements will not prompt, but most of us in triage felt that case doesn't matter.

IanButterworth · 2021-04-24T15:22:26Z

Now updated along with JuliaLang/Pkg.jl#2307 to @StefanKarpinski's suggestion of checking for all packages that will be loaded in the ast, then handling all packages in one act.

It seems to work quite nicely.

A single package

julia> using ImageIO
 │ Package "ImageIO" not found, but "ImageIO" is available from a registry.
 │ Project `~/.julia/environments/v1.7/Project.toml`
 └ Add "ImageIO" to the active project and load? (y/n) [n]: n
ERROR: ArgumentError: Package ImageIO not found in current path:
- Run `import Pkg; Pkg.add("ImageIO")` to install the ImageIO package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:873

Where all packages are missing, but only two packages are available in a registry

julia> using ImageIO, TiffImages, DoesNotExist
 │ Packages ["ImageIO", "TiffImages", "DoesNotExist"] not found, but ["ImageIO", "TiffImages"] are available from a registry.
 │ Project `~/.julia/environments/v1.7/Project.toml`
 └ Add ["ImageIO", "TiffImages"] to the active project and load? (y/n) [n]: n
ERROR: ArgumentError: Package ImageIO not found in current path:
- Run `import Pkg; Pkg.add("ImageIO")` to install the ImageIO package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:871

Where specific imports are declared

julia> using ImageIO: save
 │ Package ["ImageIO"] not found, but ["ImageIO"] is available from a registry.
 │ Project `~/.julia/environments/v1.7/Project.toml`
 └ Add ["ImageIO"] to the active project and load? (y/n) [n]: n
ERROR: ArgumentError: Package ImageIO not found in current path:
- Run `import Pkg; Pkg.add("ImageIO")` to install the ImageIO package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:871

In guarded code

julia> if false if false using ImageIO, TiffImages, DoesNotExist end end
 │ Packages ["ImageIO", "TiffImages", "DoesNotExist"] not found, but ["ImageIO", "TiffImages"] are available from a registry.
 │ Project `~/.julia/environments/v1.7/Project.toml`
 └ Add ["ImageIO", "TiffImages"] to the active project and load? (y/n) [n]: n

The first examplle but selecting y

julia> using ImageIO
 │ Package "ImageIO" not found, but "ImageIO" is available from a registry.
 │ Project `/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_U2GrdN/Project.toml`
 └ Add "ImageIO" to the active project and load? (y/n) [n]: y
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed Netpbm ──── v1.0.2
   Installed PNGFiles ── v0.3.7
   Installed ImageCore ─ v0.9.0
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_U2GrdN/Project.toml`
  [82e4d734] + ImageIO v0.5.3
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_U2GrdN/Manifest.toml`
...
Precompiling project...
  40 dependencies successfully precompiled in 48 seconds (1 already precompiled)

julia> ImageIO
ImageIO

The second example but selecting y

julia> using ImageIO, TiffImages, DoesNotExist
 │ Packages ["ImageIO", "TiffImages", "DoesNotExist"] not found, but ["ImageIO", "TiffImages"] are available from a registry.
 │ Project `~/.julia/environments/v1.7/Project.toml`
 └ Add ["ImageIO", "TiffImages"] to the active project and load? (y/n) [n]: y
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed Netpbm ───── v1.0.1
   Installed TiffImages ─ v0.3.1
    Updating `~/.julia/environments/v1.7/Project.toml`
  [82e4d734] + ImageIO v0.5.3
  [731e570b] + TiffImages v0.3.1
    Updating `~/.julia/environments/v1.7/Manifest.toml`
...
Precompiling project...
...
ERROR: ArgumentError: Package DoesNotExist not found in current path:
- Run `import Pkg; Pkg.add("DoesNotExist")` to install the DoesNotExist package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:873

What it actually looks like on MacOS

Doesn't add noticeable overhead to REPL eval

StefanKarpinski · 2021-04-24T19:28:24Z

Beautiful!

simeonschaub · 2021-04-26T12:54:25Z

stdlib/REPL/src/REPL.jl

+            if first(arg.args) isa Symbol # i.e. `Foo`
+                push!(mods, first(arg.args))
+            else # i.e. `Foo: bar`
+                push!(mods, first(first(arg.args).args))


I think this might need to be made more robust. I don't think this works for something like import A.b as c yet. Does this currently know to properly ignore imports like using .A?

I didn't think about either of those. I'll investigate and add tests

Fixed the using .A case, and the other worked already. Added tests. Are there any other forms to watch out for?

JeffBezanson · 2021-04-26T21:27:05Z

stdlib/REPL/src/REPL.jl

 function eval_with_backend(ast, backend::REPLBackendRef)
+    if !isempty(install_packages_hooks)
+        check_for_missing_packages_and_run_hooks(ast)


This should probably run in the backend. It could even use the existing repl_ast_transforms hook (not a "transform" per se, but no big deal).

Ok. Moved to the backend, but given that the modules_to_be_loaded parser resides here, it seemed like a misuse to place that on repl_ast_transforms given that if IJulia/Pluto etc. wanted to do a different install hook they'd probably want to go after the modules have been identified by modules_to_be_loaded so keeping install_packages_hooks would be necessary anyway. These use models are a bit imaginary though.. my thinking might be off

IanButterworth · 2021-04-30T17:12:19Z

For those following this, there's some unresolved discussion about the message formatting in JuliaLang/Pkg.jl#2307
If anyone has any opinions it would be good to get that resolved & merged.

… provided (JuliaLang#39026)

IanButterworth force-pushed the ib/offer_to_install branch 2 times, most recently from a6aecac to 29ec192 Compare December 29, 2020 09:44

This was referenced Dec 29, 2020

allow packages to hook into code loading #26469

Closed

Add timeout to Base.prompt #39027

Open

IanButterworth mentioned this pull request Dec 29, 2020

Add REPL hook to prompt to install missing packages, if available in registry JuliaLang/Pkg.jl#2307

Merged

IanButterworth changed the title ~~Enable REPL to offer to install missing packages, via new ModuleNotFoundError and handler~~ Enable REPL to offer to install missing packages, via new ModuleNotFoundError and install hooks for Pkg etc to provide Dec 29, 2020

IanButterworth force-pushed the ib/offer_to_install branch from 38fb0f1 to 149afd8 Compare March 13, 2021 08:03

simeonschaub reviewed Mar 13, 2021

View reviewed changes

JeffBezanson added the forget me not PRs that one wants to make sure aren't forgotten label Apr 22, 2021

REPL: offer to try to install package if not found

ba8e166

IanButterworth force-pushed the ib/offer_to_install branch from c68f689 to ba8e166 Compare April 24, 2021 14:57

IanButterworth changed the title ~~Enable REPL to offer to install missing packages, via new ModuleNotFoundError and install hooks for Pkg etc to provide~~ Enable REPL to offer to install missing packages if install hooks are provided Apr 25, 2021

tidier code & don't check modules if no hooks available

086a556

IanButterworth force-pushed the ib/offer_to_install branch from e70611a to 086a556 Compare April 25, 2021 13:22

simeonschaub reviewed Apr 26, 2021

View reviewed changes

JeffBezanson reviewed Apr 26, 2021

View reviewed changes

IanButterworth added 3 commits April 26, 2021 22:38

exclude dotted local imports

35ef519

add tests for loaded modules parser

1c12fab

run in backend

86cc121

JeffBezanson merged commit df27063 into JuliaLang:master Apr 27, 2021

IanButterworth deleted the ib/offer_to_install branch April 27, 2021 23:01

ElOceanografo pushed a commit to ElOceanografo/julia that referenced this pull request May 4, 2021

Enable REPL to offer to install missing packages if install hooks are…

6f4784e

… provided (JuliaLang#39026)

jarlebring pushed a commit to jarlebring/julia that referenced this pull request May 4, 2021

Enable REPL to offer to install missing packages if install hooks are…

9ae2f02

… provided (JuliaLang#39026)

antoine-levitt pushed a commit to antoine-levitt/julia that referenced this pull request May 9, 2021

Enable REPL to offer to install missing packages if install hooks are…

dab041d

… provided (JuliaLang#39026)

simeonschaub mentioned this pull request May 10, 2021

Comment in REPL triggers an exception #40776

Closed

BioTurboNick mentioned this pull request May 26, 2021

Hint or helper at REPL for using/importing non-package modules #40959

Closed

simeonschaub removed the forget me not PRs that one wants to make sure aren't forgotten label May 29, 2021

johanmon pushed a commit to johanmon/julia that referenced this pull request Jul 5, 2021

Enable REPL to offer to install missing packages if install hooks are…

6f50980

… provided (JuliaLang#39026)

This was referenced Jul 14, 2021

Make REPL ask if you want to install a package, rather than throwing error? #35062

Closed

Macro to add "using" packages automatically JuliaLang/Pkg.jl#2027

Closed

juandarias mentioned this pull request Feb 2, 2022

Compatibility with new Package Manager feature (#39026) when executing code in REPL julia-vscode/julia-vscode#2712

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable REPL to offer to install missing packages if install hooks are provided #39026

Enable REPL to offer to install missing packages if install hooks are provided #39026

IanButterworth commented Dec 29, 2020 •

edited

Loading

KristofferC commented Dec 29, 2020

oxinabox commented Dec 29, 2020

IanButterworth commented Dec 29, 2020 •

edited

Loading

DilumAluthge commented Dec 31, 2020

IanButterworth commented Dec 31, 2020 •

edited

Loading

IanButterworth commented Jan 2, 2021 •

edited

Loading

carstenbauer commented Jan 2, 2021

IanButterworth commented Jan 2, 2021

StefanKarpinski commented Jan 4, 2021

IanButterworth commented Jan 15, 2021 •

edited

Loading

IanButterworth commented Mar 13, 2021

simeonschaub Mar 13, 2021

IanButterworth Apr 4, 2021

miguelraz commented Apr 22, 2021

JeffBezanson commented Apr 22, 2021

JeffBezanson commented Apr 22, 2021

IanButterworth commented Apr 24, 2021 •

edited

Loading

StefanKarpinski commented Apr 24, 2021

simeonschaub Apr 26, 2021

IanButterworth Apr 26, 2021

IanButterworth Apr 27, 2021

JeffBezanson Apr 26, 2021

IanButterworth Apr 27, 2021

IanButterworth commented Apr 30, 2021

Enable REPL to offer to install missing packages if install hooks are provided #39026

Enable REPL to offer to install missing packages if install hooks are provided #39026

Conversation

IanButterworth commented Dec 29, 2020 • edited Loading

KristofferC commented Dec 29, 2020

oxinabox commented Dec 29, 2020

IanButterworth commented Dec 29, 2020 • edited Loading

DilumAluthge commented Dec 31, 2020

IanButterworth commented Dec 31, 2020 • edited Loading

IanButterworth commented Jan 2, 2021 • edited Loading

carstenbauer commented Jan 2, 2021

IanButterworth commented Jan 2, 2021

StefanKarpinski commented Jan 4, 2021

IanButterworth commented Jan 15, 2021 • edited Loading

IanButterworth commented Mar 13, 2021

simeonschaub Mar 13, 2021

Choose a reason for hiding this comment

IanButterworth Apr 4, 2021

Choose a reason for hiding this comment

miguelraz commented Apr 22, 2021

JeffBezanson commented Apr 22, 2021

JeffBezanson commented Apr 22, 2021

IanButterworth commented Apr 24, 2021 • edited Loading

StefanKarpinski commented Apr 24, 2021

simeonschaub Apr 26, 2021

Choose a reason for hiding this comment

IanButterworth Apr 26, 2021

Choose a reason for hiding this comment

IanButterworth Apr 27, 2021

Choose a reason for hiding this comment

JeffBezanson Apr 26, 2021

Choose a reason for hiding this comment

IanButterworth Apr 27, 2021

Choose a reason for hiding this comment

IanButterworth commented Apr 30, 2021

IanButterworth commented Dec 29, 2020 •

edited

Loading

IanButterworth commented Dec 29, 2020 •

edited

Loading

IanButterworth commented Dec 31, 2020 •

edited

Loading

IanButterworth commented Jan 2, 2021 •

edited

Loading

IanButterworth commented Jan 15, 2021 •

edited

Loading

IanButterworth commented Apr 24, 2021 •

edited

Loading