Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meta-issue for improving the REPL experience #50817

Open
10 of 15 tasks
tecosaur opened this issue Aug 7, 2023 · 11 comments
Open
10 of 15 tasks

Meta-issue for improving the REPL experience #50817

tecosaur opened this issue Aug 7, 2023 · 11 comments
Labels
REPL Julia's REPL (Read Eval Print Loop)

Comments

@tecosaur
Copy link
Contributor

tecosaur commented Aug 7, 2023

I've ended up on a path to improve the REPL experience in a few ways, and thought I'd actually write up the overall plan.

Here's the vision for the REPL I currently have in mind:

image

GraphViz code
digraph {
    graph [bgcolor="transparent"];
    node  [shape="underline" penwidth="2" style="rounded,filled" fillcolor="#efefef" color="#c9c9c9" fontcolor="#000000" fontname="Alegreya Sans"];
    edge  [color="#aaaaaa" penwidth="1.2" fontname="Alegreya Sans"]
    rankdir = "LR"

    // Merged
    node [color="#a371f7"];
    "#46372 (JuliaSyntax)"
    "#50797 (Terminfo parser)"
    "#49583 (AnnotatedStrings + StyledStrings)"
    "StyledStrings#6 (Legacy support)"
    "StyledStrings#3 (Precompile fix)"
    "#51867 (move banner function)"
    "ColorTypes.jl#293 (convert to SimpleColor)"
    "#51806 (Improved annotation *-concat)"
    "#51810 (JuliaSyntaxHighlighting)"
    "#51807 (AnnotatedIOBuffer)"
    "StyledStrings#12 (printstyled specialisation)"
    "#51869 (load StyledStrings in REPL)"
    "#X Merged"
    // Pending
    node [color="#3fb950"];
    "#51802 (AnnotatedStrings in log messages)"
    "#51816 (Themable stacktraces)"
    "StyledStrings#36 (Themable stacktraces)"
    "#51829 (StyledStrings for Logging)"
    "#51914 (Improved join method for AnnotatedStrings)"
    "#51928 (StyledStrings for Markdown)"
    "#51811 (Resizing tweaked banner)"
    "#X Open"
    // Draft
    node [color="#6e7680"];
    "#51887 (StyledStrings for REPL Prompts)"
    "#X Draft"
    // Conceptual
    node [fillcolor="#fafafa" color="#dddddd" fontcolor="#444444"];
    "#? Conceptual"

    subgraph cluster_0 {
        style=rounded
        fillcolor="#ffffff"
        color="#222222"
        rankdir="LR"
        "#? Conceptual" -> "#X Draft" -> "#X Open" -> "#X Merged"
    }

    "#50797 (Terminfo parser)" -> "#49583 (AnnotatedStrings + StyledStrings)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51802 (AnnotatedStrings in log messages)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51806 (Improved annotation *-concat)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51807 (AnnotatedIOBuffer)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51810 (JuliaSyntaxHighlighting)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "ColorTypes.jl#293 (convert to SimpleColor)"
    "#46372 (JuliaSyntax)" -> "#51810 (JuliaSyntaxHighlighting)"
    "#46372 (JuliaSyntax)" -> "#??? (Prettier JuliaSyntax errors)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#??? (Prettier JuliaSyntax errors)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (Prettier JuliaSyntax errors)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#51928 (StyledStrings for Markdown)"
    "#51807 (AnnotatedIOBuffer)" -> "#51914 (Improved join method for AnnotatedStrings)"
    "#51807 (AnnotatedIOBuffer)" -> "#51928 (StyledStrings for Markdown)"
    "#51807 (AnnotatedIOBuffer)" -> "StyledStrings#12 (printstyled specialisation)"
    "#51807 (AnnotatedIOBuffer)" -> "#??? (show/showerror with StyledStrings)"
    "#??? (string similarity matcher in Base)" -> "#??? (More helpful error messages)"
    "#??? (show/showerror with StyledStrings)" -> "#??? (More helpful error messages)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51869 (load StyledStrings in REPL)"
    "#51869 (load StyledStrings in REPL)" -> "#51811 (Resizing tweaked banner)"
    "#51867 (move banner function)" -> "#51811 (Resizing tweaked banner)"
    "#51811 (Resizing tweaked banner)" -> "#??? (Spiffed up banner)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51816 (Themable stacktraces)"
    "#51869 (load StyledStrings in REPL)" -> "#51816 (Themable stacktraces)"
    "StyledStrings#36 (Themable stacktraces)" -> "#51816 (Themable stacktraces)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51887 (StyledStrings for REPL Prompts)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#51887 (StyledStrings for REPL Prompts)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "#??? (Deeper StyledStrings ↔ REPL integration)"
    "#51869 (load StyledStrings in REPL)" -> "#??? (Deeper StyledStrings ↔ REPL integration)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (REPL syntax highlighting)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (REPL syntax highlighting)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "StyledStrings#3 (Precompile fix)"
    "#49583 (AnnotatedStrings + StyledStrings)" -> "StyledStrings#6 (Legacy support)"
    "StyledStrings#6 (Legacy support)" -> "#51869 (load StyledStrings in REPL)"
    "StyledStrings#6 (Legacy support)" -> "#51829 (StyledStrings for Logging)"
    "StyledStrings#3 (Precompile fix)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51829 (StyledStrings for Logging)"
    "#51869 (load StyledStrings in REPL)" -> "#51928 (StyledStrings for Markdown)"
    "#51869 (load StyledStrings in REPL)" -> "#??? (StyledStrings for Pkg)"
    "#??? (Basic terminal pager)" -> "#??? (Pageable help)"
    "#??? (Resolution of [@ref] links)" -> "#??? (Interactively navigable docs)"
    "#??? (Pageable help)" -> "#??? (Interactively navigable docs)"
    "#51928 (StyledStrings for Markdown)" -> "#??? (Pageable help)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (Basic terminal pager)"
    "#??? (Deeper StyledStrings ↔ REPL integration)" -> "#??? (REPL menu revamp)"
    "#??? (REPL menu revamp)" -> "#??? (Multi-modal menus)"
    "#??? (REPL menu revamp)" -> "#??? (Interactive history search)"
    "#51810 (JuliaSyntaxHighlighting)" -> "#??? (Interactive history search)"
}
@brenhinkeller brenhinkeller added the REPL Julia's REPL (Read Eval Print Loop) label Aug 7, 2023
@c42f
Copy link
Member

c42f commented Aug 8, 2023

I am so onboard with this plan! Not confident how much I can promise to work on the code directly, but I'm happy to support via JuliaSyntax adjustments and advice as/if necessary.

Because REPL is a stdlib, I think you can feel free to use the the JuliaSyntax API (directly, not only via Meta.parse()) and I hope that should make REPL syntax highlighting fairly easy once we've got Styled strings merged. A nice side effect of this should somehow be syntax highlighting of broken syntax within the REPL as people type. So people can fix that syntax as they type rather than trying to eval it first. OhMyREPL already has some of this via the JuliaSyntax parser, but it's still based on tokens rather than the parse tree so some of that info goes missing along the way.

Maybe we can add another item to the list:

  • Reimplement REPL completion based on the JuliaSyntax parser.

Because last time I looked at the REPL.jl completion code it's ... a bit of a scary disaster area of regexes and heuristics IIUC

@c42f
Copy link
Member

c42f commented Aug 8, 2023

Another item to add to the list:

  • Make the REPL an external (upgradable) stdlib. I suspect this would greatly improve our ability to make progress. See Upgradable stdlibs #50697

@tecosaur
Copy link
Contributor Author

I'm wondering, would it be worth making a channel on Zulip/etc. for chatting about this? I think it could be worth making a space for discussion around this, and I'm not sure that just this issue suffices.

@caleb-allen
Copy link

@tecosaur I think that's a good idea. I was surprised that a #repl channel doesn't already exist in the Slack

@tecosaur
Copy link
Contributor Author

Personally, I'm much more a fan of the Zulip, so if other people are fine with that (yes?), I'll put in a request for a #repl channel there.

@tecosaur
Copy link
Contributor Author

I've now made a Zulip stream: https://julialang.zulipchat.com/#narrow/stream/404905-repl 🙂

@tecosaur
Copy link
Contributor Author

FYI, there is now also a Slack channel (that currently basically says "go to Zulip"), and the Slack/Zulip channels should be listed as part of the new "REPL working group" soon (with me as the contact).

Meetings etc. TBD, but for now I thought it would be a nice start to better collect people interested in driving this aspect of the Julia experience forward 🙂.

KristofferC pushed a commit that referenced this issue Oct 12, 2023
Fix #51194

This PR fixes a regression introduced in
#49294, so I believe it should be
backported to v1.10.
In the current code, completion of `qux(foo, bar.` is detected by
parsing `foo(qux, bar` as an incomplete expression, and then looking for
the sub-expression to complete (here, `bar.`). This approach fails
however for infix calls, since completing `foo + bar.` starts by parsing
`foo + bar`, which is a complete call expression, and so the code
behaves as if completing `(foo + bar).` instead of `bar.`. This leads to
the current problematic behaviour:
```julia
julia> Complex(1, 3) + (4//5).#TAB
im
re
```
which would be correct for `(Complex(1, 3) + (4//5)).#TAB`, but here we
expect
```julia
julia> Complex(1, 3) + (4//5).#TAB
den
num
```

This PR fixes that by trying to detect infix calls. In the long term,
all this ad-hoc and probably somewhat wrong string processing should be
replaced by proper use of `JuliaSyntax` (as mentioned in
#49294 (comment),
#50817 (comment)
and probably other places), but for now at least this fixes the
regression.
KristofferC pushed a commit that referenced this issue Oct 12, 2023
Fix #51194

This PR fixes a regression introduced in
#49294, so I believe it should be
backported to v1.10.
In the current code, completion of `qux(foo, bar.` is detected by
parsing `foo(qux, bar` as an incomplete expression, and then looking for
the sub-expression to complete (here, `bar.`). This approach fails
however for infix calls, since completing `foo + bar.` starts by parsing
`foo + bar`, which is a complete call expression, and so the code
behaves as if completing `(foo + bar).` instead of `bar.`. This leads to
the current problematic behaviour:
```julia
julia> Complex(1, 3) + (4//5).#TAB
im
re
```
which would be correct for `(Complex(1, 3) + (4//5)).#TAB`, but here we
expect
```julia
julia> Complex(1, 3) + (4//5).#TAB
den
num
```

This PR fixes that by trying to detect infix calls. In the long term,
all this ad-hoc and probably somewhat wrong string processing should be
replaced by proper use of `JuliaSyntax` (as mentioned in
#49294 (comment),
#50817 (comment)
and probably other places), but for now at least this fixes the
regression.

(cherry picked from commit e949236)
@tecosaur
Copy link
Contributor Author

I've added a plot to the end of the initial message that people subscribed to this issue may find of interest, I've also plopped it in a zulip thread.

@tecosaur
Copy link
Contributor Author

From slack, Ruby's interactive REPL has had a fun update that we might want to look at in this effort: https://railsatscale.com/2023-12-19-irb-for-ruby-3-3/

@tecosaur
Copy link
Contributor Author

tecosaur commented Feb 1, 2024

Thanks to a bout of recent merging, this effort seems to be lurching forwards again 😀

@tecosaur
Copy link
Contributor Author

tecosaur commented Feb 5, 2024

I'm starting an effort to provide a fresh take on REPL.TerminalMenus, see the Zulip thread for more info: #repl Developing InteractivePrompts.jl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
REPL Julia's REPL (Read Eval Print Loop)
Projects
None yet
Development

No branches or pull requests

4 participants