Support for undercurl

[ssoriche]

Neovim (and maybe others but this is where I notice it) has support for undercurl, a wavy underline that appears, with my configuration underneath spelling mistakes. I'm making the migration from kitty to wezterm and this is one thing that I've noticed is missing.

Attaching images to better demonstrate the wanted behaviour and what currently exists.

kitty:

wezterm:

Both are using the same version of Neovim and the same configuration (running on one computer).

I was missing undercurl in tmux and ended up finding this issue which describes the problem and the solutions that others (including kitty) have used to support undercurl

tmux/tmux#1492

[wez]

Here's the iTerm2 issue as well: https://gitlab.com/gnachman/iterm2/-/issues/6382

[wez]

The biggest challenge is the non-standard use of : in a CSI sequence; the vt100/DEC conforming vtparser state machine defines that as invalid and ignores that sequence. There's discussion in the various issues about using terminfo to specify the actual sequence that is used. If we could pick a conforming sequence (really just some unused numbers--for instance, we already support double underline via CSI 21 m so we don't need a colon based sequence like CSI 4:2 m for that) instead and configure those via terminfo then it would be pretty trivial to wrap this up (I have most of the code already written).

I took a quick look at changing the vtparser logic; it's not impossible to accommodate but does require more state bits (unfortunately bumping us over from a pair of 4 bit nybbles and some clever lookup/mapping logic, which makes for a larger architectural change in that area), and representing the CSI expansion parameters as an array of either an integer or a sequence of colon separated integers, and then fanning that out through a lot of other code. The effort required for all of that is much higher than the rest of this feature!

Even with that done, the parser would then be potentially prone to misbehaving with some invalid inputs, whereas the behavior today is well defined and conforming.

[wez]

As of master (which should show up in a nightly build within the next 1-2 hours), the following escape sequences relating to underline are supported:

   CSI 24 m   -> No underline
   CSI 4 m    -> Single underline
   CSI 21 m   -> Double underline
   CSI 60 m   -> Curly underline
   CSI 61 m   -> Dotted underline
   CSI 62 m   -> Dashed underline

   CSI 58 ; 2 ; R ; G ; B m   -> set underline color to specified true color RGB
   CSI 58 ; 5 ; I m           -> set underline color to palette index I (0-255)
   CSI 59                     -> restore underline color to default

wezterm uses CSI 60 m rather than CSI 4:3 m as mentioned above.

You can exercise these from the command line:

$ printf "\x1b[58;2;255;0;0m\x1b[4msingle\x1b[21mdouble\x1b[60mcurly\x1b[61mdotted\x1b[62mdashed\x1b[0m"

which renders for me like this:

You can teach vim about this; place this in foo.vim:

" Inform vim how to enable undercurl in wezterm
let &t_Cs = "\e[60m"
" Inform vim how to disable undercurl in wezterm (this disables all underline modes)
let &t_Ce = "\e[24m"

hi SpellBad   guisp=red    gui=undercurl term=underline cterm=undercurl
hi SpellCap   guisp=yellow gui=undercurl term=underline cterm=undercurl
hi SpellRare  guisp=blue   gui=undercurl term=underline cterm=undercurl
hi SpellLocal guisp=orange gui=undercurl term=underline cterm=undercurl

set spell

" A text with spellling mistake in vim. check that underline or undercurl are OK.

Then vim foo.vim and inside vim source foo.vim to activate it; I see the following:

[ssoriche]

Wow, this is incredible. Thank you.

Already tested and is working as expected thus far.

EDIT: I spoke too soon. When I tried sourcing the above in Neovim it had no effect. Misspelled words are still regular underline.

I started searching through Neovim issues to see if I could find anything and came across the implementation issue here neovim/neovim#7479

[bew]

(just chimin here, didn't test anything but this might help you)

It works in vim (as @wez showed) but not in neovim, because the variables t_Cs & t_Ce are not available in neovim, see: https://neovim.io/doc/user/vim_diff.html#t_xx

Nvim does not have special t_XX options nor <t_XX> keycodes to configure
terminal capabilities. Instead Nvim treats the terminal as any other UI,
e.g. 'guicursor' sets the terminal cursor style if possible.

Nvim never uses the termcap database, only |terminfo| and |builtin-terms|.

I started searching through Neovim issues to see if I could find anything and came across the implementation issue here neovim/neovim#7479

The implementation of neovim's TUI changed a bit since that initial implementation, the current lines (in current neovim's master) are:
https://github.com/neovim/neovim/blob/2ea3127697692b0b2c480d585ef6529e90a72b0e/src/nvim/tui/tui.c#L2028-L2044
And seems to be hardcoded to use \e[4:{some_number}m

[wez]

There's some commentary in neovim/neovim#9270 on the preference of the use of : as a separator for these sequences (both the underline and the color sequence). I can see the logic, but also think that the cat is out of the bag to a certain extent. It's probably a good idea for nvim to support some configuration around these options, but also, I should likely suck it up and teach wezterm's vtparse crate how to deal with those being output.

As I mentioned above, it's a bit of a PITA, but not impossible.

The tmux issue you linked has a discussion around advertising some of these capabilities in terminfo. Thus far, wezterm has been good to successfully run with TERM=xterm-256color. This change may be the first one to warrant adding a definitive wezterm terminfo file to report this capability.

[ssoriche]

Personally, I'm not as concerned with the tmux issue, with wezterm I won't be using a local tmux any longer.

I really like the fact that I do not have to specify my term when ssh'ing into systems, or receiving error messages that my terminal is not fully functional.

[wez]

Yeah, not requiring special terminfo/TERM setup has been a soft goal of mine as part of this project (although: I do end up with installing a current xterm terminfo in my remote shell setup anyway to ensure that I get working italics when running vim remotely).

This line in nvim: https://github.com/neovim/neovim/blob/2ea3127697692b0b2c480d585ef6529e90a72b0e/src/nvim/tui/tui.c#L2030 appears to require that the terminfo have an Smulx entry to identify support for this feature.

If that isn't present it falls back to looking up a different boolean property to identify kitty by its terminfo, and finally falls back to probing to see if the terminal is vte-based.

If you can configure nvim to override Smulx then you can avoid requiring a terminfo file.

The default Smulx value assumed by nvim is \x1b[4:%p1%dm

That's a string that takes an integer parameter to specify the number after the colon. Two values are important; 0 to disable underline and 3 to enable curly underline.

Looking at man terminfo, there's a way to "switch" and emit different values: I think something like this might do the job for wezterm, but I don't have time or a convenient way to verify this right at this moment:

\x1b[%?%p1%d%t60%e24%;m

breaking that down:

`\x1b[`          CSI
`%?`             "if"
   `%p1%d`       the integer value of the first parameter is non-zero
   `%t`          "then"
        `60`     curly underline code
   `%e`          "else"
        `24`     no underline
   `%;`          "endif"
`m`              end of SGR sequence

That might be sufficient. Technically, that first parameter should be tested against 1 for normal underlines, 2 for double, 3 for curly, 4 for dotted and 5 for dashed as well, but I don't know if nvim uses any of those.

If you could give that a try before I come back to this a bit later, that could save me a bit of time this evening!

[wez]

Well, good news and bad news.

Good news: master now supports the kitty style colon-based sequences for underlines. The 60, 61, 62 sequences I mentioned above no longer function in master; I didn't see the need to preserve something that I made up for this!

My earlier example for vim needs a small change:

" Inform vim how to enable undercurl in wezterm
let &t_Cs = "\e[4:3m"
let &t_Ce = "\e[4:0m"

Bad news: For nvim, I don't see a way to override terminfo for it without actually overriding terminfo :-/

I've added a wezterm.terminfo source to the repo; you can install it via:

$ curl https://raw.githubusercontent.com/wez/wezterm/master/termwiz/data/wezterm.terminfo | tic -x -

Then you can set TERM=wezterm and nvim will know how to activate undercurl.

I need to put a little bit of thought into properly versioning the terminfo file(s) as they evolve. As I mentioned above, I'd love for these to not need to change often, or even be required, but for software like nvim it's difficult to avoid it!

[ssoriche]

Thank you. I explicitly set the $TERM prior to launching Neovim like env TERM=wezterm nvim and I now have undercurl in all my spelling mistakes.

I haven't looked yet at if I can add the terminal configuration directly within Neovim so that I don't have to set it on the command line, but I think I saw something yesterday suggesting that it could be done.

The curl command provided above didn't work for me on macOS Catalina, the tic command didn't want to read from STDIN, I used the following:

tempfile=$(mktemp) \
  && curl -o $tempfile https://raw.githubusercontent.com/wez/wezterm/master/termwiz/data/wezterm.terminfo \
  && tic -x -o ~/.terminfo $tempfile \
  && rm $tempfile

In case someone else stumbles across this problem.

[wez]

I think the only thing left to do here now is add some docs for this. Did you find a way to get nvim to use this feature without needing to install the terminfo file?

[wez]

I made an FAQ entry here:

https://wezfurlong.org/wezterm/faq.html#how-do-i-enable-undercurl-curly-underlines

[ssoriche]

Thank you. I was digging through Neovim documentation and libraries to see if there's another way that the terminal info could be set, including looking at using vte-256color, but ran into a dead end. Either through my understanding of how this all works, or support not being there for another method.

[allumik]

Using Iosevka Term on version 20210314-114017-04b7cedd through tmux with options set -sa terminal-overrides ',*:Smulx=\E[4::%p1%dm' works, but the undercurl looks a bit off and there are no colors. I tried it also with Hack Nerd Font and got the same result.

For comparison the same in kitty:

Also tried without tmux and got the same weird undercurl, but this time coloring works:

Any hints for tmux or wezterm configuration or is this some font handling issue?

[rawhat]

I didn't see anything listed here about it, but is this expected to work on Windows as well?

Screenshot for comparison (the front window is running on Windows 10).

Apologies if I missed something!

[wez]

@rawhat if you use wezterm ssh or Multiplexing then undercurl works. Otherwise, wezterm uses the windows PTY layer which doesn't support passing through undercurl. microsoft/terminal#7228 is tracking that issue in the Windows Terminal project

[kiyoon]

It seems like with Gnome Terminal on Ubuntu 22.04, undercurl works out of the box in neovim without needing the terminfo database.
With WezTerm it is still required to download the terminfo.
Any idea why? Maybe the recent xterm-256color terminfo has this spec?
Sorry, I'm not an expert on this topic

[wez]

@kiyoon neovim must have some built-in logic to decide whether it can use undercurl independently from what is in the terminfo file for xterm. I would suggest opening an issue with the neovim folks to see if they can extend it to recognize wezterm as well.

[kiyoon]

@kiyoon neovim must have some built-in logic to decide whether it can use undercurl independently from what is in the terminfo file for xterm. I would suggest opening an issue with the neovim folks to see if they can extend it to recognize wezterm as well.

Thank you for your reply. I just made a request there.

[1ocate]

@wez
Hi I'm really love wezterm and thank you for develop this great termianl.

I use westerm mac and window wsl.

And follow this, I can see undercurl from neovim with my mac.

But with windows (WSL2) I try this same thing not working....

It is relaction with this?
microsoft/terminal#10783

Also I attach my settings.

Thank you so much.

[wez]

@1ocate see #415 (comment)

[1ocate]

@1ocate see #415 (comment)

Ah I think I see all comment.. thank you reply and clear explain

[trev-dev]

I'm doing something which may turn out to be silly. I have wezterm installed natively on Windows, and $ printf "\x1b[58:2::255:0:0m\x1b[4:1msingle\x1b[4:2mdouble\x1b[4:3mcurly\x1b[4:4mdotted\x1b[4:5mdashed\x1b[0m\n" results in plain text, no undercurls/lines in my wsl2 instance. Is there a work around for this or should I surrender to using xserver?

Edit: The underlines/curlies do work with xserver, ugly white window frame and all

[wez]

try using wezterm ssh to connect to your wsl2 instance; that bypasses conpty. read more about conpty
See also: #415 (comment)

[trev-dev]

Thank you @wez

[jackfranklin]

@wez sorry for the stupid question, but how do I connect to a wsl2 instance over ssh? The docs say that it only works over WSL1, but I am struggling to find an example. Thank you :)

[kiyoon]

@jackfranklin if you want to connect from native Windows to WSL2, I think you can do ifconfig inside WSL2 to get the ip for that instance. Sorry if that wasn't your question.

[wez]

@jackfranklin you'd start an ssh server inside the WSL2 instance, determine its ip (using either ifconfig or ip addr), then you can wezterm ssh THAT-IP

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.