Protocol extension: multiple cursors #720
Comments
|
Personally, I have never needed multiple cursors, but sure, I can see how they might be useful. It isn't particularly hard to design a protocol to do this in kitty, but it should ideally have involvement from somebody who is implementing this feature in some terminal editor, so that the protocol can be designed to meet their needs. If you can get somebody like that to join the discussion, I will be happy to work with them on designing and implementing such a protocol in kitty. I'll leave this issue open for a while, lets see if there is buy-in from editor developers. |
|
Good news! Let me ping a few kakoune's contributors directly, to see if anyone is interested in getting the native support for multiple cursors from terminal emulators: @mawww, @lenormf, @Delapouite, @alexherbo2, @casimir, @danr, @ekie, @occivink. |
|
Hello, For Kakoune, I believe an easy approach for that would be an attribute, similar to underline/bold/itallic... This attribute would just mean "highlight each character as-if they had the cursor on top of them". This would stay separate from the "real" hardware cursor, which would remain at the hardware cursor position and would be displayed according to the current cursor display mode. The cursor attributes would not respect the current cursor display mode, as it could make sense to hide the real cursor while still display various characters with the cursor attribute, we might even want multiple cursor style attributes (block/vertical line), combined with existing blinking attribute. The tricky part for Kakoune is that we use ncurses for output, which unfortunately does not give us a way to use arbitrary attributes, so before being able to support this, we would need to rewrite the terminal UI code to replace ncurses (at least for the display parts). Those are my quick thought on that, but to be clear, I dont consider this a priority in Kakoune at the moment, and it depends on replacing (parts of) ncurses, which is a significant amount of work. |
|
That has a few concerns:
I think a better design is to have a dedicated escape code to create/style/position/delete extra cursors. This has the additional advantage that you dont need to change our ncurses based code at all. Simply write the extra escape codes to stdout in between calls to ncurses' API. Offtopic and a bit ranty: ncurses is a horrible library, I strongly urge you to get rid of it anyway for your own future sanity. |
|
Regarding your concerns, I was thinking just do exactly as the terminal does for any other attributes, my idea was that displaying characters as-if a cursor was there would work exactly like displaying say an underline attribute on a single character. In Kakoune display code, we only know of the cursor position in order to tell the terminal at the end of redraw where to place the hardware cursor so that it can use that as a hint for its eventual widgets (such as displaying an emoji picker, or placing the input method window OnTheSpot...). The rendering code itself just paints the full window with ncurses, and cursors have no special semantics, they are drawn exactly as the rest (Kakoune display structure is basically a list of For the implementation side, I am not sure how kitty paints its cursor, but couldn't it reuse the cursor painting logic and apply it if the cursor attribute is on when it paints a cell ? No need for a second pass. |
|
The problem is that different attributes are treated differently in various situations. For example, reverse video has no effect, background may or may not change depending on the bce terminfo property, underline may or may not take effect depending on the character drawn, etc. So the semantics of this "cursor attribute" would need to be defined carefully keeping in mind interactions with all other terminal escape codes. As for implementation, kitty does not draw individual cells, it is GPU based, which means it draws all cells at once, in parallel on the GPU. So while this is fine for block cursors, which as I said are drawn as normal cells, just with colors changed, for non block cursor there is a separate draw call needed. This separate draw call currently draws only a single cell where the hardware cursor is. With your proposal it would need to draw all cells, since it cannot know which cells have special cursors in them. And worse, it would have to do that all the time, even if no special cursors are being used, which will be the case the vast majority of the time. With my proposal, just as you tell the terminal where to put the hardware cursor after drawing the lines, you tell it where to put the extra cursors. This is low overhead for you (you dont have to maintain an extra attribute per character which is used only rarely) and low overhead for the terminal, for the same reason. Of course, it does mean that it works differently from how you do special cursors now, which I guess, is why you dont like it. |
|
And I would like to add, semantically, a cursor is not a text attribute. A cursor is instead an element of the UI that is independent of the text under it. Treating it as a text attribute seems wrong to me. |
|
I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ? Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...) Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though. And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is). All that said, I do not care that much, as I already said this is not a priority for Kakoune at the moment. Cheers. |
|
On Mon, Jul 16, 2018 at 11:22:00PM -0700, Maxime Coste wrote:
I still dont quite understand why drawing a cursor on top of a cell cannot be done while the cell is drawn, even when done on a GPU, how is that fundamentally different from being able to draw a colored wavy underline on that cell ?
Underlines are drawn as special characters. I suppose I could draw
cursors as special characters as well, but that comes with a new set of
overheads, essentially on every render call a byte of extra data would
need to be sent to the GPU, and the GPU would need to perform several
extra calculations per pixel on every render. The point is not so much
that this cannot be done, but that it adds extra overhead in the case
of no special cursors, which is by far the common case. I would like a
solution that means the CPU/GPU are doing no significant extra work when there are
no extra cursors.
Placing cursors after the fact seems pretty complex to me, we now have some additional state to track between the application and the terminal emulator, which is more potential bugs, and we need a new protocol to control those (whats the lifetime of those cursors ? do they disappear if I repaint the cell ? Does the escape sequence that creates them replaces all the existing ones, or add new ones ? etc...)
Any text attribute is also state that needs to be tracked between the
application and the emulator, there is no difference there. The lifetime
of the cursor is simple: It exists until deleted by the application or
on transition to/from alternate screen mode or on reset.
Repainting the cell has no effect on the cursors, just as it has no
effect on the hardware cursor. The extra cursors are semantically
indistinguishable from hardware cursors. This seems like the most
natural behavior.
Using an attribute is pretty easy, and avoid inventing something new (although it seems there is still a few questions to answer, I was not aware that terminals have different behaviours on cell erase depending on the attributes the cell contains). The alternate implementation would not be difficult to write though.
See the discussion of background color erase in the ncurses FAQ for some
of the complexity of this.
http://invisible-island.net/ncurses/ncurses.faq.html#bce_mismatches
And finally, the semantics of cursors in a fullscreen terminal application really only concerns the application and the user, I dont really see what the terminal could be doing besides displaying those, which is why I though attributes would make sense (the hardware cursor though has semantic meaning for the terminal, it tells it where to draw, and can act as a hint for where user attention is located, which is why I proposed to keep it as-is).
That's true, however, I dont see what it has to do with the mechanism of
transmission of the location of the extra cursors. Using either my
proposal or yours the terminal still knows where the extra cursors are
and still has to draw them. The only difference is, in one case that data
is tied to text state which is semantically orthogonal to UI state and
cursors are UI not text.
|
Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).
Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.
Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursors by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them"). |
|
On Tue, Jul 17, 2018 at 12:15:53AM -0700, Maxime Coste wrote:
> I would like a solution that means the CPU/GPU are doing no significant extra work when there are
no extra cursors.
Aren't we already paying that price to support wavy underlines ? I assume you disable that if there are no wavy underlines to display, which you could do as well if there was no extra cursor to display (the real hardware cursor does not need to go through this path, it could still be handled as it is right now).
The thing about GPUs is that they dont like branches (it breaks
parallelism). So underlines are implemented as a blend of colors with an
alpha which is set to zero when no underline is present. In order to do
cursors that way, would imply an extra blend operation per pixel (EDIT: and an extra texture sample per pixel)
> Any text attribute is also state that needs to be tracked between the application and the emulator, there is no difference there. The lifetime of the cursor is simple: It exists until deleted by the application or on transition to/from alternate screen mode or on reset.
Text attributes is state we already know how to track both on application and terminal side, tracking additional cursors is more involved, imagine an app with multi-cursor crashes and hence does not remove them, the user now has a shell with multiple cursors displayed on the screen, only one of which is relevant.
Exactly the same thing would happen if the application crashes while
using a cursor text attribute. The screen would still have lots of extra
blinking cursors. In general, terminals are not robust against
application crashes, you pretty much always have to reset the terminal
after a crash.
> Repainting the cell has no effect on the cursors, just as it has no effect on the hardware cursor. The extra cursors are semantically indistinguishable from hardware cursors. This seems like the most natural behavior.
Do you mean that outputting a character to the terminal would insert it at every cursor position, and move those cursor by one cell ? I dont think it would be manageable, I think only the hardware cursor should have those semantic properties, the other cursors are strictly for display (which is why in my view attributes makes a lot of sense, it just means "display those cells as-if there were cursors on them").
No, extra cursors are entirely unaffected by text operations in any way.
Repainting a cell, outputting a character, erasing the screen, erasing
lines, words, characters, etc, all have no effect
on the cursor. The one exception is, perhaps, scrolling the screen,
though I can see arguments for either way in that case.
|
|
Hi guys, The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far: On one hand this sounds like a bit of an overkill to me, on the other hand I can also imagine other useful situations. E.g. even a relatively simple text editor such as "joe" can show multiple viewports to the same file at once, it would be nice if the cursor was shown in all of them (or would it be distracting, since the user couldn't be sure which one was the main one, i.e. the one which will start scrolling to keep the cursor inside the viewport, and where special operation (such as Find) will work?). Tmux-like apps could have synchronized (multiplexed) input sent to all panes, and show the cursor in all. Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature. I don't really care about ncurses either, and I don't care about hacks where an app uses a mixture of ncurses and off-ncurses output. Ncurses already has quite a few limitations that make it unsuitable for most "big" projects. E.g. due to its palette approach, it's impossible to display arbitrary (not predefined) colors as they arise (e.g. "mcview" cannot display ANSI colors like "less -R" does, "watch" cannot support 256 colors etc.), and then we haven't even talked about true colors. Then there's the colored and wavy underline, the explicit hyperlinks (VTE and iTerm2; not yet in Kitty), and presumably more. Any application that wishes to use multiple cursors probably sooner or later will also wish to use some of these as well, so will have to switch to its own screen drawing (or some other library) anyway. I think you guys have already agreed that there would still be only one "main" cursor (where new incoming data is printed etc.), the rest are "extra" cursors. I concur. Usually the user probably wouldn't be able to tell which is the main one, but there might be exceptions. E.g. when you press Ctrl+Shift+U to enter a Unicode character, Kitty brings up its own stuff at the top of the window, so this would be unaffected; however, VTE presents an inline box at the cursor, I think it would still go to the "main" one. Other emulators have visual aids helping you locate the cursor (all the time, or upon a keypress), they might decide to highlight the main one only, not all of them. In VTE we experimented with hiding the cursor on focus out as pretty much every graphical app does; even though we dropped this idea, it might make sense (I'm not sure, we'd need to try it out) to hide the "extra" cursors on focus out. With this hidden concept of "main" vs. "extra" cursors, the extra cursors carry some semantics in the applications (text editors) towards the users, but they don't have any semantics for the terminal emulator. For the terminal emulator it's just a visual attribute. As for the overall design (just another attribute, or a totally different kind of escape sequence updating and maintaining the extra cursors' location) I'm firmly in favor of the "just another attribute", that is, mawww's approach.
With this approach, this question is already answered. The same should happen that to any other attribute, there's absolutely no point in treating this attribute differently. The background color wrt. bce is a bit special so it deserves a sentence: I don't see any point in applying the new "make it look like as if it was a cursor" attribute to the rest of the line in case of a bce, so still the background color should be the only special one here, the new attribute should behave just like the existing ones (e.g. underline, strikethrough) do and not get applied here.
That's exactly what happens now with any attribute that an app needs to modify, so I don't see any problem here.
In VTE there's no special handling of reverse or underline anywhere. Why is there in Kitty? Do we miss anything? I believe all our non-background attributes have the very same handling.
It seems to me that for you, "low overhead" means saving a bit, or some CPU cycles. For me, "low overhead" primarly means cleaner code, saving developer time, and hence reusing existing pieces wherever appropriate rather than coming up with something brand new.
So we'd need two brand new escape sequences: Placing an extra cursor at a specific position, and deleting. We'd even need a "delete all", for when the application repaints its screen, e.g. to repair a potentially damaged one. Then maybe we don't even need a "delete one". Plus, figure out how these positions are affected by text being output or deleted, by scrolling (and scroll regions), by clear-to-eol, by window resize and so on and so forth. Sounds magnitudes more complex and error-prone (e.g. there's not even a "standard" terminal emulator to follow; what if two emulators implement some corner case differently and it causes troubles later on?) to me than assigning let's say ANSI 66 and 67 to enable/disable "extra cursor" mode and just doing everything the same way we do for other attributes. A few more points: What would happen on the normal screen? Would extra cursor be supported at all? Could it scroll out to the scrollback buffer? What would happen to it on rewrap-on-resize? Repainting the screen after a text editor operation (e.g. page down) can take a few kilobytes of traffic, which might get stuck over ssh. With off-channel communication for the cursor positions, this might easily result in intermittent visual glitches, the extra cursor being shown at the wrong place. With in-channel communication I believe the visual experience would also be better.
I think it's not just for Kakoune, but for most apps out there. If they have an existing infrastructure of tracking where underline/bold/italic are present, it's way easier to just add yet another attribute to this infrastructure, rather than coming up with a new one for the sake of the cursors. It feels to me that this approach is not only cleaner, but is also easier to implement (replacing ncurses put aside) in most applications, and most terminal emulators (sorry to hear if Kitty is not one of these). |
My bad, I simply didn't know where to open such discussion, and since I'm using kitty and I saw it already implement a few protocol extensions, I decided to propose here yet another one. Of course this is intended to be adopted by other terminals as well, so your input is very valuable 👍 |
Fair enough, actually there's no right place (other than cross-posting to a few). Kitty is probably the "bravest" emulator now, experimenting with new features, so it's a great place to start. I've notified VTE's main developer, as well as iTerm2's developer. Thanks for posting this idea! |
|
On Tue, Jul 17, 2018 at 04:59:07AM -0700, egmontkob wrote:
Hi guys,
The issue starts by saying "More and more text editors", yet, surprisingly to me, only one terminal emulator is involved in the discussion. Do you guys intend this to be a Kitty-only feature, or one hopefully adopted by other notable terminal emulators as well? In the former case, let me know and I'm outta here leaving the rest up for you to discuss :-) In the latter case, my (vte/gnome-terminal developer) thoughts so far:
You are welcome to contribute to the discussion.
Saving a few CPU/GPU cycles in one particular terminal emulator should be the very last factor in designing this feature. With all respect, I just cannot care at all which approach saves a few row scans or whatever operatoins in Kitty (or in VTE, or any other emulator). We should go for the design that's simple, clean, builds on top of what we already have, makes it relatively easy for apps to adapt this new feature.
It's fine that it does not matter to you, it does however matter to me.
So let's not just dismiss it out of hand. Not everyone is happy with
just waving their hands and saying its a few CPU cycles, who cares.
Those few CPU cycles you so casually dismiss for every feature (I've
heard this argument form you before) add up over time. For proof of
that, just compare the CPU usage of kitty+X when scrolling a file in
less: https://sw.kovidgoyal.net/kitty/performance.html
Now to the rest. Rather than responding point-by-point, let me
summarize how the two proposals will work in practice, from the
perspective of the bytes to send to the terminal.
Let's take the use case illustrated by @maww namely having the data
stored in lines which contain runs of formatted text. Suppose we want an
extra cursor at column 2 in the line ABCD
1) Using a text attribute
Bytstream
<SGRCODE to set text attributes> A <SGR code to set extra cursor
attribute>B<SGR code to reset extra cursor attribute> CD
2) Using an escape code
<SGC code to set text attributes> A <ESCape code to set extra cursor>
BCD
So, in this case my proposal is actually simpler to implement for an
application developer.
Now suppose that the editor wants to exit multiple cursor mode:
1) using a text attribute
Two possible approaches:
a) Redraw entire screen
b) manually move the cursor to every extra cursor location and output
the character there again without the cursor text attribute.
2) Using a separate escape code
Simply output the escape code to delete all extra cursors
Again, my proposal is much simpler.
Finally, suppose that the editor wants to change the set of cursors
displayed, hide some, move some, etc.
1) Using text attribute
Redraw entire screen
2) Using separate escape code
Hide all cursors and then either redraw entire screen or just redraw the
cursors
In this case it is pretty much a wash.
I think this conclusively demonstrates that my proposal is actually
*easier* to implement than using text attributes for application
developers (putting aside issues with ncurses, which I agree with
@egmontkob are irrelevant).
Now to address a few points raised by @egmontkob in detail:
P1) there needs to be only a single escape code in my proposal. It can
take parameters to decide what effect it has, i.e. creating a new
cursor, deleting a cursor, deleting all cursors, whatever.
P2) extra cursors in my proposal have no interactions with
erasing/deleting/printing characters lines etc. So there are no
complications there that emulator developers can get wrong.
P3) Regarding normal screen and scrollback
In my proposal you can have a separate set of cursor for the normal and
alternate screens and they obviously dont move into the scrollback
buffer, since they are not text attributes. If you do want to make them
text attributes, you would need to waste even more CPU cycles sanitizing
lines before copying them into the scrollback.
And finally, let me re-iterate my main concern, that no one seems to
want to address:
Cursors are **not text attributes**. They are part of the UI. In no other
display system is cursor information carried by text attributes.
Nowhere, ever. The very idea of putting cursor information into text is
a horrible hack and requires extra-ordinary justification.
|
|
And let me add, that if there is some use case I missed in my bytestream demonstration above, feel free to point it out, and I will attempt to show how it is easier to do using dedicated escape codes rather than overloading text attributes. And just to reinforce the point, @mawww my proposal allows you to continue storing extra cursor information as text attributes in you application, if that is what you want to do. The terminal emulator does not care what internal data strucutre you use. This discussion should be about the protocol between the application and the terminal emulator and the interactions of the extra cursors with other terminal commands such as erase, scroll, print characters, etc. |
|
Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:
No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.
Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.
For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.
I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach. Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar between characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless). While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it. So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:
|
|
On Tue, Jul 17, 2018 at 06:55:10AM -0700, egmontkob wrote:
Sticking to the "just another attribute" philosophy for a while, which I'll get back to later:
> 1) Using a text attribute [...] So, in this case my proposal is actually simpler to implement for an application developer.
No, it's not easier to implement. It involves fewer bytes of traffic. It would be easier to implement if it was the very first attribute to handle. But assuming that other attributes are already handled the same way, it's easier to stick to that same method, rather than coming up with something new. You miss the fact that the first variant is already implemented and hence doesn't need to be implemented again, while the second isn't yet.
Already implemented by what? Nothing I know of. To take the concrete
example in this thread of kakoune, it uses ncurses, so to implement this
it would either need to ditch ncurses or get ncurses to implement it. In
either case whoever does the implementation, it is easier to output fewer
escape codes than more escape codes.
As I mentioned to @mawww in my follow up post, he can keep using extra
cursors as text attributes in his internal implementation if he likes to
do that, the only difference between the two approaches is in one case
you output more new escape codes than in the other. So when he is
translating his internal text attributes into escape codes for the
terminal he would do less work with a dedicated escape code.
The point is that his total draw cycle appears to be to redraw the
entire line on changes. In this use case, as I demonstrated, using a
dedicated escape code is better or at least no worse than using a new
text attribute.
BTW @mawww apologies if you are female, please read the above he as
she.
> [...] Again, my proposal is much simpler.
Again, it would be simpler if either approach would need to be implemented from scratch. That isn't the case. One is already implemented, while the other one isn't yet.
See above.
> Finally, suppose that the editor wants to change the set of cursors displayed, hide some, move some, etc. 1) Using text attribute Redraw entire screen [...]
For some mysterious reason here you forgot about the possibility of redrawing only the affected characters. Nevermind.
That is exactly the same as redrawing only the affected cursors with the
escape codes and is why I concluded this use case is wash. Apologies for
neglecting to spell it out.
---
> And finally, let me re-iterate my main concern, that no one seems to want to address: Cursors are **not text attributes**.
I have to admit that I missed this bit, or more precisely, missed the strong emphasis you put on this approach.
Given that block cursors are a legacy thing, the cursor in all modern apps is a vertical bar _between_ characters, which is reasonably well emulated using the I-beam cursor shape in terminal emulators, I have to stop here for a while and let this approach sink in. I think you might have convinced me (making our previous debate pointless).
While now I agree that this is the right philosophy, I still believe it makes implementation harder (something new needs to be designed and implemented, rather than reusing something that already exists), and hence it's not necessarily the most practical choice, but probably the correct one. I'm not yet explicitly in favor of this approach (give me a few days to ponder about this), but I'm okay with it.
Sure take your time, questions like this deserve due consideration.
So to move forward from here, I guess we'd need escape sequences to some of these (which ones exactly?) and answers to the questions:
- Leave a "trail", place an extra cursor where the real cursor currently is.
Not sure what you mean by this? The real cursor will remain where it is.
Are you saying, you would like to distinguish it from other cursors? If
so isn't the point of multiple cursors that they all behave identically,
so that if you insert text, it is inserted at all cursor positions?
- Place an extra cursor at an arbitrary (x,y) position.
Easy to do just have the escape code take an extra optional parameter or
require the application to move the real cursor to place the extra
cursor.
- Would printing data over an extra cursor (of if thinking in I-beam shape: after the extra cursor) leave it there, or would it wipe it out?
I strongly vote for leave it there, otherwise every time your print a
character you have to check if there is an extra cursor there. And then
if you print characters in "insert mode" you have to check for extra
characters all down the line and potentially cascading all down the
screen thanks to line wrap.
- Remove the extra cursor from the real cursor's current location.
Not sure what you mean here? If there is both a real cursor and an extra
cursor at some location the terminal will draw only the real cursor.
- Remove the extra cursor from an arbitrary (x,y) position.
Again no problem, simply add a parameter to the escape code. By the way
for how I think this escape code should be designed see the graphics
escape code that kitty uses, it also takes lots of parameters.
https://sw.kovidgoyal.net/kitty/graphics-protocol.html
- Remove all extra cursors.
Same, parameter to escape code.
- How would an extra cursor be affected if the content underneath moves in any of the four directions? (Vertical scroll, or a previous character in the line inserted/deleted.) Would the cursor follow them, or stay at the same absolute place, or get wiped out?
IMO the extra cursor should not be affected at all. This is the simplest
way and the one least likely to be implemented incorrectly. The one
exception is perhaps vertical scroll, but I dont feel strongly about it
either way.
- Would the (x,y) positions affected (and bound) by the scrolling region?
IIRC (and I could be wrong) scrolling regions only affect cursor
movement commands. There are no movement commands as such for extra
cursors, so the question does not arise. (Instead to move a cursor, you
remove and re-create it).
- What to do on resize? Is it okay to wipe them out all (as e.g. the scroll region is reset), expecting the app to do a full repaint anyway?
I am OK with either wiping them all or just bounding them in the new
size as you do with the real cursor.
- Nit: How would a cursor positioned to the middle of a wide character be handled?
The same way the real cursor is handled in this case, i.e. it would
stretch over two cells if block or replace and one cell if beam (it is
an error to place a cursor in the middle of a wide character).
- Nit: With I-beam shape in mind, would it be possible to place a cursor after the last column?
I dont see the point, I think this should be the same as how the real
cursor is handled.
- Nit: Would all extra cursors automatically share the main one's properties (shape, color, blinking)? Or could they each have their own ones?
I am in favor of them all being the same. That is the use case needed
for editors and allowing them to be different is a fair bit of overhead
for no actual use case.
- Anything that I'm missing or overcomplicating? :)
There's the question of clear screen and reset. However,
let's leave the fine details to the second stage after we have consensus
on the basic question of whether to implement this or not and if so,
whether to use text attributes or extra escape codes.
See my next post.
|
|
So for the moment I propose we focus on Stage 1, which is answers to the
|
|
Though multicursor support is in the works for Nvim, I don't yet see a need for special terminal features.
The way I envision multicursor is a "primary" cursor plus other "marker" (secondary) cursors. The secondary cursors don't need to look identical to the primary cursor, they are just visual clues. I would guess that a human is not greatly aided by all of the cursors looking like the primary cursor, rather it may increase confusion. Actually, I'd be more interested in new text attributes as @mawww hinted. E.g., ability to display (1) a hollow "box" around a cell, (2) a bar on the left side, and (3) bar on the right side. Entirely orthogonal to cursor(s). |
|
iTerm2 author here: I'm fine with adding this, but I don't want it to be an attribute. I only have a few bits of attributes left before my struct needs to grow. That will increase memory usage and decrease cache efficiency for an attribute that would almost never be set. That's an implementation detail, but it's an important one. More generally, it's easy to draw an analogy between the "real" cursor and the list of "virtual" cursors. It's a thing that has a coordinate that gets drawn on top of the text and there are some rules for how it's drawn that are configurable both by the user's preferences (e.g., color, shape) and by escape sequences (shape). Provided we go with a list of cursors, I would be happy to implement it. I think we'd need escape codes to:
I would also propose that virtual cursors change position only when told to and have no automatic behavior. This pushes the complexity into the app that wants them, but makes it much more predictable in the face of scrolling regions and double-width characters. However, I'm not quite sure how this would work with an ncurses-based app. Someone who has more experience can perhaps comment here. |
I'm not so sure, I would say it is quite rare when I actually need to know which of the cursors is the primary one. Look in the gif in the first post, I'm manipulating a few lines and I want the same manipulation to happen in all lines, regardless of where is the primary cursor position. I would much prefer to see all cursors turn into identical vertical bar when I'm about to prepend the word "private" on every line, because this would indicate to me that I'm about to insert some text on all of these lines. Sometimes it is helpful to know where the primary selection is, not the cursor (at least in kakoune), because there are some extra actions you can do with the primary selection (search for the word "main" here), but no actions to do with the primary cursor — and the primary selection is highlighted by using the white text color. However the above is based on my experience with kakoune, I realize that the situation might be different in other editors. |
SGR 51, 60, 62 might be relevant here.
I can easily think of use cases for either approaches. Maybe the best is if we consider supporting both kinds straight from the beginning.
I guess I'd rather they scroll along with the content. Otherwise e.g. cat'ing a binary file may result in visual glitches (additional cursors) that don't even scroll out as you tap Enter. |
You still don't understand what I'm speaking about. This will probably be my last attempt in explaining. Let's assume a non-ncurses based editor (there are plenty of them). Presumably it already handles bold, italic, underline etc. By some internal means it figures out which cells it wants to render with such attributes, and then implements telling this to the terminal emulator. Given this current state, it's much easier to add another attribute that follows this pattern, than an attribute that uses a totally different approach. Easiness doesn't mean fewer bytes to be emitted, easiness doesn't mean something that would be easier to implement from scratch, easiness means easier to implement and maintain given the current codebase, that is, less deviations and less additional code to introduce, reusing whatever's already available (i.e. the way bold/italic/underline are handled by this app). |
|
On Fri, Jul 20, 2018 at 09:16:58AM +0000, egmontkob wrote:
> If the application is maintaining a list of extra cursors anyway, the id can simply be the number in that list. I dont see how it adds significant extra burden.
_If_ the application is maintaining a _list_ of extra cursors anyway. What if not? What if it uses just another bit per cell in its already existing internal WxH matrix denoting what's already on the screen (i.e. treats them as if they were text attributes, as you told mawww it'd be fine)?
It doesn't matter how it stores them. In either case you have to output
the escape codes for them, when you do simply output an incrementing
counter. It's two extra lines of code.
> Not to mention that knowing the location is not always easy (think of wide characters).
I'm not sure I get it. I can't imagine a text editor that doesn't have a consistent internal representation (including wide characters) of what it believes is on the screen. If the code places a cursor at (x,y) then removing a cursor specifying the same (x,y) coordinates will remove that. If an app places an extra cursor at the real cursor's current location, then later repositioning the real cursor there and removing the extra cursor _should_ (see below) also remove it.
Remember that the escape code places the extra cursor at the current
cursor position, bu default, which apps often dont know, and cannot in
general know unless they either query the emulator or have a wcwidth()
implementation that exactly matches that of the emulator. Obviously when
the app specifies the location (x, y) in the escape code, it knows the
location. When it does not, it may or may not. ids on the another hand
it always knows.
And you forgot about scrolling and resizing of the screen. If we make
the extra cursors move with scrolling then the app has to
update their locations just to address them.
Similarly after a resize it cannot know the new locations. And given
that screen scroll all the time, for example when wrappiing is enabled,
this is not a small issue.
ids are just better :) And sorry, but needing to add two extra
lines of code in applications that dont care about ids is not an undue
burden, to make the implementation robust for applications that do care
about addressing individual cursors.
> [vertical scrolling] Still it's not an absolute no-no from me.
Another use case that occurred to me is if the editor crashes. This is not something that has never happened to me. The prompt might be displayed amidst the leftovers from the file's contents, it might be colored (but _my_ prompt does reset that), there's no scrollback buffer since we're still on the alternate screen, but I just press a couple of Enters to get to the bottom of the screen and it's usable. Repeating myself: I wouldn't want cursors to remain there cluttering my command line.
If the editor crashes, I dont see how you can possibly hope to recover
other than by doing a reset. Everything from colors, line discipline,
echo mode, all the dozens of bbits of state in termios and the emulator,
the keyboard mode, etc, etc, etc could be wrong. And reset or
clear removes extra cursors. Terminals are not robust against program
crashes and never will be. It's pointless worrying about it.
> [one escape sequence for multiple operations]
We've heard about at least one app that would always clear all existing extra cursors and place them again (rather than keeping track of IDs, moving them etc.). Being able to specify multiple actions in a single sequence has the small advantage that the terminal emulator can handle all of them as an atomic step. That is, even if the connection stalls while receiving them, it could avoid the visual flickering of removing and later at the same place reinstalling some extra cursors. My bad for not mentioning this the last time. It's by far not the most important issue here, but one that we might consider nevertheless.
If the connection stalls in the middle there is no visual flickering,
you get some cursors that appear before and some cursors that appear
after. Just as you get some text that appears before and some text that
appears after. Terminals do not paint the screen in an atomic manner,
they never have and I doubt they ever will. But if they ever do,
whatever mechanism they use for it, can just as well apply to this
protocol just as it would have to apply to all the rest.
|
|
And just because I'm bored, let's do a bit of simple maths to calculate the probability of their being one extra cursor on the screen after catting a 10MB binary file. You need a minimum of an 8-byte sequence to create an extra cursor. The probability of such a sequence is ((1/255)**7) * (1/245) which is 1e-20 (technically its a bit higher since you can have longer than 8 byte sequences as well that create extra cursors, but I doubt the contribution from them will be significant compared to the shortest sequence). There are approx 1e-6 8 byte sequences in a 10 MB file. Which leaves you with a first order probability of 1e-14 of seeing a one single extra cursor on your screen after catting a 10MB file. I can live with that. |
If the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs. If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs is an additional, nontrivial burden.
Why can't? Won't it be part of the specification?
Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something shouldn't change, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible. |
|
On Fri, Jul 20, 2018 at 03:06:00AM -0700, egmontkob wrote:
> In either case you have to output the escape codes for them, when you do simply output an incrementing counter. It's two extra lines of code.
_If_ the app always removes and reinstalls all the cursors at once. If that was the only intended way of using these escape sequences, there'd clearly be no need for IDs.
If an app decides to use the more fine grained sequences, that is only remove the cursors that are no longer necessary, only add the newly appearing ones, and leave untouoched the ones that don't move, then having to have IDs _is_ an additional, nontrivial burden.
Huh? It is precisely in that case that having ids is *superior* to not
having them. If the application wants to address individual cursors in
the terminal, then it must be able to address them internally as well,
which means they already have an internal id (or at least some kind of
internal data structure to represent an individual cursor). Which means
there is no extra burden giving them an id in the escape code, simply
add an id field to the internal structure which uses a globally
incrementing counter, again two lines of code.
> Similarly after a resize it cannot know the new locations.
Why can't? Won't it be part of the specification?
No, at least I dont have any interest in adding it to the specification,
but if you do, feel free to send a PR.
> Terminals do not paint the screen in an atomic manner, they never have and I doubt they ever will.
Not updating the entire screen as an atomic step is fine. What I'm mentioning is when something _shouldn't change_, but due to a lag in traffic, ends up changing back and forth on the UI. This can already happen to the real cursor, but would be nice not to extend to the extra cursors, if reasonably possible.
What shouldn't change? You are sending n cursor creation escape codes
one after the other. The *only* thing that can happen is that they might
not appear on the screen at the exact same moment. Nothing can appear
that later should not appear since you are transmitting the escape codes
immediately after each other, regardless of how many pauses there are in
transmission.
|
This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come up with something new as well.
Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker. |
|
On Fri, Jul 20, 2018 at 03:24:14AM -0700, egmontkob wrote:
> it must be able to address them internally as well, which means they already have an internal id (or at least some kind of internal data structure to represent an individual cursor)
This can be just 1 bit per cell in its already existing matrix denoting what's on the screen, next to the bold, italic etc. attributes. With no IDs. This is what you said to mawww that he'd still be able to do. This is what I believe is the simplest approach to implement this feature in any non-ncurses based app, because it reuses what the app already does for bold etc., without having to come with something new as well.
Sigh, this gain? If it is 1-bit per cell, then individual cursors are not
addressable by the application. The application does not care about ids
and must delete all cursors on each round anyway and so can use a simple global
counter for the ids.
Let me spell it out for you in pseudo code.
```
struct {
bool has_extra_cursor;
SGRAttributes text_formatting_attributes;
char *ch;
} Cell;
struct {
Cell *cells;
} Line;
counter = 1
write(<delete all cursors>);
for line in lines:
for cell in cells:
write(cell.text_formatting_attributes)
if cell.has_extra_cursor:
counter += 1;
write(<esc>_Cid=counter<esc>\)
write(cell.ch);
```
As I said, literally two extra lines of code. Initialize the counter and
increment it.
> What shouldn't change? You are sending n cursor creation escape codes one after the other. [...]
Imagine that there are three extra cursors at positions A, B and C. The app wants to change it to four positions: A, B, D and E. The app's internal logic doesn't wish to fiddle with IDs, so instead clears and reinstalls all the cursors. There's a potential for a flicker at positions A and B, the cursor disappearing for a short time. Being able to specify multiple cursor operations in a single escape sequence could avoid this flicker.
You mean between clearing the cursors and re-creating them? That would
be present regardless, even if the protocol allowed you to create
multiple cursors in one escape code.
If you mean you want to be able to both delete and create cursors in the
same escape code, I have no idea how that would work while still
remaining small and relatively easily parseable. I supose one culd do
something like this
<esc>_C<control data1>;<control data2>;...\_
Doesn't seem important enough to me to bother.
|
Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).
The terminal emulator could decide whether to process individual requests as it encounters them, or batch them up until the terminating |
Sorry, I have to correct myself, I responded too quickly. Screen drawing is probably more sophisticated than the pseudo-code you outlined. I mean they probably do some minimal diff-ing between the old and new buffers, e.g. update only the lines that changed, and maybe even within the line they omit the beginning and trailing portion that remained the same. With your approach, they'd need a bit more than those 2 lines to re-process the extra cursors in lines that didn't change. I'm not saying it's hard to do, but I'm still saying it's an unnecessary additional complexity. |
|
On Fri, Jul 20, 2018 at 03:47:16AM -0700, egmontkob wrote:
> As I said, literally two extra lines of code.
Two extra lines of code, with a design that increases by magnitudes the chance of flickering (an extra cursor disappearing and a few draw cycles later reappearing at the same position).
What on earth have ids to do with flickering? At this point you are just
arguing for the sake of arguing.
And I'll just note in passing that cursors blink, which means that
flickering for them is meaningless.
Hmm, I think I have had it with this discussion. I'm afraid I no longer
have the motivation to implement it if it involves this level of
argumentation.
@everyone: sorry for wasting your time. If somebody else wishes to
implement this in their terminal emulator feel free to use my work as a
baseline, or not, as you wish.
|
|
The maintainer of
|
With the code snippet you showed, the removal of the cursor is followed by potentially up to a few kilobytes of text data (and attribute changing escape sequences), then re-installing the cursor at the same position. The amount of data inserted in between could easily make them no longer fit in the same ethernet or similar packet, eventually noticeably increasing the chance of a network lag kicking in.
or not (oh, wait, it seems to me that in Kitty it always blinks)
A flickering during the "on" state of a blinking cursor is IMHO still a bit annoying.
No. I'm trying to argue (although I'm really lost) because I have a bad feeling that no matter how I phrase my thoughts, I cannot make them come across to you. I mean, if I felt you understood what I'm saying and still disagreed, that'd be fine. But unfortunately I feel like you don't understand what I'm saying (maybe don't take the time or don't have the motivation to try to understand what I'm saying?? I'm really unsure) then it's very hard. I sincerely hope that my feelings are incorrect. I really don't know what I am doing wrong if my thoughts don't come across clearly. I was just about to quit this conversation and let you continue in your preferred way. I'm sorry you did it first and not me, and sorry from others too. This enhancmenet request was filed against Kitty, it's your project, do as you wish. If you reopen and continue the work, I promise I'll stay out of the way. |
|
Sorry, but this discussion has made it clear to me that I lack the patience to steward something like this. As I said, feel free to implement it in your own terminals using my existing proposal as a base or not. If and when some proposal becomes widely adopted/used by editors, I will be happy to implement it in kitty. But I am done spending my valuable time creating proposals just to engage in endless meaningless arguments about them. |
|
@justinmk wrote:
This would be nice. Totally independently of extra cursors, I could see this being useful for other cases besides. They'd be very easy to add as new numbered SGR attributes |
|
@egmontkob wrote:
I hadn't heard of those before. Do you have some relevant documentation? I'd like to add them to my spreadsheet, where I'm trying to keep things like this organised: (I'm happy to add/edit things in there for other terminal emulators, also) |
|
So the "hardly ever supported" (51-65) SGR parameters most likely will continue to lack support in kitty, iterm2, VTE, because of struct space? |
|
If you want them supported, I am not adamantly opposed, but you will need to make a case for it to convince us :) kitty actually has space in its struct at the moment. The downside of supporting this in kitty will be extra per pixel operations in the shader. There are also some subtleties to be aware of, namely those are meant for ideographic characters, I dont know if it is appropriate to simply use them for ordinary characters.And one has to consider what happens with them on wide characters. |
|
And if you do want to make a case, I suggest opening a new enhancement report for it, this one is already quite long :) |
No, I don't. All I know is what's written in ECMA-48 and the Wiki page linked by Kovid. I don't even know what "ideogram" means, nor have an idea what to do with "underline or right side like", I mean which one? The feature could be useful for nice and compact representation of tables. Double width lines could be achieved by placing e.g. a character with a right side line and a next character with a left side line next to each other. Not sure if it's worth 4 new bits per cell, though. Not sure how we are with free bits in VTE, the main developer recently reorganized the bits and I haven't checked the code since. Just as Kovid, I'd also prefer to see actual (not hypothetical) use cases, utilities willing to add support. If we'd like to move forward, I'd like to propose 4 new attributes for the 4 sides. SGR 51 ("boxed" or "framed") might be a shortcut for enabling all 4 at a time. The wording of SGR 60 and 62 are so unclear to me that I wouldn't want to use them, unless someone has a clarification on them. SGR 53 (overlined, hardly ever used) may be used for the top one, or may be obsoleted by the top one. I'd like the bottom one to be separate from the underline. The underline goes slightly below the baseline of the font, and might be subject to "text-decoration-skip-ink" rendering, curliness, different color (or would we want that for the frame too?) etc. The border bits would I imagine go to the very edge of the character cell, so that adjacent ones are joined even in case of increased line spacing or character spacing. |
|
As for the very concrete escape sequences, we might be thinking along the lines of |
|
I agree with Egmont about 51:n and these being independent of underlines. Those were the choices I would have made myself. The only question is whether to use 51 or some other number. After all some terminal emulators out there may already use 51. I am curious as to what the use case for this is though. Since the border lines could potentially overlap with the rendered character, I dont know how nice it would look (think of wide characters like W that touch the edges off the cell in most fonts. But, as I said, feel free to make your case, preferably in a separate bug report. |
|
I agree that this whole discussion should continue in a separate issue. |
|
As long as we're bikeshedding, I'd like to add an extension to enable atomic drawing so that when the entire screen gets redrawn it doesn't flash a half-drawn screen for a fraction of a second before the rest of it appears. I have a hack in iTerm2 that tries to use whether the cursor is visible to provide synchronization but it is far from perfect. Since that came up in this discussion, I wonder if anyone would be open to the idea? If so I'll open a new issue and write up a proposal. |
|
Sure, I know this bother's Egmont a lot, so it's worth discussing. It should be trivial to have an escape code that turn on and off "pending mode". In pending mode all received bytes would be buffered. You'd probably also need a max pending timeout and a max buffer size. |
|
Sounds good. |
|
Link to spec and issue to host the discussion are here: https://gitlab.com/gnachman/iterm2/issues/6873 |
More and more text editors are adopting the concept of multiple cursors, I'm playing with one of them called kakoune.
In order to draw multiple cursors consistently, kakoune has to hide the terminal cursor completely and "draw" its own cursors by manipulating background color of a cell. This is ugly hack that prevents people from using the much superior native cursor with all of its features (different shape, blinking, proper color).
In vim you only see one cursor, even if you are editing multiple lines, but there you are not editing multiple lines simultaneously anyway (vim waits for ESC to be pressed, then applies the same edit on the rest of the lines).
In kakoune, multiple lines are being edited at the same time, potentially in different positions, so it's really beneficial to show multiple cursors.
What I was thinking about is to propose you to think about multiple cursors as an extension over xterm protocol.
I asked for how this works in layman terms, I was told applications like kakoune or vim take care of handling keypresses and rendering the text on the screen, and then they tell the terminal to put a cursor in a certain place, kind of like
move (x,y). What is missing is the ability to draw more cursors, e.g. likedraw (x1,y2). These "secondary" cursors should basically be a copy of the main cursor in terms of color, shape and blinking. And then there should be a way to remove all "secondary" cursors from the screen.I brought this to you because you understand a lot more about the terminals, and I want to hear your thoughts. I hope I managed to at least explain the need for this feature: multiple cursors is a thing that is becoming more and more popular and demanded by users (here's just one example in neovim), unless terminal allows drawing them, software has to completely take over drawing cursors, and of course software-rendered cursors cannot ever have the features like shapes and colors and blinking that native cursors easily provide.
The text was updated successfully, but these errors were encountered: