Terminal Game Mode - draft by needs

[msokalski]

Terminal 'Game Mode' draft

Goals:

• reliable way of tracking keyboard, mouse and window focus state using simple event reports
• automatically resettable to normal mode in case of ubnormal program termination based on timeout from latest frame marker
• atomic screen updates using frame markers in output stream

---

APPLICATION -> TERMINAL protocol

1. Frame marker Enter game mode

ESC [ ? 2017 ; <timeout> h

<timeout> must be in range 0-9, value of 0 indicates default timeout (configurable in terminal). If application won't send another frame marker within this number of seconds, terminal must automatically switch back to previous mode.

2. Ending frame loop Leaving game mode (prior to timeout from last (re-)enter frame marker)

ESC [ ? 2017 l (Disables game mode)

ESC [ ? 2017 h (Disables game mode and enters Full Keyboard mode)

TERMINAL -> APPLICATION protocol

1. Keyboard events
   Identical to kitty's full keyboard mode, except:

• terminal generates escape sequences for all keys including printable ones
• printable characters are also delivered to application in regular non-escaped unicode encoding, including auto repeated characters.

In example: pressing, holding and releasing A key should produce (assuming capslock is off):

ESC _KpAS ESC \
a
ESC _KtAS ESC \
a
ESC _KtAS ESC \
a
ESC _KrAS ESC \

2. Mouse & Focus events

ESC _K <type> <mods> <cw> <ch><col> <row> ESC \

<type> single byte character, can be:
'A':LOST_FOCUS, 'B':MOTION_ONLY,
'C':WHEEL_FWD, 'D':WHEEL_REV,
'E':RESERVED1, 'F':RESERVED2,
'G':BUT1_DOWN, 'H':BUT1_UP,
'I':BUT2_DOWN, 'J':BUT2_UP,
...
'Y':BUT10_DOWN, 'Z':BUT10_UP

RESERVED1,2 are intended for second wheel (dual wheel mice?)

<mods> is encoded exactly same as for keyboard reports
<cw> and <ch> are character cell size in some sub-character units (ie: pixels), they are encoded as single base85 digit each.
<col> and <row> are 0 based xy coordinates consisting of 3 base64 digits each, that makes it possible to address coordinates in range 0-262143. Coordinates are expressed in same units as <cw> and <ch> (ie: pixels).

Coordinate resolution is 64x greater than terminal size in characters. That enables application to use far more precise input than size of character cell and handle terminal sizes up to 16384 characters! Coordinates should be clamped to terminal size times 64 less one.

Presentation Notification
Notifies application about presentation event along with number of frames processed (marked with BSU / ESU or frame markers). That should happen right before doing swap buffers. If <frames> is greater than 1, it means that drops occurred (too many frames were sent). If <frames> is 0, it means frame queue was drained, so last frame was re-presented.
That enables application to maintain optimal presentation latency and avoiding sending frames that would be eventually dropped from presenting. Terminal should guarantee no drops if frame queue length does not exceed 2 (could be parametrized in terminal).

[kovidgoyal]

Just to note there is already an escape code for atomic screen updates, supported by kitty and a few other terminals (VTE and iTerm). https://gitlab.com/gnachman/iterm2/wikis/synchronized-updates-spec

[msokalski]

So, do you think it is better to keep BSU / ESU for that purpose, even when game mode is enabled? I can remove frame markers from the draft, but I'm also considering adding better control over frame queues, that would enable application to run in sync with terminal's frame presentation (glXSwapBuffers) making animations 100% smooth and without rendering too many frames that would be simply dropped from presenting by terminal. That would require adding terminal -> application notification on every frame 'advance'. So to avoid drops (and save cpu) application should not send new frames to terminal until number of queued frames is > 1. Keeping queue length = 0 could potentially minimize presentation latency (at the risk of reduced animation smoothness).

[kovidgoyal]

Synchronizing individual frames in a text mode game seems like overkill to me. And that should remain independent of the atomic screen update escape code since that is already standardised/deployed.

I suggest for the moment to focus on input handling, keyboard/mouse. Lets not go overboard with requirements.

[msokalski]

Ok, removing - input handling has a top priority for me too. How about window size notifications? I dislike using ioctl(TIOCGWINSZ) on linux and I don't even know how to do it on Mac. Such notification could work well with sub-character resolution mouse input, currently in my draft every mouse message encodes character cell's width and height which could be removed in favor of window/cell size notification(s).

[kovidgoyal]

You use SIGWNCH for window size change notifications.

[msokalski]

Ah, of course, thanks. As far as I remember, linux dropped pixel sizes in winsize this signal (they are 0,0), so I'm leaving character sizes in mouse messages.
Also I'm wondering if game mode mouse reports should use its own LOST_FOCUS message or should it rather interact with xterm's CSI ? 1004 h mode sending CSI I / CSI O?
Other than that, I think all top-priority things are now in the draft, there are other things like relative mouse moves (unclamped, as opposed to clamped mouse pointer to desktop size), frame queue control, etc. but they can be added later if at all.
If there's anything I should clarify / re-work, just let me know.
Thanks Kovid!

[kovidgoyal]

On Fri, Nov 22, 2019 at 01:42:00AM -0800, msokalski wrote: Ah, of course, thanks. As far as I remember, linux dropped pixel sizes in this signal (they are 0,0), so I'm leaving character sizes in mouse messages.

You listen for the signal and when it is received, you use the ictl to get the sizes. xterm and kitty will report pixel sizes as well as cell sizes.

Also I'm wondering if game mode mouse reports should use its own LOST_FOCUS message or should it rather interact with xterm's `CSI ? 1004 h` mode sending `CSI I` / `CSI O`?

I think the existing codes are fine for it.

[msokalski]

So if we want to re-use focus messages, do they need to be enabled separately or we can state that entering game mode enables keyboard, mouse and focus messages (all with single command) and reverts them all to a previous state after timeout expires or user disables it?

[kovidgoyal]

focus messages already have their own mechanism for turning them on and
off. And I prefer having separate mechanisms for keyboard and mouse mode
individualy so that an application can pick and choose.

[msokalski]

Looks like we can narrow and split 'game mode' to 'raw keyboard mode' (only one resetable with timeout) and a separate 'pixel coords mouse mode'. Do you want me to rewrite draft in that way?

[msokalski]

One more thing I'd like to see in 'game' extension(s) is control over current font / tileset. Do you think it would be possible to upload ttf or raster font like psf / bdf or just individual bitmaps with unicode indice? Or it seams to you as overkill?

[kovidgoyal]

On Mon, Nov 25, 2019 at 02:27:33AM -0800, msokalski wrote: One more thing I'd like to see in 'game' extension(s) is control over current font / tileset. Do you think it would be possible to upload ttf or raster font like psf / bdf or just individual bitmaps with unicode indice? Or it seams to you as overkill?

No, kittycan already display arbitrary raster graphics, adding support for application defined fots is a step too far.

[msokalski]

Thanks, so if there's anything else I can do about making game mode implemented in kitty, just let me know.

[kovidgoyal]

No not really, unless you are willing to contribute code, you will have
to wait till I get around to it.

[ctrlcctrlv]

adding support for application defined fots is a step too far.

I might be willing to work on this next, if you'd be willing to merge it @kovidgoyal ?

[kovidgoyal]

How would it work? Fonts in kitty are defined per OS Window, not per
window, for technical reasons. So if the user has a singel OS Window
with multiple windows, changing the font would mean changing the font in
all windows in that OS window.

[ctrlcctrlv]

I don't think that's a big hurdle. Users can just run multiple kitty processes, which on my system (Arch + KDE) is already happening without any direct intervention from me to make it so, it just naturally happens when I press the kitty icon:

[fred@pc buildI]$ ps axu|grep kitty|wc -l
7

It may even be viewed as a feature to persist fonts to multiple windows opened by eg Ctrl-Shift-N which opens a new window in the process.

Alternatively we can just make Ctrl-Shift-N create new processes.

Or...do even more work, and figure out a way to re-key the fonts per window instead.

Similar to #2248, I'd do all the work if we come to an agreement on implementation. I think it sounds like a fun feature, and could really help with games. But I won't work on it if it is not going to be merged.

Here's an implementation sketch:

We have an existing feature called symbol_map, which I think, implementation wise, should be the foundation.

We just let applications redefine the symbol map, and extend the graphics protocol to store font data also. Fonts should be understandable by FreeType/HarfBuzz, and can have OpenType layout.

We also let applications set font-feature-settings (recently merged, tyvm!) for their own fonts.

How powerful we want to make this really depends. If we make it so that the user-defined fonts can only redefine glyphs in the Supplementary Private Use Area-A, we don't really have to worry at all about fonts overwriting. If we make it so they can redefine any glyph, we need to pay more attention.

I made a Snake game in 2012. One of my first programming projects. It works best with square fonts, but most terminals don't have them. This seems like an extension of that work, somehow.

[kovidgoyal]

Well I am, in principle, OK with the following architecture: 1) Create a protocol for terminal applications to send font data to the terminal using either file paths or raw data, just as in the graphics protocol. 2) This font data can be used to define symbols for unicode code points in the private use areas (we can decide which blocks later) 3) The fonts so defined would be added to the font group of an OS window There is still the potential of conflicts since you could conceivably have two different games defining different font mappings running in tow kitty windows in the same OS Window, but this is a relatively rare corner case that I am OK with not worrying about. Incidentally, for your snake game why didn't you use the unicode block drawing/box drawing symbols? kitty for instance has custom rendering for these that means they always render perfectly, regardless of fonts.

…

On Mon, Jan 06, 2020 at 08:32:45PM -0800, Fredrick Brennan wrote: I don't think that's a big hurdle. Users can just run multiple kitty processes, which on my system (Arch + KDE) is already happening without any direct intervention from me to make it so, it just naturally happens when I press the `kitty` icon: ``` ***@***.*** buildI]$ ps axu|grep kitty|wc -l 7 ``` It may even be viewed as a feature to persist fonts to multiple windows opened by eg Ctrl-Shift-N which opens a new window in the process. Alternatively we can just make Ctrl-Shift-N create new processes. Or...do even more work, and figure out a way to re-key the fonts per window instead. Similar to #2248, I'd do all the work if we come to an agreement on implementation. I think it sounds like a fun feature, and could really help with games. But I won't work on it if it is not going to be merged. Here's an implementation sketch: We have an existing feature called `symbol_map`, which I think, implementation wise, should be the foundation. We just let applications redefine the symbol map, and extend the graphics protocol to store font data also. Fonts should be understandable by FreeType/HarfBuzz, and can have OpenType layout. We also let applications set font-feature-settings (recently merged, tyvm!) for their own fonts. How powerful we want to make this really depends. If we make it so that the user-defined fonts can only redefine glyphs in the Supplementary Private Use Area-A, we don't really have to worry at all about fonts overwriting. If we make it so they can redefine any glyph, we need to pay more attention. [I made a Snake game in 2012.](https://github.com/ctrlcctrlv/snake) One of my first programming projects. It works best with square fonts, but most terminals don't have them. This seems like an extension of that work, somehow. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #2145 (comment)

--

_____________________________________ Dr. Kovid Goyal https://www.kovidgoyal.net https://calibre-ebook.com

_____________________________________

[ctrlcctrlv]

2012 was 8 years ago, and it was one of my first programming projects, so I think we can forgive me for not knowing about box drawing symbols, or predicting Kitty's release :)

[ctrlcctrlv]

By the way, the font data, I think it should just be SFNT data (OTF/TTF), no need to reinvent the wheel, and that will make implementation a lot simpler

[ctrlcctrlv]

I just tried it, and for me U+2588 █ (FULL BLOCK) is not square in Kitty?

[kovidgoyal]

On Mon, Jan 06, 2020 at 08:45:40PM -0800, Fredrick Brennan wrote: By the way, the font data, I think it should just be SFNT data (OTF/TTF), no need to reinvent the wheel, and that will make implementation a lot simpler

Yes SFNT is fine, might consider WOFF2 as well, but that is up to you.

[kovidgoyal]

On Mon, Jan 06, 2020 at 08:52:44PM -0800, Fredrick Brennan wrote:  I just tried it, and for me U+2588 █ (FULL BLOCK) is not square in Kitty?

Cells are not square in kitty. You would need two adjacent U+2588 to get a squarish block.

[kovidgoyal]

On Mon, Jan 06, 2020 at 08:44:06PM -0800, Fredrick Brennan wrote: 2012 was 8 years ago, and it was one of my first programming projects, so I think we can forgive me for not knowing about box drawing symbols, or predicting Kitty's release :)

Wasn't a criticism, more of an FYI :)

[kovidgoyal]

Also can we move discussion of this into a separate issue, so as not to clutter this one.

[christianparpart]

I generally like the idea of improving input mapping, but I personally would strongly recommend not calling it "Game mode", because literally any client application could benefit from it.

Also, using a timeout as keepalive hint to automatically disable the alternate input mode I'd strongly discourage because it feels like that's something you can easily exploit to force an app moving out of that alternate mode (send SIGSTOP to the client app to force it not being able to send keep-alive for example, or just an accidentally blocking call that takes too long to return to the caller.)

If something like this is meant to get a broader adoption, I think it should be a discussion with all (more or less) major VTE devs on board, to come to a consensus that is more likely being implemented by more than one VTE. That would improve adoptability on the client side, too.

[kovidgoyal]

@christianparpart Trying to get all major VTE devs to agree on a keyboard reporting protocol was the issue that caused me to finally wash my hands of trying to get consensus. I wish you all the best. https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/1

[christianparpart]

haha. okay, I feel you. But many thanks for the link!

[kovidgoyal]

Closing in favor of the new keyboard reporting protocol in #3248 All other requests in this ticket are already implemented/available, with perhaps the exception of pixel based/accurate mouse cursor position reporting, for which, if needed open a separate issue.

[msokalski]

Great, new protocol looks promising!