use sgr pixelmode mouse reporting where supported

[dankamongmen]

We currently use SGR-style mouse support, 1006 DECSET, but where supported we ought use 1016. this reports in pixels rather than cells. this ought only be used if we've verified pixel info support. so long as we have, set y/x in ncinput by the reported y/cellpixy and x/cellpixx, and set ypx/xpx to y%cellpixy and x%cellpixx.

the only difficulty is knowing when 1016 is supported....

[ghost]

xterm will respond to DECRQM 1016 to let you know that SGR-Pixels will work, and is thus safe to switch to.

[dankamongmen]

sweet!

[ghost]

foot: https://codeberg.org/dnkl/foot/issues/762

[j4james]

Note that you can also get pixel mouse reporting with the DEC locator sequences (DECELR, DECSLE, etc.), identified by extension 29 in the DA1 report. Terminal emulators that are known to support this extension include XTerm, MLTerm, Mintty, RLogin, and VTStar, although they don't necessarily support all the locator sequences.

[dankamongmen]

@j4james , numerous comments of yours lead me to suspect that you have hidden away somewhere a great DEC-centric table of sequences and their support. hook a brother up?

[dankamongmen]

https://espterm.github.io/docs/Locator%20Input%20Model%20for%20ANSI%20Terminals%20(sixth%20revision).html seems to have good coverage of this.

[dankamongmen]

let's go ahead and look at knocking this out; i'm so so so so so sick of looking at debian autopkgtest failures

[dankamongmen]

here's what 1016 looks like on XTerm:

read_input_nblock:1889:read 14B from 0 (8178B left)                             
process_melange:2110:input 0 (0)/14 [0x1b] ( )                                  
process_escape:1926:initialized automaton to 1                                  
process_escape:1938:walk result on 91 ([): 0 4                                  
process_escape:1938:walk result on 60 (<): 0 261                                
process_escape:1938:walk result on 51 (3): 0 262                                
process_escape:1938:walk result on 53 (5): 0 262                                
process_escape:1938:walk result on 59 (;): 0 263                                
process_escape:1938:walk result on 52 (4): 0 264                                
process_escape:1938:walk result on 55 (7): 0 264                                
process_escape:1938:walk result on 52 (4): 0 264                                
process_escape:1938:walk result on 59 (;): 0 265                                
process_escape:1938:walk result on 49 (1): 0 266                                
process_escape:1938:walk result on 52 (4): 0 266                                
process_escape:1938:walk result on 49 (1): 0 266                                
mouse_click:541:dropping click in margins 138/471                               
process_escape:1938:walk result on 77 (M): 2 267                                
block_on_input:2194:blocking on input availability                              
block_on_input:2251:waiting on 1 fds (ibuf: 0/8192)                             
block_on_input:2278:poll returned 1                                             
block_on_input:2292:got events: tI                                              
read_input_nblock:1889:read 14B from 0 (8178B left)                             
process_melange:2110:input 0 (0)/14 [0x1b] ( )                                  
process_escape:1926:initialized automaton to 1                                  
process_escape:1938:walk result on 91 ([): 0 4                                  
process_escape:1938:walk result on 60 (<): 0 261                                
process_escape:1938:walk result on 51 (3): 0 262                                
process_escape:1938:walk result on 53 (5): 0 262                                
process_escape:1938:walk result on 59 (;): 0 263                                
process_escape:1938:walk result on 52 (4): 0 264                                
process_escape:1938:walk result on 53 (5): 0 264                                
process_escape:1938:walk result on 55 (7): 0 264                                
process_escape:1938:walk result on 59 (;): 0 265                                
process_escape:1938:walk result on 49 (1): 0 266                                
process_escape:1938:walk result on 52 (4): 0 266                                
process_escape:1938:walk result on 54 (6): 0 266                                
mouse_click:541:dropping click in margins 143/454                               
process_escape:1938:walk result on 77 (M): 2 267            

got some nasty crap from mlterm:

block_on_input:2278:poll returned 1
block_on_input:2292:got events: tI
read_input_nblock:1889:read 6B from 0 (8186B left)
process_melange:2110:input 0 (0)/6 [0x1b] ( )
process_escape:1926:initialized automaton to 1
process_escape:1938:walk result on 91 ([): 0 2
walk_automaton:570:unexpected transition on 2[77]
process_escape:1938:walk result on 77 (M): -1 0
process_melange:2110:input 1 (0)/5 [0x5b] ([)
process_melange:2110:input 2 (0)/4 [0x4d] (M)
process_melange:2110:input 3 (0)/3 [0x43] (C)
process_melange:2110:input 4 (0)/2 [0x58] (X)
process_melange:2110:input 5 (0)/1 [0x38] (8)
block_on_input:2194:blocking on input availability
block_on_input:2251:waiting on 1 fds (ibuf: 0/8192)
internal_get:2439:draining event readiness pipe 5
ncplane_erase:2385:erasing 6x56 plane

kitty with 1016 doesn't generate mouse events. wezterm looks like:

read_input_nblock:1889:read 6B from 0 (8186B left)
process_melange:2110:input 0 (0)/6 [0x1b] ( )
process_escape:1926:initialized automaton to 1
process_escape:1938:walk result on 91 ([): 0 4
walk_automaton:570:unexpected transition on 4[77]
process_escape:1938:walk result on 77 (M): -1 0
process_melange:2110:input 1 (0)/5 [0x5b] ([)
process_melange:2110:input 2 (0)/4 [0x4d] (M)
process_melange:2110:input 3 (0)/3 [0x43] (C)
process_melange:2110:input 4 (0)/2 [0x55] (U)
process_melange:2110:input 5 (0)/1 [0x29] ())
block_on_input:2194:blocking on input availability
block_on_input:2251:waiting on 1 fds (ibuf: 0/8192)
internal_get:2439:draining event readiness pipe 31
ncplane_erase:2385:erasing 6x56 plane
ncvisual_blit:1116:inblit 150x616 0@0 0x0 @ 0x0 0x611000000400
ncvisual_geom_inner:277:vis 0x0+0x0 0x7f29828f8800
ncvisual_geom_inner:370:pixel prescale: 150 616 150 616
ncvisual_geom_inner:385:pblit: 150x616 ← 0x0 of 150/616 stride 2496 @0x0 0x7f298
ncvisual_geom_inner:445:rgeom: 6 56 150 616 @ 0/0 (6 on 0x611000000400)
ncvisual_blit:1165:blit to plane 0x611000000400 at 13/10 geom 6x56
ncvisual_render_pixels:1027:pblit: rows/cols: 6x56 plane: 6/56
sprixel_hide:90:marking sprixel 12307538 hidden
block_on_input:2278:poll returned 1
block_on_input:2292:got events: tI
read_input_nblock:1889:read 6B from 0 (8186B left)
process_melange:2110:input 0 (0)/6 [0x1b] ( )
process_escape:1926:initialized automaton to 1
process_escape:1938:walk result on 91 ([): 0 4
walk_automaton:570:unexpected transition on 4[77]
process_escape:1938:walk result on 77 (M): -1 0
process_melange:2110:input 1 (0)/5 [0x5b] ([)
process_melange:2110:input 2 (0)/4 [0x4d] (M)
process_melange:2110:input 3 (0)/3 [0x43] (C)
process_melange:2110:input 4 (0)/2 [0x55] (U)
process_melange:2110:input 5 (0)/1 [0x2a] (*)
block_on_input:2194:blocking on input availability
block_on_input:2251:waiting on 1 fds (ibuf: 0/8192)
ncplane_resize_internal:840:6x56 @ 13/10 → 6/56 @ 13/10 (want 6x56@0/0)
notcurses_rasterize_inner:1274:pile 0x6080000000a0 ymax: 22 xmax: 78
notcurses_rasterize_inner:1284:sprixel phase 1
clean_sprixels:892:phase 1 sprixel 12307539 state 1 loc 13/10
sprite_redraw:730:sprixel 12307539 state 1
kitty_draw:1180:dumping 493929b for 12307539 at 2 2

[dankamongmen]

so xterm's responses are just a replacement of cells with pixels, though we do seem to report something a bit weird:

process_escape:1938:walk result on 52 (4): 0 264                                
process_escape:1938:walk result on 55 (7): 0 264                                
process_escape:1938:walk result on 52 (4): 0 264                                
process_escape:1938:walk result on 59 (;): 0 265                                
process_escape:1938:walk result on 49 (1): 0 266                                
process_escape:1938:walk result on 52 (4): 0 266                                
process_escape:1938:walk result on 49 (1): 0 266                                
mouse_click:541:dropping click in margins 138/471     

how did 141/474 become 138/471?

[dankamongmen]

how did 141/474 become 138/471?

margins, duh

[dankamongmen]

confirmed that XTerm and MLterm report 29 in DA1. so what's this weird form that MLTerm is returning to us?

[dankamongmen]

those are only coming in on clicks btw, not on mouse moves. doesn't look like we're getting anything for mouse moves in MLterm.

[dankamongmen]

hrmmm looks like we're never getting motion reports from mlterm, even with 1006

[dankamongmen]

hrmmm looks like we're never getting motion reports from mlterm, even with 1006

wezterm confirmed to provide motion reports under 1006 just fine

[dankamongmen]

so:

• xterm acts as expected for both 1006 and 1016
• mlterm doesn't seem to report move events for either. it returns unexpected events for 1016.
• wezterm acts as expected for 1006. wezterm returns unexpected events for 1016, which look superficially similar to those returned by mlterm. it does generate move events for 1016. Request: mouse mode 1016 (SGR-Pixels) wez/wezterm#1457
• kitty acts as expected for 1006, and returns nothing on 1016.
• contour reports 29 (Feature request: mouse mode 1016 (SGR-Pixels) contour-terminal/contour#574)

[dankamongmen]

wait, xterm does not seem to advertise 29 in DA1, unlike mlterm and contour

[christianparpart]

wait, xterm does not seem to advertise 29 in DA1, unlike mlterm and contour

I will be fixing this. Sorry.

[j4james]

wait, xterm does not seem to advertise 29 in DA1, unlike mlterm and contour

Just to be clear, extension 29 indicates support for the DEC locator sequences - not XTerm's proprietary 1016 mouse mode. If you want to test for 1016 support you should probably use DECRQM.

And if you're not getting a 29 report from XTerm, it's possible you don't have the locator support enabled. It needs to be be built with the --enable-dec-locator option. I'm not sure if that's typically enabled by default, since I always build XTerm myself.

mlterm doesn't seem to report move events for either.

MLTerm does support move events via the DECEFR sequence. You can verify this functionality in Vttest with test 11.8.5.8.6: Filter Rectangle (unfiltered).

[dankamongmen]

yep, DECRQM seems the right way to be doing this. i'll send it along with the other queries. thanks!

[dankamongmen]

wait, xterm does not seem to advertise 29 in DA1, unlike mlterm and contour
I will be fixing this. Sorry.

see @j4james 's comments above, @christianparpart. they don't map precisely. my error.

[dankamongmen]

using DECRQM 1016 (\e[?1016$p), we get support from XTerm, and no support from MLterm, meaning we can go ahead and start using this, i think!

mlterm: ^[[?1016;0$y
XTerm: ^[[?1016;2$

[dankamongmen]

DECRPM 1016 comes back as 2 from XTerm and Wezterm. it does not come back from MLterm nor kitty nor contour.

@klamonte is apparently changing the wezterm protocol up to match XTerm, at least on 1016. so this is exactly what we want.

[dankamongmen]

wait, wezterm seems to be working just fine?

[christianparpart]

DECRPM 1016 comes back as 2 from XTerm and Wezterm. it does not come back from MLterm nor kitty nor contour.

@klamonte is apparently changing the wezterm protocol up to match XTerm, at least on 1016. so this is exactly what we want.

I just implemented this In Contour and merged to master. I simply didn't know about that feature. It would be nice if there is a notcurses mini paint demo program for spending this (using Sixel + SGR-pixels / DEC locator) :-)

[ghost]

A mini paint application is precisely what I had in mind when I put in SGR-Pixels. 🙂

Actually what would be super handy for me is an image cropper: load image, drag rectangle over portion, crop it, save (as original format, not dithered). I am kinda tired of firing up gimp for something so small; being able to do it over ssh without X11 would be really nice.

[christianparpart]

Generally speaking.... Plus a simple image browser and an ls that can preview (image) files in the listings.