XTSMGRAPHICS "Read sixel graphics geometry" returns the different value from xterm.

[kumattau]

Contour Terminal version

0.3.1-unreleased-master-7f13fbb0

Installer source

Github: source code cloned

Operating System

Ubuntu Linux 20.04

Architecture

x86-64

Other Software

No response

Steps to reproduce

cat <<EOF > test.sh
IFS=';' read -a REPLY -s -t 0.25 -d "S" -p $'\e[?2;1;0S'
echo "width=\${REPLY[2]} height=\${REPLY[3]}"
EOF
bash test.sh

Expected Behavior

Returned width and height are smaller than or equal to screen width and height (same as the latest xterm).

Actual Behavior

Returned width and height are always 1920 and 1080 regardless of screen size.

Additional notes

By this problem, https://github.com/hackerb9/lsix can not layout pictures as expected position.

contour

xterm

ref: xterm implementation:

https://github.com/ThomasDickey/xterm-snapshots/blob/834b501a1416d7f14b1c38705083f413aea92eee/charproc.c#L4625

[uspasojevic96]

@christianparpart doesnt this just return whatever is defined in the contour.yml?

EDIT: yes it does

contour/src/contour/TerminalSession.cpp

Line 964 in 88b4d96

terminal_.setMaxImageSize(config_.maxImageSize);

[christianparpart]

Good morning. I'm not sure about how a real VT340 would respond to this, but due to the nature of the fact, that a VT340 is fairly limited to what a modern VTE is capable of, I can't judge for either side. (EDIT: It would not respond at all, as XTSMGRAPHICS is an extension from xterm).

But to my understanding, XTSMGRAPHICS is used to query the limits, that's not necessarily the same as how big a screen (terminal viewport in pixels) is. For that there is another VT sequence, CSI 14 t (XTWINOPS), as introduced by dterm, and extended by xterm (just losely quoting here).

@hackerb9, what's your stance on this? Objectively looking at it, I'd rather change lsix to use XTWINOPS on top of XTSMGRAPHICS to query the screen width as the minimum of both values.

[christianparpart]

https://github.com/ThomasDickey/xterm-snapshots/blob/834b501a1416d7f14b1c38705083f413aea92eee/charproc.c#L4622-L4626

The problem I'm (personally) having with this code, is, that screen size may be changed due to the user, using CSI 14 t (IMHO) clearly tells that you are querying viewport width/height in pixels (which is subject to change), whereas CSI [ ? 2 ; 1 ; 0 S (by looking at the docs) reads the sixel canvas size in pixels attribute, not stating its relation to the actual window's size in pixels.

[j4james]

A couple of points worth mentioning:

1. XTSMGRAPHICS is an XTerm invention - a real VT340 would not have responded to that.
2. XTerm's initial implementation of the 2;1 query returned the maximum geometry size, the same as you're doing now (see here).
3. A few other terminals copied that initial implementation, and while some have been updated to match XTerm's current behavior, I think some still report the maximum size.
4. CSI 14 t is also not a VT340 standard, and has its own problems, but it's more widely implemented, and more reliable in general.

So personally, I'd recommend apps just rely on CSI 14 t if they need to determine the screen size (taking into account that some rounding may be required). But if you also want to use XTSMGRAPHICS as a backup, only do so if it's smaller than the dimensions returned by CSI 14 t - that should give you the most accurate results.

[christianparpart]

Sounds absolutely reasonable. Thanks @j4james ;)

[kumattau]

This is just my personal opinion,

XTSMGRAPHICS is an XTerm invention, so other terminals should follow XTerm's current spec.

https://github.com/ThomasDickey/xterm-snapshots/blob/834b501a1416d7f14b1c38705083f413aea92eee/ctlseqs.txt#L569-L622

As the following explanation,

• Pa = 1 should return minimum of the window size (maybe text (screen) area size) and the maximum graphic size.
• Pa = 4 should return the maximum graphic size.

Additionally, XTWINOPS (CSI 14 t) should return text area size.

       Pa = 1  -> read attribute.
       Pa = 2  -> reset to default.
       Pa = 3  -> set to value in Pv.
       Pa = 4  -> read the maximum allowed value.

      Notes:
      o   The current implementation allows reading the graphics
          sizes, but disallows modifying those sizes because that is
          done once, using resource-values.
      o   Graphics geometry is not necessarily the same as "window
          size" (see the dtterm window manipulation extensions).
          XTerm limits the maximum graphics geometry according to
          the maxGraphicSize resource.
          The maxGraphicSize resource can be either an explicit
          heightxwidth (default: 1000x1000 as of version 328) or the
          word "auto" (telling XTerm to use limits the decGraphicsID
          or decTerminalID resource to determine the limits).
      o   XTerm uses the minimum of the window size and the graphic
          size to obtain the maximum geometry.
      o   While resizing a window will always change the current
          graphics geometry, the reverse is not true.  Setting
          graphics geometry does not affect the window size.
      o   If xterm is able to support graphics (compile-time), but
          is not configured (runtime) for graphics, these responses
          will indicate a failure.  Other implementations which do
          not use the maximum graphics dimensions but are configured
          for graphics should report zeroes for the maximum geometry
          rather than a failure.

So personally, I'd recommend apps just rely on CSI 14 t if they need to determine the screen size (taking into account that some rounding may be required). But if you also want to use XTSMGRAPHICS as a backup, only do so if it's smaller than the dimensions returned by CSI 14 t - that should give you the most accurate results.

The Applications just only are forced to work around for supporting various terminals, but the terminals should not expect the application's work-around effort and should keep the compatibility for the standard as possible.

[christianparpart]

The Applications just only are forced to work around for supporting various terminals, but the terminals should not expect the application's work-around effort and should keep the compatibility for the standard as possible.

I'm currently (today) busy at work, I can respond later tonight more detailed. But I absolutely agree with you here that apps should not suffice from implementation semantic differences.

[jerch]

My 2cents on XTSMGRAPHICS:

The sequence feels weirdly xterm implementation dependent to me, it mostly reflects restrictions/settings on xterm, that other terminals might not have in the same way. In general I think terminal sequences should stay in the general and portable interface realms, and not surface implementation details at all costs, unless the sequence is marked for private usage only. XTSMGRAPHICS fails in that part imho.

Now with xterm things are more tricky - it is treated by many app devs and even terminal devs as the de-facto terminal interface standard these days, which makes it hard to ignore any sequence, xterm comes up with. Thats the reason, why I implemented XTSMGRAPHICS as well, but only parts of it, that could be answered reasonably:

• ReGIS - all nope, answered with GaStatus.ITEM_ERROR
• colors registers - READ | SET_DEFAULT | SET | READ_MAX supported (up to 4096)
• sixel geometry - READ | READ_MAX supported, all others will fail

For sixel geometry READ it returns the current viewport size in pixels, for READ_MAX I had to cheat and return the square* of the pixel limit of the sixel decoder (default 4096x4096). The last thing is somewhat unlucky, as it is not really limited by that (an image can happily run up to 32k pixel width, if height * width stays within pixelLimit).

The impl can be found here: https://github.com/jerch/xterm-addon-image/blob/75deeccd20172e753cf8b38846140462d05b9467/src/ImageAddon.ts#L259

---

Edit: [*] square root 😸

[j4james]

Just to be clear, I agree with everyone that is saying you should match XTerm's current behavior if you're going to implement this sequence. But I'd still recommend applications avoid using it if possible.

The sequence feels weirdly xterm implementation dependent to me, it mostly reflects restrictions/settings on xterm, that other terminals might not have in the same way. In general I think terminal sequences should stay in the general and portable interface realms, and not surface implementation details at all costs, unless the sequence is marked for private usage only. XTSMGRAPHICS fails in that part imho.

I fully agree with this. That's why I've just chosen not to implement it. And if I come across an application or library that depends on it, I've encouraged them to avoid that dependency, and support a reasonable fallback.

[kumattau]

Yes, the main problem is that, as the title says, XTSMGRAPHICS "Read sixel graphics geometry" returns the different value from xterm.
It makes application confused.
The terminal can ignore unknown CSI (by not implementing CSI) and XTSMGRAPHICS defines error.
So, if implementing XTSMGRAPHICS is difficult for the terminal, it should ignore or return error rather than return unexpected value.

I fully agree with this. That's why I've just chosen not to implement it.

I agree with your way.
I think that this is the right way to handle this if the terminal can not support this CSI for some reasons.

I'd still recommend applications avoid using it if possible.

I agree with you about implementing applications.

[christianparpart]

i can change that to match xterm (as much as I dislike that :D). But regardless of that, we could still kindly ask @hackerb9 to change the query order, first test CSI 14 t, and then go for XTSMGRAPHICS if the first fails, or use the minimum value of both (if both return something). Does this sound plausible?

[hackerb9]

i can change that to match xterm (as much as I dislike that :D). But regardless of that, we could still kindly ask @hackerb9 to change the query order, first test CSI 14 t, and then go for XTSMGRAPHICS if the first fails, or use the minimum value of both (if both return something). Does this sound plausible?

Hi! Thanks for the kind request. Possible, but not plausible given my current understanding of the situation.

First, some ancient history: If there are deficiencies in XTSMGRAPHICS being used for sixel geometry, I'm the one to blame, not XTerm nor XTerm's maintainer, Mr. Dickey. While that extension existed before for changing the pixel depth, I'm the one who proposed using it to also query the sixel geometry and worked with Mr. Dickey to get it working properly in XTerm. The problem at that time, and that still exists with dtterm window ops, like CSI14t, is that window size (or geometry in the old lingo) has been overloaded to mean different things. IIRC, the dtterm escape sequences were meant to be able to position the terminal on the screen, so the "size" originally included the blank border around the text area and even the width of the titlebar and other user interface widgets.

While it is a happy accident if CSI14t returns the correct sixel geometry, it is not incorrect for a terminal to do otherwise. Even excluding UI widgets, the dtterm window size may not match the renderable sixel area: a terminal might, for example, choose to always use an 800x480 sixel geometry, no matter what the window size is. On the other hand, if a terminal has implemented XTSMGRAPHICS in a way that it returns something other than the sixel screen size, then it is in error. (And, yes, XTerm itself was in error originally, but that has been fixed since 2019.)

So, I think using dtterm's size for sixel geometry is a mistake. But, as always, I'm fallible: Please let me know if I got something wrong.

[j4james]

I hate to be the one defending the CSI t sequences, because I really don't like them either, but I've tested quite a lot of sixel terminals, and the simple truth is that you get better results using CSI t than you do using XTSMGRAPHICS (at least in my experience). In time that may change, but for now if you want your app to work on as many terminals as possible, I'd recommend making the most of that happy accident.

[jerch]

Indeed, CSI t is quite convoluted, and even was subject of several security concerns up to CVEs. The WindowOps sequences kinda lost their primary purpose to tune certain desktop integration aspects long ago (mind you, many things simply dont work anymore, as they have no counterpart on modern DEs).

So the shift of meaning is a logical consequence as in any evolutionary system (we are just that atm, as we lack a higher authority giving us red/green lines regarding the terminal interface). The problem we all have with this (at least I find it painful) - there is no strict right or wrong in such a scenario, just a more likely/unlikely, which really sucks in terms of strict correctness (computers still need that, unlikely to change until we have QPU-NPUs).

Regarding XTSMGRAPHICS I think you should not blame yourself too hard (@hackerb9), while I dont like it for leaking xterm implementation details, it certainly has several pros:

• separation of concerns, graphical stuff is a very different and delicate thing for text-driven terminals
• no convolution of other sequences / overloaded meanings (as with CSI t, which happens to work more by accident)
• extensible interface (easy to put other protocols or graphical options there, once needed)
• reporting caps (just a minor nag from my side - would have made SET options non-reporting by default, and hide reporting behind a sequence flag)

So I try to see things more opportunistic here - since XTSMGRAPHICS is already there, and ppl are likely to use it (its still xterm, we cannot ignore that until some other TE dethrones it) - lets use it to our advantage (thats the reason why I proposed in another thread to put a "reload default palette" option there).

Edit: Ah well, I think I will just do that and load the default palette on SET_DEFAULT (technically not quite correct, as it is only meant to change the register numbers, but my impl currently lacks a single sequence for reloading color values only).

[kumattau]

@christianparpart

first test CSI 14 t, and then go for XTSMGRAPHICS if the first fails,

Sorry if it's wrong. I think,

Sixel does not have width and height explicitly.

If the application uses only CSI 14 t, when the application run on very large window, that is over the supported sixel geometry by the terminal, the application requests the drawing large sixel over the supported dimension.

In the case, the result of drawing is undefined (depending on the terminal), but application cannot know the case by the CSI 14 t.

XTSMGRAPHICS is required for the application to know the supported sixel dimension by the terminal in advance.

So, the application's work-around is the following order:

• minimum of XTSMGRAPHICS and CSI 14 t (for old xterm and old contour)
• XTSMGRAPHICS (when CSI 14 t is not supported)
• CSI 14 t (when XTSMGRAPHICS is not supported)
• heuristic fixed value by the application

[j4james]

If the application uses only CSI 14 t, when the application run on very large window, that is over the supported sixel geometry by the terminal, the application requests the drawing large sixel over the supported dimension.

This is a problem that I think is exclusive to XTerm. And one of the effects of this limitation is that XTSMGRAPHICS can't actually be relied on to report the real screen size (in XTerm at least). For an app like lsix, that is probably fine - assuming all you care about is the maximum size that can be used. But most complex sixel applications are more interested in the pixels-per-cell, i.e. you want to calculate how big an image needs to be to cover a certain range of character cells. That's not something you can reliably determine with XTSMGRAPHICS.

• minimum of XTSMGRAPHICS and CSI 14 t (for old xterm and old contour)

I believe Alacritty is another terminal that still reports the max geometry for \e[?2;1;0S.

• XTSMGRAPHICS (when CSI 14 t is not supported)

I've never encountered a sixel implementation that supported XTSMGRAPHICS but not CSI 14 t.

My recommendation, if you need the cell size, would be CSI 16 t, and if that's not available, CSI 14 t divided by CSI 18 t (rounding down if necessary). If you want the screen size, calculate the cell size as described above, and multiply by CSI 18 t. I personally wouldn't bother with XTSMGRAPHICS unless you care about XTerm's geometry limitations in particular.

[kumattau]

@j4james

Thank you for very interesting comment.

And one of the effects of this limitation is that XTSMGRAPHICS can't actually be relied on to report the real screen size (in XTerm at least).

I agree with you too.
But it is because XTSMGRAPHICS does not report screen size.
I think the application that uses screen size and does not uses graphics size should not XTSMGRAPHICS.

I believe Alacritty is another terminal that still reports the max geometry for \e[?2;1;0S.

I updated to minimum of screen size and graphics size:
microo8/alacritty-sixel@83a40a6

My recommendation, if you need the cell size, would be CSI 16 t, and if that's not available, CSI 14 t divided by CSI 18 t (rounding down if necessary). If you want the screen size, calculate the cell size as described above, and multiply by CSI 18 t.

Thank you for actual and useful recommendation.
I think when CSI 16 t is available, screen size also is able to calculated with ioctl(TIOCGWINSZ) ws_row and ws_col.

I personally wouldn't bother with XTSMGRAPHICS unless you care about XTerm's geometry limitations in particular.

My point is that the result of drawing sixel over the supported dimension by the terminal is undefined (depending on the terminal).
I think XTSMGRAPHICS is needed to avoid the undefined behavior, not to determine the exact sixel size.
I think every terminal has some kind of limitation same as xterm because resources are finite.

When XTerm's spec is respected, should the application like lsix do the following ?

1 get maximum image size by XTSMGRAPHIC Pa=4.
2 get current screen size by CSI 14/16/18, ioctl(TIOCGWINSZ) ws_row and ws_col, and the combination.
3 get the minimum of "maxiumum image size" and "current screen size".

[hackerb9]

@kumattau writes:

When XTerm's spec is respected, should the application like lsix do the following ?

1. get maximum image size by XTSMGRAPHIC Pa=4.
2. get current screen size by CSI 14/16/18, ioctl(TIOCGWINSZ) ws_row and ws_col, and the combination.
3. get the minimum of "maxiumum image size" and "current screen size".

Good question. When that happens — which shouldn't be long now that you've patched Alacritty and Contour is already fixed — the answer is pretty simple:

1. Get sixel graphics geometry by XTSMGRAPHICS(Pᵢ=2, Pₐ=1)

That's it.

The purpose of XTSMGRAPHICS with Pₐ = 1 is to provide an explicit way to read the sixel height and width.

[hackerb9]

I've been wondering if this bug could have been averted if I had added the following note in the description of XTSMGRAPHICS in XTerm's ctlseqs:

The "graphics geometry" attribute is meant for applications to read the size of the visible screen that is available for use. It is never larger than the terminal window, but may be smaller depending upon the terminal implementation.

Would that have helped? If so, I'll see about getting it added to ctlseqs.ms.

[christianparpart]

@hackerb9 to be hoonest? yes. in my case at least. because I looked at ctlseqs.txt and kind of trusted in what it said, given the completeness of that document and the reputation it had, at least the way I perceived it, I did not feel in need to verify every documentation line with its implementation in the xterm source code.

Regardless of that, improving documentation is always welcome. :)

[hackerb9]

@christianparpart writes:

to be honest? yes. in my case at least.

Thank you. I'm glad I found something that might help others in the future. Before I contact Mr. Dickey about fixing it, was what I wrote sufficient? Could it be clearer? More succinct?

I looked at ctlseqs.txt and kind of trusted in what it said, given the completeness of that document and the reputation it had, at least the way I perceived it,

And, this is where my mea culpa becomes mea maxima culpa. I wrote the relevant part of the documentation and Mr. Dickey implemented it in XTerm. It should have been a big hint to me that there was a problem in my spec when he was initially wrote the code so that XTerm's maximum was returned for both Pₐ = 1 and Pₐ = 4.

If you wish to know my confusion, read on, otherwise please simply accept my apologies.

The problem was that I was writing as an application developer. For applications, knowing the usable screen size is critical and an obvious deficit in DEC's sixel specification. I didn't see any point in reading the maximum geometry value — what would that even mean for an unsettable attribute? Only today did it dawn on me that, for terminal developers, the maximum must be a much more salient number and clarification is needed.

I did not feel in need to verify every documentation line with its implementation in the xterm source code.

Nor should you have to. The irony is that if you had checked XTerm's source code before 2019, it would have been wrong. Even more ironic: I had been thinking that the error in the XTerm source code wasn't so important since the documentation was clear. Boy, that'll learn me! 😖

Again, please accept my apologies for the confusion.

[j4james]

Note that the documentation does actually point out that graphics geometry is not necessarily the same as the window size. It's in the "notes" section below the description of the sequence. But I don't think it helps having that information so far removed from where the graphics sequence is described.

Anyway, I'm still of the opinion that this is a bug in XTerm that should be fixed.

[kumattau]

Anyway, I'm still of the opinion that this is a bug in XTerm that should be fixed.

I think it is no good to change XTSMGRAPHICS Pa=1 spec because there are terminals which implement XTerm's spec expectedly.

But I am interesting the bug-fixed spec details which @j4james thinks.
If the spec is reasonable and useful for the application on the terminals, the new XTSMGRAPHICS Pa (Pa=5 ?) can be proposed, I think.

[j4james]

Sorry, I wasn't very clear. I didn't mean the XTSMGRAPHICS sequence was a bug. I meant the fact that XTerm can't display certain image sizes is a bug.

Imagine any other graphics library that had a BlitImage operation that would just fail to work on certain parts of the screen. But you're told that's not a bug - you just need to call the IsBlitImageGoingToWork function first. And then somehow redesign your app on the fly to work around those limitations. Also, for compatibility, all other graphics libraries are encouraged to add an IsBlitImageGoingToWork function, even though they all just returns true, because nobody else has this ridiculous bug.

This is not like a function failing under exceptional conditions, like you've run out of memory, or you were trying to load an image 10 times bigger than the screen itself. This is just an image that is size of the terminal window - there's no reason that shouldn't work under normal conditions.

[jerch]

I second @j4james here, imho xterm's restriction, that the graphical canvas geometry might be smaller than the visible text area, is a no-go these days. (Though I have an idea, where this comes from, the sixel painting is awfully expensive in xterm, esp. with background_select=1, so this might be an optimization to save some drawing time and memory).

[kumattau]

Sorry, I'm not good at English.

I didn't mean the XTSMGRAPHICS sequence was a bug. I meant the fact that XTerm can't display certain image sizes is a bug.

I understood that you say XTSMGRAPHICS spec does not have problem but XTerm implementation has problem.

I think xterm implementation is off topic in this issue ticket, but I share my test result of displaying 5000 px width sixel on some terminals.

• XTerm: no image output
• alacritty-sixel: no image output with error message
• contour: 1920px image truncated
• mlterm: 5000px image output
• wezterm: 5000px image output

I think the thing that some terminals limit image size is justified by the following reasons:

• terminal is not graphic library for general purpose
• outputting image is not much important for character terminal
• outputting image is expensive in terms of memory and processing time for character terminal

I think that xterm's 1000px limit is too small and it is better to relax the limit, but I don't think xterm should not have the limit.

[jerch]

I think the thing that some terminals limit image size is justified by the following reasons:

* terminal is not graphic library for general purpose

* outputting image is not much important for character terminal

* outputting image is expensive in terms of memory and processing time for character terminal

I think that xterm's 1000px limit is too small and it is better to relax the limit, but I don't think xterm should not have the limit.

The points you bring up here, account more towards no graphics support at all - which is perfectly fine for a TE, if its environment has strict resource limits (e.g. embedded stuff, kernel console). Then make it optional, e.g. by a compile switch. But if the switch is on, the TE should imho not restrict the graphics area to a smaller size than the allowed viewport. Machines as of today have enough resources to handle that fairly well, this artificial restriction is just an annoyance to create frictions on app dev and terminal user side for no good reasons anymore.

So either support full width, or disable graphics output at all. Thats my stance here. SIXEL itself allows up to 32k width - which is very easy to handle, if you do processing in sixel bands, not the whole thing at once (only 6 * 32k * 4 bytes in RGBA8888). And thats from a +30ys old semi-spec - and we still have some decades to go, until we hit that screen resolution area (some cinema projectors operate up to 64k already, but that's not the first audience for terminal output rendering...).

[kumattau]

But if the switch is on, the TE should imho not restrict the graphics area to a smaller size than the allowed viewport. Machines as of today have enough resources to handle that fairly well, this artificial restriction is just an annoyance to create frictions on app dev and terminal user side for no good reasons anymore.

contour limits image size to the primary screen size.
It causes smaller image size than terminal screen size if contour extends over the dual display.

I think that we should have @jerch issue a new ticket to discuss it because it does not relate to XTSMGRAPHICS.
What do you think @christianparpart ?

[christianparpart]

I am not kicking anyone out, not at all. @kumattau, you have a point to the actual creation of this ticket though.

My suggestion: Just create a discussion in the discussion board in this project, if you don't feel too alienated by the fact that this is a Contour TE repo, I am very willing to offer that space for general TE discussions (any type). :-)

[kumattau]

@christianparpart @j4james @jerch

I am sorry if I have offended you.
I am just worry that the ticket can not be closed due to a discussion not directly related to this ticket.

For now, I created the following discussion about image size supported by the terminal: #680
Let's discuss it on there if more necessary.

[christianparpart]

All good @kumattau! I'm all in for productive constructive conversations. We should just try to not run in cycles. (common problem sadly). Moving over to #680.