Should the terminal that can display images support large image equal to screen size ?

[kumattau]

transfer from: #656 (comment)

I think the thing that some terminals limit image size is justified by the following reasons:

• terminal is not graphic library for general purpose

• outputting image is not much important for character terminal

• outputting image is expensive in terms of memory and processing time for character terminal
  I think that xterm's 1000px limit is too small and it is better to relax the limit, but I don't think xterm should not have the limit.

The points you bring up here, account more towards no graphics support at all - which is perfectly fine for a TE, if its environment has strict resource limits (e.g. embedded stuff, kernel console). Then make it optional, e.g. by a compile switch. But if the switch is on, the TE should imho not restrict the graphics area to a smaller size than the allowed viewport. Machines as of today have enough resources to handle that fairly well, this artificial restriction is just an annoyance to create frictions on app dev and terminal user side for no good reasons anymore.

So either support full width, or disable graphics output at all. Thats my stance here. SIXEL itself allows up to 32k width - which is very easy to handle, if you do processing in sixel bands, not the whole thing at once (only 6 * 32k * 4 bytes in RGBA8888). And thats from a +30ys old semi-spec - and we still have some decades to go, until we hit that screen resolution area (some cinema projectors operate up to 64k already, but that's not the first audience for terminal output rendering...).

[christianparpart]

I did change the code in Contour to match the implementation of xterm. However, while I am all in for convergence here, I wanted to at least make my point of view / stance a little bit more obvious:

• I considered the xterm implementation of XTSMGRAPHICS to be buggy, as it wasn't following the docs. And I still hold on that point because of the existence of CSI 14 t, that can be easily used to retrieve the actual viewport in pixels. I also just learned that xterm is doing that min(a, b)-thingy only due to internal technological limitations. Other TEs should not suffer just because of a not so optimal implementation on the xterm side. (I know, "fix xterm" is easier said than done, if ever)
• having a split interpretation of the dimension of the viewport in pixels and the dimension of the drawing area did make sense to me so far, however:
• I did not give up on the "Good Image Protocol" movement we're having, I just need some time to finish polishing existing areas before opening up another one (code wise), plus some personal health-circumstances that did bite me, so:
• I think in todays world of all TEs wishing to implement images (let it be sixel or not), I believe that a TE that wishes to support images should always allow an image size of at least as big as the current viewport size. I do not even see 4k monitors as an issue here (having one).

So my question is, while we are at it, @hackerb9, are we simply going to change the ctlseqs.txt to match the limitations of xterm, are we going to fix xterm's internal issues in order to get rid of that min()-call? (highly unlikely), or are we talking about future ideas to better define such constraints and requirements for a future protocol. - or what are you thoughts & plans on this?

[hackerb9]

• I considered the xterm implementation of XTSMGRAPHICS to be buggy, as it wasn't following the docs. And I still hold on that point because of the existence of CSI 14 t, that can be easily used to retrieve the actual viewport in pixels. I also just learned that xterm is doing that min(a, b)-thingy only due to internal technological limitations. Other TEs should not suffer just because of a not so optimal implementation on the xterm side. (I know, "fix xterm" is easier said than done, if ever)

There was a brief period of time in which xterm's implementation of XTSMGRAPHICS was not what I had proposed. That was fixed years ago.

So my question is, while we are at it, @hackerb9, are we simply going to change the ctlseqs.txt to match the limitations of xterm,

I think I get what you're trying to say, but I have to point out that ctlseqs is literally the documentation for how XTerm works, so of course it is going to match XTerm. However, the more important point is that people seem to think XTSMGRAPHICS is about XTerm's particular implementation details. It is not. Yes, XTSMGRAPHICS helps to alleviate XTerm's annoying limitation on the image size, but that is not its purpose.

are we going to fix xterm's internal issues in order to get rid of that min()-call? (highly unlikely),

Sorry, what min() call? If an application wants to know the size of the actual screen in literal pixels, for example to align other windows around a terminal or overlay a 3-D view, then DECTerm's window ops (CSI 14 t) is the correct answer. If an app wants to know the canvas addressable by sixel (as opposed to what a window manager or GIP would see), then it'd use XTSMGRAPHICS. No min call needed in Contour. No min call needed for client developers, either.

or are we talking about future ideas to better define such constraints and requirements for a future protocol. - or what are you thoughts & plans on this?

I am not expert enough to define a future protocol, other than to say, I like that sixels are embedded in the text and scroll properly when used from the command line. I wouldn't trade them for anything that doesn't do that.

While sixel is clearly not the future, I think that it is not a bad idea to fill in the obvious holes in sixel's definition, like querying the geometry and number of color registers. XTSMGRAPHICS exists because DEC failed to define a way to do that. Back then, the user had to manually configure the driver depending upon which output device they wanted sixel code generated for. That made sense in a time when sixels were mainly a printer language sent over a unidirectional cable.

DEC implemented the WindowOps such as CSI 14 t for DECTerm (and probably VWS before that) as a method of window management over a telnet connection, not for drawing graphics. In fact, if you tried to draw ReGIS graphics in a DECTerm, by default it presented a virtual canvas size of 800x600, not the result of CSI 14 t. This is similar to @j4james's sixel terminal in which the graphics geometry is not dependent upon pixels but upon the number of rows and columns of text. XTSMGRAPHICS abstracts away the drawable size from physical screen pixels and solves this problem. (By the way, if a terminal developer wanted to use it, XTSMGRAPHICS already handles returning a different graphics size for ReGIS than for sixel.)

[j4james]

Ultimately it's up to the terminal author to decide what they want to support, and if their users don't like that, they can always choose to use another product. But personally, if I'm evaluating a terminal, and it can't display an image that is the full size of the screen, I would consider that a bug.

That said, a bug like that can have different levels of severity. For example, a terminal that can't cope with a 12K image stretched across three high definition monitors is less of concern to me than one that can't handle anything larger than 1024 pixels.

But I'm not arguing for any spec changes, or for any terminals to change their behavior if they don't want to. I just choose not to support the XTSMGRAPHICS sequence myself, and I don't depend on it in any sixel applications I write. If others find it useful, though, that's fine.

[kumattau]

My opinion is:

I think the document/specificaiton/agreement including "the TE should imho not restrict the graphics area to a smaller size than the allowed viewport." is not exist, so there is no problem that xterm and some terminals has the limit of image size.

[j4james]

The best reference I could find was in the DEC STD 070-9 manual describing the sixel graphics extension. See Chapter 11: ANSI TEXT INTERACTIONS.

11.2.2 Right Margin

Sixels defined to be printed past the right margin are not printed.

While it doesn't explicitly say so, that kind of implies that sixels are expected to print up to the right margin, and not just stop at a random point in the buffer.

Still, I don't particular care if you want to claim it's not a bug. Terminals frequently ignore the standards anyway. I'm just surprised that anyone would defend this as reasonable behavior.

[kumattau]

@j4james

Thank you for the information.

11.2.2 Right Margin
Sixels defined to be printed past the right margin are not print

I think above is weak evidence of the claim but the claim may be justified if there are the following reasons.

• The terminals which is older than xterm and support sixel supported full-width sixel.
• Additionally, xterm needs XTSMGRAPHICS as extension because xterm does not support full-width sixel unlike the other terminals.

If they are true, I agree you with "the TE should imho not restrict the graphics area to a smaller size than the allowed viewport." and xterm restriction is unreasonable.

And I 100% agree you with ignoring (not supporting) XTSMGRAPHICS if it is unreasonable.

[jerch]

While doing a small sixel image puzzle app (https://github.com/jerch/sixel-puzzle) I stumbled over contour limiting the usable sixel geometry to 800x600 (as reported by XTSMGRAPHICS, using latest official release build). So I wonder if a hard limit still is a thing in contour (also did not find a way around or draw beyond it)?

(While I really think that sixel is a dead end in the long run, I am somewhat surprised how well it works across the TE board.)

[uspasojevic96]

Currently it's hard capped through the config file, however that can be changed:

images:
    # Enable or disable sixel scrolling (SM/RM ?80 default)
    sixel_scrolling: true
    # Configures the maximum number of color registers available when rendering Sixel graphics.
    sixel_register_count: 4096
    # If enabled, the ANSI text cursor is placed at the position of the sixel graphics cursor after
    # image rendering, otherwise (if disabled) the cursor is placed underneath the image.
    sixel_cursor_conformance: true
    # maximum width in pixels of an image to be accepted (0 defaults to system screen pixel width)
    max_width: 0
    # maximum height in pixels of an image to be accepted (0 defaults to system screen pixel height)
    max_height: 0

[christianparpart]

(While I really think that sixel is a dead end in the long run, I am somewhat surprised how well it works across the TE board.)

I can only secon that @jerch ; and I promise I want and will head back to my GIP work as soon as private life permits. Until then, please ping me if I can improve on anything wrt. Sixel or image support in general in Contour.

p.s.: that's a really nice project idea, that image puzzle on a terminal ;)

[jerch]

@christianparpart A plz dont feel pressed by any means, atleast not by my "hatelove" for sixels. I already settled with the fact, that sixels are wide-spread (the sixel genie is out of the bottle), and libs already using it for higher res plotting (like https://github.com/joouha/euporie). Lets see it as an evolutionary stepping stone - ppl first have to grasp that graphics in terminals is a thing (slowly getting there), then how ugly sixel is - to create demand for a proper protocol.

[jerch]

From my side there is nothing left to be done in Contour, after @uspasojevic96 pointed me to the settings, I just lifted the limits and it works like a charm in the puzzle. No clue where the limits came from (were at 800x600), maybe from an older prev contour version...

I just have a few questions regarding 0 in the settings:

• What happens if the system is switching display resolution in between? Automatically updated?
• Is there a higher cap in place? What happens on a 64k x 64k display wall? (prolly a good idea to have an upper limit just in case, or you might end up allocating GBs of memory before finally segfaulting)