Make it easy to crop image pixels or update images by character cells.

[amano-kenji]

Many terminal UI libraries would prefer to just display images in given character cells(for example, 30 columns 20 rows) and update images cell by cell.

Is it going to be inefficient to draw an image for the first time cell by cell? Updating an image cell by cell could be efficient.

To manipulate images cell by cell, programmers would need to know how many image pixels fit in a character cell horizontally and vertically. Physical pixels of a terminal emulator may not have the same size as image pixels.

Haskell's vty library only knows character cells. It doesn't know image pixels. If vty was able to draw an image cell by cell efficiently, it could support images easily. Because vty has to know what each character cell contains in order to know which character cells to actually update, it wouldn't use alpha channel in image pixels. For example, drawing a partially transparent image character cell over a text character cell would complicate the computation of whether a character cell needs to be updated.

[dankamongmen]

i'd disagree with this. Notcurses is, as far as i'm aware, the only terminal ui library with reasonable bitmap support. it absolutely does not want to restrict itself to character cells at a time, and does not require this from users, who can feed it an arbitrary bitmap and arbitrary pixel coordinates, and have this interact sanely with text.

it goes to fairly extreme lengths to make this work, but it works well (see https://www.youtube.com/watch?v=dcjkezf1ARY)

https://github.com/dankamongmen/notcurses/blob/8f972a3e0e638191e143845c5b980bb17c6936c6/src/lib/termdesc.h#L112 <-------------- some of the pixel information notcurses derives regarding the terminal

[AnonymouX47]

@amano-kenji I'm in support of and can testify to your first statement:

Many terminal UI libraries would prefer to just display images in given character cells

but not the rest.

Drawing graphics cell-by-cell might seem very attractive and useful to TUI developers but will be tedious for terminal emulator developers.

This will require encoding/compressing (on the application's end) and decoding/decompressing (on the terminal emulator's end) pixel data cell-by-cell, which is a really bad idea considering the encoding/compression time on the application's end, amount of memory used on the application's end (if storing the escape sequences), transmission time and decompression/decoding time on the terminal emulator's end.

At least, from my little experience, I've encoded images line-by-line and I can say that the difference between that and encoding images as a whole is quite clear. Some terminal emulators barely cope with the "line-by-line" images (though IMO that is mostly due to their bad design) e.g iTerm2.

[amano-kenji]

it absolutely does not want to restrict itself to character cells at a time

Okay. But, it's good to be able to fill character cells fully by knowing how many image pixels fit in a charater cell vertically and horizontally.

[amano-kenji]

For TUI libraries, cropping images by character cells is going to be important. To do that, they need to know how many image pixels fit in a character cell.

[AnonymouX47]

It may actually be the place of a graphics protocol to specify a means for this but there are already a couple known methods:

• TIOCGWINSZ ioctl command
  • The fastest but not always properly/fully implemented. So the size in pixels may not be correct on some terminal emulators.
  • The only problem with this is, it's not specified which dimensions are actually reported (i.e text area only or text area + chrome) and various terminal emulators mix this up.
  • From my experience, all terminal emulators that actually implement graphics protocols probably support this and give correct values.
• Xterm's XTWINOPS (Extension of ddterm's window manipulation controls; See CSI ... t here)
  • CSI 16 t (The most precise but least implemented)
  • CSI 14 t along with any means to determine the terminal size in cells and some division.
    • This is the most widely implemented but slower than TIOCGWINSZ since it requires querying the terminal.
    • Beware though, some terminal emulators mix this up with CSI 14 ; 2 t, which is very different.
    • Also, some terminal emulators wrongly swap the dimensions e.g Older VTE-based terminal emulators (but it seems to be fixed in recent versions).

It's possible for a graphics protocol to define an APC for this purpose but I'm not sure it's necessary.

EDIT: Added some points for completeness.

[amano-kenji]

The only problem with this is, it's not specified which dimensions are actually reported

Then, a graphics protocol may define TIOCGWINSZ precisely.

[amano-kenji]

There is a possibility that physical pixels of a terminal emulator may not be of the same size as image pixels.

[AnonymouX47]

Then, a graphics protocol may define TIOCGWINSZ precisely.

No, it can't because:

1. It's just not done. ioctl is a completely separate thing.
2. What if a terminal emulator implements multiple protocols?

The best that can be done IMO is to define an APC (Application Program Command; See ECMA-48 Section 8.3.2 Pg 33) or some other non-conflicting control sequence.

There is a possibility that physical pixels of a terminal emulator may not be of the same size as image pixels.

I don't understand 🤔. Care to explain?

[amano-kenji]

I don't understand. Care to explain?

https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/26#note_1770479

[AnonymouX47]

Oh, I see. You're talking about retina/HiDPI displays.

IMO, that's really for the terminal emulator to handle. As long as the terminal emulator uses the same dimensions (physical or logical) it reports to an application to process and render the pixel data sent by that application, it won't matter to the application which kind of dimensions the terminal emulator reports.

By the way, I had no idea that terminal-wg thread was still alive. Seems you woke things up :)

[amano-kenji]

The best that can be done IMO is to define an APC

Maybe, we should do this?

[AnonymouX47]

Why would you want to though?

Can you site one case where the existing methods do not suffice on a terminal emulator that actually implements a graphics protocol?

[amano-kenji]

You are the one who said

The best that can be done IMO is to define an APC

I want the best

[AnonymouX47]

I meant the best, if it had to be done.

Remember what I also said before that:

It's possible for a graphics protocol to define an APC for this purpose but I'm not sure it's necessary.

[amano-kenji]

haskell vty library records what each character cell contains in order to know which character cells to update.

For images, each rendered character cell would have to contain the hash of image pixels in the character cell or the entire image.

Should it be the hash for image pixels in a character cell or an entire image?

[ismail-yilmaz]

@amano-kenji

Should it be the hash for image pixels in a character cell or an entire image?

This depends. My approach from the start has been to keep the hash (of the image string, not of the pixels) of the complete image in the cell, along with the position (row/col info to map the image) info. It works well and is battle-tested. But of course, my approach assumes that there is -still- no new/modern image protocol.

[amano-kenji]

the image string

What is image string?

[ismail-yilmaz]

the image string

What is image string?

the image is usually a sixel stirng or a base64 encoded (passed as string via sequences, I mean). Hashing the decompressed and drawn image is expensive than hashing a
string.

[amano-kenji]

You mean compressed terminal image encoding of an image.

[amano-kenji]

I think the hash for terminal encoding of the entire image and the position info should be enough.

[ismail-yilmaz]

I think the hash for terminal encoding of the entire image and the position info should be enough.

Yes, that's what I mean.

[amano-kenji]

So, terminal emulator developers want an entire image to be drawn at once?

[amano-kenji]

By the way, man ioctl_tty says

Use of ioctl() makes for nonportable programs. Use the POSIX interface described in termios(3) whenever possible.

[ismail-yilmaz]

So, terminal emulator developers want an entire image to be drawn at once?

That info can be supplied to vte via the "new" protocol, or the host app can send partial images (some TUIs do just that.)

[amano-kenji]

Do you calculate character cells of an image that should be updated? Or, do you just draw an entire image every time?

[ismail-yilmaz]

Do you calculate character cells of an image that should be updated? Or, do you just draw an entire image every time?

No, I calculate and draw only the updated portions.

[amano-kenji]

What's your algorithm for calculating updated portions of an image?

My algorithm would be to calculate multiple contiguous portions of an image and draw one contiguous portion at a time.

[ismail-yilmaz]

What's your algorithm for calculating updated portions of an image?

My algorithm would be to calculate multiple contiguous portions of an image and draw one contiguous portion at a time.

Exactly :)

[ismail-yilmaz]

That's the optimal way -IME.

[amano-kenji]

If I draw an entire image and then draw another layer on top, then people will see flickers before the above layer is drawn?

[amano-kenji]

I wish it was efficient to draw an image cell by cell so that I don't have to calculate contiguous character cell portions of an image...

[ismail-yilmaz]

If I draw an entire image and then draw another layer on top, then people will see flickers before the above layer is drawn?

Well, that depends on the drawing methods your vte uses. It can be designed to refresh down to a singe cell-level, but I have restrictions stemming from the toolkit I use, so I never tried that. But it can be done. In my case it would be suboptimal...

[amano-kenji]

If a terminal emulator uses double buffering, then drawing an entire image and drawing another layer on top can be done without flickering.

The most painless way for TUI developers would be to just draw an image cell by cell.

[amano-kenji]

What's the toolkit you use?

[ismail-yilmaz]

If a terminal emulator uses double buffering, then drawing an entire image and drawing another layer on top can be done without flickering.

The most painless way for TUI developers would be to just draw an image cell by cell.

You can take a look at here, for what can be done, and get some visual ideas (follow the links)
The most sophisticated TUI that uses such approah is Jexer, I would suggest you test your vte using it.

[ismail-yilmaz]

What's the toolkit you use?

U++ , A C++ RAD toolkit.

[amano-kenji]

such approah

What approach? Cell-by-cell drawing? Or, double buffering? Or, contiguous character cell portions of image?