Sixel quality improvements

[hpjansson]

Discussed in #173

^{Originally posted by helma October 4, 2023}
I intend to use chafa as a previewer for my photos, but have sometimes problems with the rendering quality of slow color gradients (e.g. cloudy skies), that come out very quantized with distinct color bands. Crisp photos render however perfectly as sixels. I have tried chafa, notcurses, img2sixel and imagemagick and in general I can get the best results with the -I option of img2sixel, followed by magick, notcurses and chafa (my terminal is foot version: 1.14.0 +pgo +ime +graphemes -assertions from nixos 23.05). Here are the resulting sixel files, png images have been generated with sixel2img:

Original (not very beautiful, but shows the effect very clearly):

chafa source.jpeg > chafa.six (Chafa version 1.12.4)

https://files.pdp8.info/chafa.six

ncplayer -t0 -q -k -s scalehi source.jpeg > notcurses.six (ncplayer version 3.0.9)

https://files.pdp8.info/notcurses.six

magick source.jpeg -resize 1914x1188> sixel:- > magick.six (Version: ImageMagick 7.1.1-18 Q16-HDRI x86_64 8c6cdf58e:20230923 https://imagemagick.org)

https://files.pdp8.info/magick.six

img2sixel -I -w 1914 source.jpeg > img2sixel.six (img2sixel 1.10.3)

Conversion to png spoils it completely, but img2sixel gives by far the best approximation of the original with the -I option.
The high color option may cause however artefacts at the bottom of some images (maybe related to libsixel/libsixel#16).

https://files.pdp8.info/img2sixel.six

Are there any options in chafa to improve sixel rendering of photos? Dithering can improve the appearance of this file, but I loose details in sharper shots. I would prefer to use chafa over the other options because of speed and active maintainance.

[hpjansson]

As mentioned in the discussion, there's currently no way for the user to get better quality.

Let's use this issue for improvements to the default quantizer and ways to trade off work/quality with the -w option.

[johnd0e]

we're probably allocating a disproportionate amount of palette entries to the most common color ranges (cf. the dark areas), and too few to rarer colors.

Ideally, the palette should not be the same for every image, but rather tailored to each one.

[hpjansson]

Ideally, the palette should not be the same for every image, but rather tailored to each one.

It does use a dynamic palette, but the palette generator isn't great for quality. I think it's possible to get better results without making performance suffer too much, though.

[jerch]

Wow that cloud image is an interesting one, it also steps into an edge case in my octree impl cutting off too many of the "second line colors" during my 2nd-pass tree pruning. Thx for that image, a good test case indeed.