quantanal claims that onedot.png misses color

[dankamongmen]

In #2501 I introduced the quantanal tool with which I'll be measuring our efforts to improve Sixel performance and quality. Running it on data/onedot.png, we would expect a perfect match (all 0 deltas). Instead, quantanal claims a significant miss:

[schwarzgerat](0) $ ./quantanal ../data/onedot.png 
analyzing ../data/onedot.png...
 source pixels: 1x1
 rendered: 1x1 4B
 control sequence: 34 bytes (850%)
 00000001 pixels analyzed Δr 46 Δg 77 Δb 35
 1px Δr 46 (46) Δg 77 (77) Δb 35 (35)
 avg diff per pixel: 158
done with ../data/onedot.png.
[schwarzgerat](0) $ 

so either quantanal is not quite yet ready for primetime, or our sixel quantization is well and truly fucked.

[dankamongmen]

technically we don't expect exact consonance, since Sixel is fundamentally incapable of expressing all colors due to [0..255] being mapped to [0..100]. but we expect a much closer match than this.

[dankamongmen]

original pixel: 0xffcb2ea9
transmogrified: 0xffa87b7b

[schwarzgerat](1) $ oiiotool --info --dumpdata -a ../data/onedot.png 
../data/onedot.png   :    1 x    1, 3 channel, uint8 png
    Pixel (0, 0): 169 46 203 (0.662745 0.180392 0.796079)
[schwarzgerat](0) $ 

hrmmm, looks like we might have something reversed here?

cb2ea9: 203 46 169
a87b7b: 168, 123, 123
actual: 169 46 203

well we're learning all kinds of things today, aren't we

[dankamongmen]

it appears to be rendered as a82eca

[dankamongmen]

so a82eca (the actual render color) is what we would expect -- 168, 46, 202. that's one off on R and B, dead-on for G, the amount of distortion we would expect from the Sixel reduction. so this is a quantanal problem, or possibly one with the pixel API.

[dankamongmen]

so what's up? is our sixel intake busted? that seems plausible.

[dankamongmen]

Got color 0: 66 18 79

that's 168.3, 45.9, and 201.45.

analyzing ../data/onedot.png...
 source pixels: 1x1            
 rendered: 1x1 4B
 control sequence: 34 bytes (850%)
Got color 0: 0xffc92da8 168 45 201
0xffcb2ea9 0xffc92da8
 00000001 pixels analyzed Δr 1 Δg 1 Δb 2

 avg diff per pixel: 4
done with ../data/onedot.png.2)

ahh, much better

[dankamongmen]

oughtn't 45.9 become 46, not 45?

[dankamongmen]

[schwarzgerat](0) $ ./quantanal ../data/worldmap.png 
analyzing ../data/worldmap.png...
 source pixels: 475x860
 rendered: 475x860 1634000B
 control sequence: 130695 bytes (7.998%)
 00408500 pixels analyzed Δr 828526 Δg 899641 Δb 877151
 408500px Δr 828526 (2.03) Δg 899641 (2.2) Δb 877151 (2.15)
 avg diff per pixel: 6.38
done with ../data/worldmap.png.
[schwarzgerat](0) $ 

we're looking pretty solid on data/worldmap.png now that we're getting valid data.

[dankamongmen]

well, we know my algorithm is pretty precise. that's why it's so slow, and why we can only handle 256 colors without spinning out. it's not like libsixel's mincut is superior to my flattening algorithm; it's that 1024 colors are going to waste 256.

[dankamongmen]

analyzing /media/chungus/images/allrgb/balloon.png...
 Image too large, scaling to display
 source pixels: 4096x4096
 rendered: 1380x880 4857600B
 control sequence: 1465894 bytes (30.18%)
 avg diff per pixel: 275d Δr 102562853 Δg 94829553 Δb 116289796
done with /media/chungus/images/allrgb/penrose-tiling.png.9796 (95.8)
analyzing /media/chungus/images/allrgb/rainbow-portal.png...
done with /media/chungus/images/allrgb/balloon.png.
analyzing /media/chungus/images/allrgb/flaming-rectangle.png...
 Image too large, scaling to display                           
 source pixels: 4096x4096                                            
 rendered: 1380x880 4857600B
 control sequence: 4080714 bytes (84.01%)          
 avg diff per pixel: 275d Δr 99133730 Δg 88082858 Δb 91080903  
done with /media/chungus/images/allrgb/penrose-tiling.png.03 (75)
analyzing /media/chungus/images/allrgb/rainbow-portal.png...
done with /media/chungus/images/allrgb/flaming-rectangle.png.
analyzing /media/chungus/images/allrgb/fractal-hue.png...
 Image too large, scaling to display                         
 source pixels: 4096x4096                                        
 rendered: 1380x880 4857600B                                    
 control sequence: 2159447 bytes (44.46%)                            
 01214400 pixels analyzed Δr 101650298 Δg 106241769 Δb 100921336
 1214400px Δr 101650298 (83.7) Δg 106241769 (87.5) Δb 100921336 (83.1)
 avg diff per pixel: 254                                
done with /media/chungus/images/allrgb/fractal-hue.png.
analyzing /media/chungus/images/allrgb/newtonian-fractal-1-redux.png...
 Image too large, scaling to display                            
 source pixels: 4096x4096                                             
 rendered: 1380x880 4857600B
 control sequence: 1571759 bytes (32.36%)                    
 01214400 pixels analyzed Δr 90273094 Δg 75537885 Δb 109529687         
 1214400px Δr 90273094 (74.3) Δg 75537885 (62.2) Δb 109529687 (90.2)
 avg diff per pixel: 227 
done with /media/chungus/images/allrgb/newtonian-fractal-1-redux.png.
analyzing /media/chungus/images/allrgb/newtonian-fractal-2-redux.png...
 Image too large, scaling to display                            
 source pixels: 4096x4096                                             
 rendered: 1380x880 4857600B
 control sequence: 1689771 bytes (34.79%)                            
 01214400 pixels analyzed Δr 91675957 Δg 105308909 Δb 84890729         
 1214400px Δr 91675957 (75.5) Δg 105308909 (86.7) Δb 84890729 (69.9)
 avg diff per pixel: 232 
done with /media/chungus/images/allrgb/newtonian-fractal-2-redux.png.
analyzing /media/chungus/images/allrgb/newtonian-fractal-3-redux.png...
 Image too large, scaling to display                          
 source pixels: 4096x4096                                           
 rendered: 1380x880 4857600B
 control sequence: 1707661 bytes (35.15%)                            
 01214400 pixels analyzed Δr 118841339 Δg 73517948 Δb 84382543         
 1214400px Δr 118841339 (97.9) Δg 73517948 (60.5) Δb 84382543 (69.5)
 avg diff per pixel: 228 
done with /media/chungus/images/allrgb/newtonian-fractal-3-redux.png.

[dankamongmen]

it occurs to me that we ought be easily able to calculate a best-case expectation function for the allrgb images. you have 4096x4096 geometry (2^12 * 2^12 == 2^24) and one pixel of each of 16M (2^24) colors. so you're mapping the entire RGB space to the Sixel color space. ideally this would lead to 2.56 RGB shades per Sixel component, or 16.77 RGB colors per Sixel color (16.77 == 2.56^3 == 2^24/100^3). at 256 colors, we want 2^16 (64K) RGB colors per Sixel color; at 1024, we want 2^14 (16K) RGB colors per Sixel color. you want the 64K with the minimal distance to your sixel color. so that's gonna be

ways to differ by 0: 1
differ by 1: 6 (change one component by 1) == 6
differ by 2: 6 (change one component by 2) + 15 (change two components by 1) == 21
differ by 3: 6 (change one component by 3) + 30 (change one component by 2, and another by 1) + 20 (change three components by 1) == 56

well fuck all that it's gonna be the cube root of 2^16 so a little over 2^5 max on each component, so 32* 32*64 / 2, so about 2^5.32/2 ~= 20 away on any component. total miss of about 60, right?

hrmm so for a 0 or 255, they can't accommodate smaller or larger values. so ought they have larger absolute value distances than other numbers in such a unform distribution? like if you had to map [0..98] to [0..2], you'd want [0..32] mapped to 0, [33..65] mapped to 1, and [66..98] mapped to 2, right? hrmmm well i guess it depends on whether you map back to {0, 33, 66} or {0, 49, 98}; in the first case, you're always taking the 32 above the truncated form; in the latter, you've got bounds of {0, 32}, {-16, 16}, and {-32, 0}. well, i suppose that's just a matter of unit convenience, then.

[dankamongmen]

how exactly does the sixel space get translated back to RGB? like does 0 sixel equal 0 or 1.28 (1)? does 1 equal 3 or 3.84 or 5.12? does 99 equal 253.44, or 254.72, or 255?

[dankamongmen]

let's take this to #1452