take advantage of kitty 0.20.0 animation protocol

[dankamongmen]

When we're doing kitty_null(), we're going into the actual string-based glyph and hacking out a transparent section. From what I'm reading of the animation protocol introduced in 0.20.0, we would be able to send just a rectangle referring to the previous image, and that would update the original. If so, this is just about perfect for our needs -- we could eliminate the entirety of kitty_null(), which is both complex and fairly slow, with an ab initio rgba={0,0,0,0} rectangle. We would use the "client-driven" model, send the update, and immediately display it. I think s=2 ought be sufficient to not enter into any kind of looping if we were using the server-driven model, but it doesn't matter.

unfortunately, this doesn't hit the streets until 0.20.0, which hasn't been released yet. we'll need a way to detect support, and we'll probably want to keep around kitty_null() as implemented for a good while, le sigh.

[dankamongmen]

This isn't really going to help us for arbitrary multiframe media. This seems more for a situation where we have some knowledge of how some aspects of a larger region are moving. For arbitrary full-size frames, this protocol doesn't seem to bring much advantage, unless there's a rectangular subset of the two frames with no change.

[dankamongmen]

so where we could use this very effectively is in wiping and rebuilding cells. it's perfectly tailored to the task.

[dankamongmen]

so this ought definitely work for rebuilding. let's test it for wiping. if it works for that (i.e. if a transparent section can replace that section of the original image, making visible material underneath), let's proceed on this ASAP. i think this requires the X=1 key.

[dankamongmen]

write(1, "                            \33_Ga=T,q=2,t=t,s=500,v=500,I=3040790678;L3RtcC90bXByN2kzanNxNi0wLnJ
write(1, "\33_Ga=a,I=3040790678,r=1,z=80\33\\", 30) = 30                                                   
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xLnJnYmE=\33\\", 85) 
write(1, "\33_Ga=a,s=2,I=3040790678,z=-1\33\\", 30) = 30                                                   
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0yLnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0zLnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi00LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi01LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi02LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi03LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi04LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi05LnJnYmE=\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xMC5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xMS5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xMi5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xMy5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNC5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=298,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNS5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=491,v=282,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xNi5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=299,I=3040790678,y=142,z=80;L3RtcC90bXByN2kzanNxNi0xNy5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=f,q=2,t=t,s=500,v=304,I=3040790678,y=137,z=80;L3RtcC90bXByN2kzanNxNi0xOC5yZ2Jh\33\\", 85) 
write(1, "\33_Ga=a,s=3,I=3040790678,z=-1\33\\", 30) = 30                                                   
write(1, "\n", 1)                       = 1                        

[dankamongmen]

it looks like you have to send both a a=f and a=a. the a=a will need a c=[N], where N increases by one with each a=f.

one thing that worries me here is that i could arbitrarily many frames. imagine a white 1x1 block running back and forth atop a bitmap (assume i can't use z=-1 for the bitmap) until the user presses a key. it might run for days. each move is going to create a new frame. is that going to lead to unbounded memory consumption (or at least blow out the image quota)? if so, we might need to recreate the image from scratch now and again, yuck...

[dankamongmen]

it looks like you have to send both a a=f and a=a. the a=a will need a c=[N], where N increases by one with each a=f.

one thing that worries me here is that i could arbitrarily many frames. imagine a white 1x1 block running back and forth atop a bitmap (assume i can't use z=-1 for the bitmap) until the user presses a key. it might run for days. each move is going to create a new frame. is that going to lead to unbounded memory consumption (or at least blow out the image quota)? if so, we might need to recreate the image from scratch now and again, yuck...

it looks like i can use d=f to delete animation frames.

[dankamongmen]

magenta-animation-working.txt

[dankamongmen]

transparent-animation-working.txt

woohoo!

[dankamongmen]

sweet sweet proof of concept: we can indeed cut a hole using kitty animations (look at the upper left of the bitmap)

[dankamongmen]

you know, if we broke the thing up into cell-sized mosaics, we could use the "delete placements intersecting specified cell" to do this very quickly. mosaics are the future for sure, if they're ever supported well.

[dankamongmen]

alright, this will be the first goal for 2.3.8.

[dankamongmen]

in addition, it looks like we can use this to reload graphics directly, i.e. without deleting them and redrawing them. it doesn't actually save much bandwidth, since deletes are small, and synchronized updates hide the effect, but it's still probably the right thing to do.

[dankamongmen]

in addition, it looks like we can use this to reload graphics directly, i.e. without deleting them and redrawing them. it doesn't actually save much bandwidth, since deletes are small, and synchronized updates hide the effect, but it's still probably the right thing to do.

yeah, we would do that by changing sprixel_recycle(). right now it calls sprixel_hide() and sprixel_alloc() for kitty. when we have animation handy, don't do that, and just return the sprixel we're passed. we'll just need some way to determine in write_kitty_data() that we're a rewrite, assuming we need send different codes (i think we will -- we'll need to send an a=f,X=1 covering the graphic, then an a=a).

[dankamongmen]

i've got the initial things set up. first off, i'm running into an EFBIG error from Kitty when I shoot over a bunch of erasure blocks, almost certainly my error. here's the relevant kitty code:

static Image*                                                                                                                                                                                                                                                                                                                                          
load_image_data(GraphicsManager *self, Image *img, const GraphicsCommand *g, const unsigned char transmission_type, const uint32_t data_fmt, const uint8_t *payload) {                                                                                                                                                                                 
    int fd;                                                                                                                                                                                                                                                                                                                                            
    static char fname[2056] = {0};                                                                                                                                                                                                                                                                                                                     
    LoadData *load_data = &self->currently_loading;                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                       
    switch(transmission_type) {                                                                                                                                                                                                                                                                                                                        
        case 'd':  // direct                                                                                                                                                                                                                                                                                                                           
            if (load_data->buf_capacity - load_data->buf_used < g->payload_sz) {                                                                                                                                                                                                                                                                       
                if (load_data->buf_used + g->payload_sz > MAX_DATA_SZ || data_fmt != PNG) ABRT("EFBIG", "Too much data");                                                                                                                                                                                                                              
                load_data->buf_capacity = MIN(2 * load_data->buf_capacity, MAX_DATA_SZ);                                                                                                                                                                                                                                                               
                load_data->buf = realloc(load_data->buf, load_data->buf_capacity);                                                                                                                                                                                                                                                                     
                if (load_data->buf == NULL) {                                                                                                                                                                                                                                                                                                          
                    load_data->buf_capacity = 0; load_data->buf_used = 0;                                                                                                                                                                                                                                                                              
                    ABRT("ENOMEM", "Out of memory");                                                                                                                                                                                                                                                                                                   
                }                                                                                                                                                                                                                                                                                                                                      
            }                                                                                                                                                                                                                                                                                                                                          
            memcpy(load_data->buf + load_data->buf_used, payload, g->payload_sz);                                                                                                                                                                                                                                                                      
            load_data->buf_used += g->payload_sz;                                                                                                                                                                                                                                                                                                      
            if (!g->more) { load_data->loading_completed_successfully = true; load_data->loading_for = (const ImageAndFrame){0}; }                                                                                                                                                                                                                     
            break;                                          

[dankamongmen]

EFBIG.txt

[dankamongmen]

even the very first animation operation draws the EFBIG so i'm pretty sure this is my bug, malformed data maybe

[dankamongmen]

yeah i've got the wrong number of As lol

[kovidgoyal]

On Wed, Jun 30, 2021 at 12:36:22PM -0700, Nick Black wrote: > @kovidgoyal, sorry if i've missed something--is there a way you'd recommend to proceed? essentially i want to be able to write animation frames composed of wholly transparent rectangles, knock out chunks of the original image with them, _and remove them, potentially out of order, restoring the original data_. i can achieve everything but the out-of-order part with what i have now. i can of course preserve the original chunk and bring that back in with an `X=1` merge, but that's undesirable for two reasons: * memory cost of keeping the old data, not a big deal * potentially creating arbitrarily many frames in a single image, and blowing out my image memory in kitty the second is the bigger issue.

Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example.

[dankamongmen]

Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example.

sure, sorry about that.

i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.

currently, i implement this by completely reloading a new image with the desired alterations, deleting the old one, and showing the new one. i can already get a slight improvement with the animation protocol by doing a complete X=1 reload of the existing image, when for instance two frames from multiframe media are completely different.

what i really want to do, though, is wipe and restore the cells using the animation protocol, so i need transmit minimal material.

save the attached file and run it as a bash script, and it will hopefully make things clear. thanks mang!

example.txt

[dankamongmen]

i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.

and yes, i'm aware of other solutions like using multiple images, but if there's a way to do this with the animation protocol, that'll be simpler for me.

[kovidgoyal]

On Wed, Jun 30, 2021 at 09:42:21PM -0700, Nick Black wrote: > Am not sure I follow what it is you are trying to do exactly, can you elucidate, possibly with an example. sure, sorry about that. i am looking to "wipe" cell-sized regions from a graphic that's already displayed, and restore the wiped-out material. The order of wipes and restores is not necessarily the same. i might need wipe out 5 cells, restore 2 of them, and wipe some more, restore one of the originals etc.

Assuming the size of wipes is much smaller than the overall image simply use r key to edit frame data. In detail: 1) Store base image as frame 1 2) Create frame 2 as a copy of frame 1 without any data transmission 3) Transmit your wipes as multiple escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat

[dankamongmen]

escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat

yes -- how do i delete a frame of the animation? i tried using c= and r= but neither seemed to work.

but this doesn't work once you have multiple deletions involved, right? because let's say i

• op1, paint c=1
• op2, remove a cell, resulting in c=2
• op3, remove another cell, resulting in c=3
• op4, remove another cell, resulting in c=4

i now want to restore only the cells removed in 2 and 4, ideally in a single operation, but multiple are ok if i don't need to retransmit the old cell (ie O(N) is ok if they're all tiny). if i could delete an operation, that works. if i can only delete a frame, then my options seem to be:

• deleting op2 and op4 from c=4, if that's supported (great!), or
• composing op3 with c=1 (requires retransmit of possibly many wipes -- need compose all unrestored ops), or
• retransmit data wiped in op2 and op4, composing with frame c=3 (requires retransmit of restored wipes)

if i go with the last option, that's not so bad from a bandwidth perspective, but i need to ensure i clean up all unused frames (here, c=2 and c=4). i can do that if it's needed, but i need know how to delete a frame (hence question at the beginning of this comment). otherwise, i consume arbitrarily many resources in kitty.

[kovidgoyal]

On Wed, Jun 30, 2021 at 10:07:31PM -0700, Nick Black wrote: > escape codes with the r key to edit frame 2 in rectangular regions 4) If you want to do it again with different wipes, delete frame 2 and create a new copy of frame 1 and repeat yes -- how do i delete a frame of the animation? i tried using `c=` and `r=` but neither seemed to work.

use the f or F keys with a=d

but this doesn't work once you have multiple deletions involved, right? because let's say i * op1, paint c=1 * op2, remove a cell, resulting in c=2 * op3, remove another cell, resulting in c=3 * op4, remove another cell, resulting in c=4

You can perform any number of operations on a single frame, you dont need to create a new one each time.

i now want to restore only the cells removed in 2 and 4, *ideally in a single operation*, but *multiple are ok if i don't need to retransmit the old cell* (ie O(N) is ok if they're all tiny). if i could delete an *operation*, that works. if i can only delete a *frame*, then my options seem to be:

No you cannot delete operations, only frames.

* deleting op2 and op4 from c=4, if that's supported (great!), or * composing op3 with c=1 (requires retransmit of possibly many wipes -- need compose all unrestored ops), or * retransmit data wiped in op2 and op4, composing with frame c=3 (requires retransmit of restored wipes)

No what you do is create c=2 perform all your wipes on it. Then when you want to restore some, create c=3 as a copy of c=2 and retransmit the data for the cells you want restored. And then display c=3 and delete c=2. You can have code decided which delta is smaller from 2->3 or 1->3 and create 3 accordingly.

[dankamongmen]

yep, that was pretty much my assessment. all right, let me do some analysis of all this and see what route to take. i think i need fresh benchmarks of mosaics. if those can be made to work, that's always the best option.

[dankamongmen]

so i'm going to pursue the strategy alluded to above: we will have a single frame on which we execute X=1 cumulative wipes. since we never need keep around the encoded graphic (we don't need redisplay it in toto as we do sixels), we can free it as soon as it's written, except that we need the data for restores. sooooo, rather than keeping the encoded kitty graphic, we'll keep X*Y encoded cells (not much more space required). go ahead and store them as auxvecs in the TAM -- we thus have a NULL s->glyph. for wiping, we don't need any encoded data; just drop the correct number of As at the correct location, and fold it in with X=1. for a rebuild, i think it might be sufficient to just drop it in with an X=1 directly from the TAM encodings. if not, build a new one and kill the old one, as kovid suggests above. either way, we never need to have more than one "live" frame plus the TAM encodings.

this ought drop our bandwidth for wipes and rebuilds on a Y*X cell graphic by a factor of (Y*X-1)/(Y*X), which ought be quite significant, and far less prone to flicker. yes. let us proceed with this plan.

together with the load+delete+display work of #1865, this ought pretty much resolve all kitty flicker, even if we don't have SUM at our disposal.

[dankamongmen]

#1898 this worked out pretty fucking well

[AnonymouX47]

Was referred here from hpjansson/chafa#104 (comment).

Just wanted to say thanks so much for documenting your thoughts and the process. I intend to embark on a similar (though much less complex, i guess) endeavor and I have high hopes that what you have here will come in handy.

Thanks.

[dankamongmen]

Was referred here from hpjansson/chafa#104 (comment).

Just wanted to say thanks so much for documenting your thoughts and the process. I intend to embark on a similar (though much less complex, i guess) endeavor and I have high hopes that what you have here will come in handy.

no problem. what's your project? this gets pretty complicated, and if you can rely on notcurses, it'll probably save you a lot of time. see https://github.com/dankamongmen/notcurses/blob/master/src/lib/kitty.c just for the kitty implementation, and there are several of those.

also, be sure you've read https://nick-black.com/dankwiki/index.php?title=Theory_and_Practice_of_Sprixels!

[AnonymouX47]

Here's my humble little project: https://github.com/AnonymouX47/term-image
Tracking kitty graphics protocol support here: AnonymouX47/term-image#40

if you can rely on notcurses, it'll probably save you a lot of time.

I'm not so sure about that given the differences in target use cases of both projects but who knows... :)
Also, the last time I looked into this, I found out the state of python bindings for notcurses wasn't so good. See ihabunek/toot#243

see https://github.com/dankamongmen/notcurses/blob/master/src/lib/kitty.c just for the kitty implementation

Will definitely look into it. Thanks

also, be sure you've read https://nick-black.com/dankwiki/index.php?title=Theory_and_Practice_of_Sprixels!

I read some of it a while back, still got the tab open... I'll find some time to finish up. Thanks