Long response streaming freezes Emacs

[ultronozm]

Sometimes when an agent finally responds after a long session, my Emacs freezes for several seconds while the text of the response populates the buffer. I suspect this is because the streaming updates rebuild the whole block every chunk, which is O(n^2) in the length of the chunk.

[xenodium]

Thanks for filing. Could be the same issue as #168. Can you update to v0.27.1 and see if that helps?

[ultronozm]

Thanks, I've updated and tested the past few days -- the issue persists. I could share a profiler report if that'd help. I suspect the issue is "rebuilding the whole block".

[ultronozm]

Another reason why I'd like to see the responses streamed less aggressively (with progressive updates rather than replacing the entire block with each chunk) is that it would allow me to add overlays (e.g., latex previews via AUCTeX) to the response while it streams, rather than waiting for the entire response to complete. Happy to look into this if you think the direction sounds reasonable.

[xenodium]

it would allow me to add overlays (e.g., latex previews via AUCTeX) to the response while it streams

From what I've seen, the streamed responses aren't always structurally complete in the context of markups, including markdown. So applying overlays would be tricky without complete ranges. This is one of the reasons agent-shell is currently applying the markdown overlays using fragment body granularity.

agent-shell/agent-shell.el

Line 1689 in e2c0aeb

(when-let ((body-start (map-nested-elt range '(:body :start)))

I intend to offer different hooks at a similar granularity so users can apply custom logic like the latex preview.

There's a PR in this space #120 but hasn't seen activity for some time. With the author's permission, I may iterate on it to get it merged.

[ultronozm]

Thanks, I see that it's a bit tricky. And yeah, I'm looking forward
to that PR.

For now, I'm using a WIP experimental branch that adds a defcustom
agent-shell-ui-fast-streaming-append to append streamed chunks in
place (instead of rebuilding the whole block). See
https://github.com/ultronozm/agent-shell/tree/perf/fast-streaming-append.
It’s been working well in my testing over the past day. In
particular, there's no problem applying LaTeX overlays to the response
as it streams. I'm not even thinking of proposing a PR until I've had
the chance to understand it properly (you can guess how it was
generated), but figured I'd share it even in an unpolished state just
in case it's of any use or inspiration.