Quitting with a non-empty buffer causes hang on restart

[TLipede]

Before Proceeding, please acknowledge:

• I have consulted the Troubleshooting guide.
• I have searched Issues and Discussions in this repo.
• Feature requests should be initiated as Discussions. This is a bug report.

Describe the bug
If the editor is closed with the quit keybinding with a non-empty buffer, future restarts of harlequin hang. I think this is cache related as deleting the harlequin cache directory fixes this.

To Reproduce

• Start harlequin
• Type something in the editor
• Quit (^q or otherwise)
• Start harlequin

Expected behavior
Harlequin should run normally on second invocation.

Actual behavior
Harlequin hangs on second invocation.

Contributing
Are you interested in contributing a fix?

• Yes
• Maybe
• No

Additional context
I can load the cache locally after stopping harlequin using the load_cache() method, so the cache isn't corrupted. If I use the methods to clear the editor cache, harlequin also behaves normally.

What is the output of harlequin --version?

What database adapter are you using with Harlequin? (Default is duckdb)

This occurs with duckdb and postgres.

What other options are you using when invoking Harlequin? (If you are using a profile, please add relevant items from your profile here).

Just postgres connection details & limits for duckdb.

Can you tell us more about your system?

• Shell: Zsh
• Terminal: Alacritty
• OS of the shell: Void Linux (also seen on Arch Linux)
• OS of the terminal (if different from the shell): -
• Default locale / language for your OS/Terminal/Shell: en_GB.UTF-8

Tip

For example, for my system, these are:

• Bash
• Windows Terminal
• Ubuntu 22.04 / WSL2
• Windows 11
• en_US/UTF-8

How did you install Harlequin?

• pipx
• conda/mamba
• pip (global install)
• pip + venv/virtualenv or pipenv
• Poetry
• uv/Rye
• other: _____________

[tconbeer]

I can't reproduce, but I've never tried on Void+Alacritty.

Definitely sounds cache-related; load_cache() could be the problem, or it could be restoring the buffers from the cache contents (which is typically much slower than reading the cache from the disk). Harlequin restores the buffers in a separate thread after it starts (you would see a loading animation), so if Harlequin is hanging with a black screen before the first paint, then it wouldn't be that.

I wonder if there could be a deadlock on the cache file somehow?

[harrymconner]

@tconbeer @TLipede I'm having the same issue as well. Only way I can get harlequin to start up again is to clear the cache. Please let me know if there is anything else I can do to help!

• harlequin: 1.25.0
• Adapter: duckdb
• Shell: Powershell
• Terminal: Windows Terminal 1.20.11781.0
• OS of the shell: Windows 11

[tconbeer]

Thanks, @harrymconner (and sorry). I tried to reproduce on my Windows 11 machine with exactly the same setup, and couldn't, with either one or multiple buffers. Does this happen every time, or does it depend on the content of the buffers?

[harrymconner]

@tconbeer No worries at all! As far as I can tell, it doesn't matter what is in the buffer(s). It's happening every time.

Some more information if it's helpful:

1. I installed with uv
2. I'm getting the same result when I pip install harlequin into a fresh virtual environment
3. It's also happening when I switch to cmd in Windows Terminal
4. Here is an actual picture, so you can see exactly how it's hanging
   

[tconbeer]

That did it! I was able to reproduce by installing uv in Windows and then running uvx harlequin.

To verify, can you install with pipx and confirm that harlequin works fine? Nevermind just re-read your post.

[TLipede]

I was the one who installed with pipx @tconbeer. Interestingly, the cache works fine when I clone harlequin and install with poetry

[TLipede]

Okay, I think I've narrowed it down a bit. If I install the repo directly (using pip, poetry etc.) everything works as expected. But if I install from a package file (whether building the wheel locally or installing from PyPI) cache behaviour doesn't work. I'd guess at some subtleties with pickle (de-)serialization. Have you considered using a different format to store cache @tconbeer? Was there a reason you chose pickle? I was going to try implementing something super basic in json or something to see if that worked

[tconbeer]

Yeah, it's even weirder than that -- it can restore the cache if the cache is only whitespace (including newlines). But if you have a selection or non-whitespace characters, the packaged version can't restore the cache, and just hangs.

@TLipede Pickle was just a convenience.

Given WHEN this is hanging (and what I wrote above), I suspect we're reading from the cache file fine, and it's hanging when the CodeEditor/TextArea is being initialized with some text.

[nathanchapman]

I'm not sure why my previous comment was deleted, but I'm experiencing this again today with the version installed from pip as well.

Harlequin: 1.25.0
Adapter: mysql
Shell: zsh
Terminal: Warp v0.2024.10.08.08.02.stable_02
OS: macOS Sequoia 15.0.1

How are y'all clearing the cache? I can't seem to find documentation for that on the repo or the website

[sgpeter1]

There are multiple ways to go about "clearing the cache". The fastest might be just erasing the pickle files. The catalog cache seems to be working, so you might be able to just leave that file as is.

On Windows, you'll find the cached pickles here:
~\AppData\Local\harlequin\harlequin\Cache

On macOS, I think they're stored here:
~/Library/Caches/harlequin

On Linux, I think they're stored here:
~/.cache/harlequin

[tconbeer]

I think this is only an issue with 1.25.0.

If anyone could please test installing 1.24.x with pipx or uv, that would be helpful. I am considering yanking the 1.25.0 release because I don't have much time to troubleshoot this for the next 1-2 weeks

[harrymconner]

I think this is only an issue with 1.25.0.

If anyone could please test installing 1.24.x with pipx or uv, that would be helpful. I am considering yanking the 1.25.0 release because I don't have much time to troubleshoot this for the next 1-2 weeks

@tconbeer It's still hanging for me on 1.24.0 and 1.24.1 when installed with uv

[tconbeer]

@harrymconner ok thanks

[pablosantosluaces]

I think this is only an issue with 1.25.0.

If anyone could please test installing 1.24.x with pipx or uv, that would be helpful. I am considering yanking the 1.25.0 release because I don't have much time to troubleshoot this for the next 1-2 weeks

I reproduced it with 1.20 too

#666 (comment)

[tconbeer]

Update:

I was able to reproduce reliably by using uv to install my local copy of harlequin:

uvx --no-cache --from ./harlequin harlequin

Using the dev tools (message logger) isn't really possible inside the uv bundle, so I started debugging with assert False, "Here!" to find the place in the workflow where the hang was occurring... and then I couldn't reproduce the hang any more. Strange.

So then I used my normal poetry installation of harlequin to change the buffer and quit, and then running the uvx version I was able to reproduce the hang again.

I can confirm that it's not hanging in load_cache(), which reads the pickle, nor in the EditorCollection's on_mount, which populates new buffers with the cache contents. So the cache might be related, but is probably indirectly causing a race condition; the cache itself is a red herring.

There's a larger issue with the async/message pump system that is causing the infinite loop, and is probably also causing #668.

The regression seems to have come in v1.18:

uvx harlequin@1.17 # this works
uvx harlequin@1.18 # this hangs

I'm out of time right now; will try to get to the bottom of this in the next week. In the meantime, the work-around is to downgrade all the way to 1.17.0

[gatesn]

I can also reliably reproduce this with uv / pipx installed Harlequin on MacOS. The thing I noticed, is it's not deadlocking, it's hanging with 100% CPU. I used py-spy to get a thread dump which shows it's spinning in this function:

Thread 0x1F4090F40 (active+gil): "MainThread"
    _split_cells (rich/segment.py:136)
    split_cells (rich/segment.py:178)
    crop (textual/strip.py:412)
    render_line (textual/widgets/_text_area.py:1222)
    render_line (textual/_styles_cache.py:441)
    render (textual/_styles_cache.py:218)
    render_widget (textual/_styles_cache.py:119)
    render_lines (textual/widget.py:3420)
    _get_renders (textual/_compositor.py:974)
    _render_chops (textual/_compositor.py:1126)
    render_partial_update (textual/_compositor.py:1082)
    render_update (textual/_compositor.py:1024)
    _compositor_refresh (textual/screen.py:823)
    _refresh_layout (textual/screen.py:983)
    _on_timer_update (textual/screen.py:842)
    _invoke (textual/_callback.py:45)
    invoke (textual/_callback.py:85)
    _tick (textual/timer.py:185)
    _run (textual/timer.py:177)
    _run_timer (textual/timer.py:148)
    _run (asyncio/events.py:80)
    _run_once (asyncio/base_events.py:1909)
    run_forever (asyncio/base_events.py:603)
    run_until_complete (asyncio/base_events.py:636)
    run (asyncio/runners.py:44)
    run (textual/app.py:1630)
    inner_cli (harlequin/cli.py:381)
    new_func (click/decorators.py:33)
    invoke (click/core.py:782)
    invoke (click/core.py:1434)
    main (rich_click/rich_command.py:152)
    __call__ (click/core.py:1157)
    harlequin (harlequin/cli.py:414)
    <module> (harlequin:8)

Rich 13.9.2 was released ~2 weeks ago and mentions a fix to "split_cells": https://github.com/Textualize/rich/releases/tag/v13.9.2 - suspiciously around the time these reports started coming in.

Downgrading rich to 13.9.1 seems to fix it for me.

[edit] I think this is the problem/fix: Textualize/rich#3532

[tconbeer]

Thank you!! I didn't know how to do that and couldn't manage to get a trace. Thanks also for the upstream issues!

[tconbeer]

Upstream issue is fixed -- need to bump the textual version to 0.85.0 to bring the fix to Harlequin.

[tconbeer]

I have a branch with the upgrade, and can confirm it fixes this issue; unfortunately it introduces a few new ones, and it'll take me some time to iron that out. (Time I don't have today)