[BUG] debugger inside progress bar

[brittonsmith]

I have noticed that trying to use a debugger inside of a rich progress bar gives some unexpected behavior (at least for me). If I run the following:

from rich.progress import Progress

with Progress() as progress:
    breakpoint()

I will find myself in the debugger, but anything I type quickly vanishes from the terminal. Commands still run, but the text I type is being swallowed. Additionally, from this point, typing ctrl-c to end the session just results in --KeyboardInterrupt-- being printed to the screen, but the session does not end. I am able to end the session with a ctrl-d, but the terminal itself is left in an altered state where the cursor is no longer visible. I have tried this both on Linux and Mac (OS 10.15 using iterm) and I get the same behavior. The main issue here is that debugging when inside the progress bar is more difficult. I wonder if there is a way to improve this so the pdb environment is what one would expect and the terminal is not left in this altered state afterward.

My apologies if I have not explained this in the best way. I think trying the example will illustrate the experience I've had. Thanks for your help!

In both cases, this is rich 10.9.0.

[willmcgugan]

You are seeing this behaviour because Rich runs a thread to do the rendering. Your breakpoint has suspended the main thread, but the progress bar thread is still running and writing to stdout. When you Ctrl+C its the main thread (running the debugger) that captures it.

You could work around it entirely and use a GUI debugger like vscode.

Alternatively, you could set disable=True on the Progress constructor to disable it while you are debugging.

I suspect it is possible for the Rich thread to detect that a debugger is active and stop writing to stdout, but then the code you are running will be different when debugged, leading to potential Heisenbugs. Which is why I've not implemented it to date. But I could be persuaded.

[neutrinoceros]

Hi Will, I'll try to support our case here:
Right now it looks like this is the main (possibly only) hurdle for us at yt to migrate from tqdm to rich for progress bars. Having to adapt our debbuging workflow around it as you're suggesting @willmcgugan will likely be a deal breaker, and in any case it is disruptive for us coming from tqdm.

For reference, here's an self contained example of the how the progress bar interface is implemented in yt (the rich implementation isn't merged)

Example

from typing import Optional, Sequence
from tqdm import tqdm
from rich.progress import (
    BarColumn,
    Progress,
    ProgressColumn,
    SpinnerColumn,
    Task,
    Text,
    TextColumn,
    TimeElapsedColumn,
    TimeRemainingColumn,
)

# tqdm implementation
class TqdmProgressBar:
    def __init__(self, title, maxval):
        self._pbar = tqdm(leave=True, total=maxval, desc=title)
        self.i = 0

    def update(self, i=None):
        if i is None:
            i = self.i + 1
        n = i - self.i
        self.i = i
        self._pbar.update(n)

    def finish(self):
        self._pbar.close()


# rich implementation
class CompletionColumn(ProgressColumn):
    """Renders the current number of iterations completed."""

    def render(self, task: Task) -> Text:
        return Text(f"{task.completed} it", style="progress.percentage")


class CompletionSpeedColumn(ProgressColumn):
    """Renders human readable completion speed."""

    # This is based off rich.progress.TransferSpeedColumn

    def render(self, task: Task) -> Text:
        """Show data transfer speed."""
        speed = task.finished_speed or task.speed
        if speed is None:
            return Text("?", style="progress.data.speed")
        data_speed = int(speed)
        return Text(f"{data_speed} it/s", style="progress.data.speed")


class RichProgressBar:
    """A thin wrapper around rich.progress.Progress"""

    def __init__(
        self,
        title: str,
        maxval: Optional[int] = None,
        columns: Optional[Sequence[ProgressColumn]] = None,
    ):
        if columns is None:
            columns = [
                SpinnerColumn(),
                TextColumn("[progress.description]{task.description}"),
            ]
            if maxval is None:
                columns += [
                    CompletionColumn(),
                    TextColumn("[progress.data.speed]@"),
                    CompletionSpeedColumn(),
                ]
                # this is a workaround. As of rich 9.13.0, some of the columns
                # we're using here require a non-None value in `task.total` to render
                maxval = np.inf
            else:
                columns += [
                    BarColumn(),
                    TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
                    TimeRemainingColumn(),
                ]
            columns += [TimeElapsedColumn()]
        self._pbar = Progress(*columns)

        self._pbar.start()
        self.task = self._pbar.add_task(title, total=maxval)

    def update(
        self, *, completed: Optional[int] = None, advance: Optional[int] = None
    ) -> None:
        # We only expose the relevant subset of keyword arguments
        # supported by `rich.progress.Progress.update`
        # note that they should NOT be combined (use either `completed` or `advance` but not both)
        self._pbar.update(self.task, completed=completed, advance=advance)

    def finish(self):
        self._pbar.stop()


if __name__ == "__main__":
    pb = TqdmProgressBar('', 10)
    for i in range(10):
        pb.update(i+1)
        if i == 5:
            breakpoint()

    pb = RichProgressBar('', 10)
    for i in range(10):
        pb.update(completed=i)
        if i == 5:
            breakpoint()

running this example on my machine, I note that, other than the problem mentioned by @brittonsmith, the rich progress bar never completes even after stepping out of pdb with continue. In fact it seems that it's never updated again once the debugger has been called. On the other hand, the tqdm progress bar is completed after I return control to the script.

[willmcgugan]

tqdm uses some very similar techniques to Rich. I'll have a look at their implementation to see what they handle differently.

[waydegilliam]

Any updates on this?

[willmcgugan]

I’m uncertain what the right way to solve this would be. There’s no easy way to detect if the debugger is active in another thread, and all the potential workarounds have downsides.

Would disabling the progress bar with an env var not be a reasonable compromise?

[waydegilliam]

Would disabling the progress bar with an env var not be a reasonable compromise?

That's what I'm doing right now. I haven't looked too much into fixing this myself either, so while it's a bit annoying to lose the progress bars during debugging I can work with it.

[willmcgugan]

Going to close this for now, but I am still cognizant of the issue.

I don't know what or if there is a solution to this, other than use a GUI debugger.

[github-actions]

Did I solve your problem?

Why not buy the devs a coffee to say thanks?