Feature request: Live updating plots

[cool-RR]

It is time for the greatest VisiData feature request of all time. I've wanted this for a long time, and it's pretty ambitious so it might never get implemented, but let's put it on the table anyway.

I want to have live plots. I'm running experiments that take a while to produce output. I'm anxiously awaiting the interesting parts of their results so I could draw conculsions and run the next experiment. I want to see that progress on a plot, live. This means I want to draw a plot of a certain sheet, and then as more rows are written to that sheet, I want the plot to automatically add new points on the screen.

[midichef]

Hi @cool-RR, I submitted #2597 and #2598. Once you install both of those, you should be able to monitor your graphs with reload-every.

It was not especially complicated to implement as the basic functionality was already implemented. So imo the greatest VisiData feature request of all time is still #2: make a package for visidata.

Also, until these patches are incorporated into visidata, you may find the watch shell tool useful. Something like: watch -n 10 vd -p draw_graph.vdj

[cool-RR]

Hmm, this doesn't work for me. I checked out the develop branch and then merged your two branches. I did pip install -e . and I ran VisiData on a jsonl file that's getting new data pushed into it continuously. First I tried to use reload-modified on the sheet and draw a plot, and increased the x range expecting to see more points automatically, but I didn't. Then I went back to the sheet and tried reload-every instead which didn't work. Am I doing something wrong.

[anjakefala]

Hey @cool-RR!

To make things easier, both commits are now in develop.

The steps I would do are:
pip uninstall visidata - Make sure this is the only version of VisiData in your environment
pip install git+https://github.com:saulpw/visidata.git

And then try it out. I believe the step is to use reload-every on the underlying sheet and on the graph sheet. Let us know if you get it working!

[midichef]

Hmm, I hadn't actually tested reloading graphs while another sheet reloads from a file. That case is not working. It looks like the source sheet changes are not being reflected on the graph. I'll continue to look into it.

(My test case was a custom sheet that generates more rows over time in memory, not on disk.)

[cool-RR]

Thanks, let me know when that can be tested. Also, I think that a good UX would not require the user to go back to the source sheet and use one of the reload-* commands, but rather allow making the plot live without leaving the plot sheet. So maybe something like allowing to run the reload-* commands on the plot sheet and propagating them to the source sheet.

[cool-RR]

I recently discovered btm:

https://github.com/ClementTsang/bottom

It's like htop except it shows live updating plots of CPU usage, memory usage etc, with braille dots like VisiData. I recommend giving it a try. It's a good example of how to show a live updating plot.

[cool-RR]

@midichef What is the proper way to test the functionality you implemented? Because I tried what @anjakefala said and the plot went empty, refreshed once a second and showed this error message: [42x] still running reload_every from previous command

[midichef]

@cool-RR Anja's commands should work.

pip uninstall visidata  #make sure this is the only version of VisiData in your environment
pip install git+https://github.com:saulpw/visidata.git

Can you show me a minimal example of data file, .vdj commands, and exact vd command line that reproduces the error message (maybe file a bug report with OS etc)?

What I'm doing is:
echo "x\ty\n1\t2\n" |vd -
with the keypresses:
! # Right # .
Ctrl+^ Space reload-every 5
Ctrl+^ Space reload-every 5

The current develop branch of visidata has two drawbacks for your purpose:

1. GraphSheet does not reflect changes to the underlying sheet.
2. When the underlying sheet reloads, it loses its key columns and column types.

But there should be no message: still running reload_every from previous command.

[cool-RR]

Your example still generates that error message for me. Before I try to debug this, I want to understand: How could your example even be relevant, given that new data is not being written to the column? Why reload a plot if there's no new data?

[midichef]

My example is a minimal test case with the only goal being to probe for the error message. It's not a useful example for actually reloading a plot.

Since you're getting the error message, my best guess at what's going on is that your visidata installation is for some reason not using the latest develop branch. To check this, can you confirm that your installed features/reload_every.py matches the latest develop version?

For me, that means looking in my pip directory:
pip show visidata |grep Location

Location: /home/midichef/.venv/lib/python3.12/site-packages

md5sum /home/midichef/.venv/lib/python3.12/site-packages/visidata/features/reload_every.py |cut -f 1 -d ' '

aeb8ba99efdfe3dae5ebada78a93ff84

curl -s https://raw.githubusercontent.com/saulpw/visidata/refs/heads/develop/visidata/features/reload_every.py |md5sum

aeb8ba99efdfe3dae5ebada78a93ff84 -

Or perhaps your setup has more than one Python environment, and the old version of visidata is being run from there and not the pip installation directory?

[midichef]

Also, I suspect the reload features are going to have a lot of rough edges for the indefinite future. I'm running into unexpected difficulties working on the next steps.

If you can make do by just rerunning visidata every so often using watch with vd -p plotcmds.vdj, that's what I'd do.

[cool-RR]

Indeed I'm running the right version of the code. I tried running vd with -N and the problem went away. Then I tried running without -N and the problem doesn't appear either, so I don't know what happened.

So the feature does seem to work for that trivial case. The next step would be an example that actually writes new data.

If you can make do by just rerunning visidata every so often using [watch] (https://unix.stackexchange.com/questions/318859/how-to-use-watch-command-with-a-piped-chain-of-commands-programs) with vd -p plotcmds.vdj, that's what I'd do.

I see, too bad. I probably won't use that workaround but thanks anyway.

[anjakefala]

@cool-RR While this implementation may not fully meet your needs yet, @midichef has put effort into moving us toward the live plotting functionality you requested, and I want that acknowledged. They fixed the major bugs blocking this, helped troubleshoot your issues testing the code, been transparent about the current limitations, and offered you a workaround for them.

I really appreciate you taking this on @midichef.

[cool-RR]

Sorry if my previous response was flippant. @midichef : Thank you for all of your work on this feature.

[midichef]

No worries, I appreciate the acknowledgement.

There are two remaining issues. First, GraphSheet won't update because it uses its own copy of the rows in the source sheet, and can't see the changes. That's easy to fix, and I can submit a PR for that.

The next problem is getting the GraphSheet to redraw itself. Even if I redraw the sheet inside reload_every(), after a second of no events and no input, Visidata goes idle and starts waiting for a keypress. With ordinary background threads, Visidata would know not to go idle:

visidata/visidata/mainloop.py

Lines 261 to 262 in ff489d3

	elif vd.unfinishedThreads:
	vd.curses_timeout = nonidle_timeout

But because reload_every() has the @asyncignore decorator, the main loop does not consider there to be a background thread. So visidata goes idle.

I'm not sure how to address that issue.