Interrupt planned cell execution

[Phyks]

Hi,

Sometimes a user can plan to execute multiple cells, by pressing Shift + Enter on them. It happens to me a lot recently to just plan execution of a bunch of cell. Some cell can take a very long time to execute, and it is just then that I realize that I would like to cancel the planned execution of the next cells. This is not possible at the moment (the only way is to interrupt the kernel, which would actually stop current cell execution and not future planned executions). I am not sure whether this could be doable, but it would be really useful (and I did not find info about such a feature on the web).

Typical usage workflow would be for a notebook like this one (pseudo-notebook):

CELL_1  # Long simulation (takes several minutes / hours)
CELL_2  # Some computation on the results of the simulation (takes a couple of minutes)

When opening the notebook, one could execute the whole notebook. Then, during the simulation, one may realize there is an error in the next computation (CELL_2), or actually that CELL_2 could be written in a more efficient way. There is no way to cancel the planned execution to reschedule it with updated code.

Thanks!

Note: This is also an issue at the moment when a cell is planned for execution and then deleted.

[gnestor]

I'd suggest just running these cells individually vs. "Run all" or "Run all below". This allows you to edit the second cell while the first cell is running.

[Phyks]

I am not sure to understand your point.

I'd suggest just running these cells individually vs. "Run all" or "Run all below".

You mean running them one by one, starting the next one whenever the first one has finished?

My workflow may be a bit non standard, but this does not solve the issue. What I am proposing here is rather to have some "queue" for planned executions of cells and be able to edit it (reordering cells, adding new cells to the queue and cancelling some planned executions).

[gnestor]

The idea of an execution queue is a good idea. I think such a feature would be best implemented as a notebook extension. It could use the cell toolbar (see an example of "cell tags" toolbar) to mark cells for upcoming execution by pressing a button or selecting a position in the queue (e.g. 1, 2, 3...). If you interested in working on such an extension, I am happy to provide direction.

[eseiver]

I'd also be interested in such a feature/extension!

[wandonye]

I found it is very common that after I "Shift + Enter" a cell so that cell enters the queue waiting other cells to finish, I found a error in the cell I just submitted. Now even the queue has not been run at all, there is no way I can fix the bug and put the cell back to the queue. A queue editor would be a good solution to this.

[uduse]

+1 on this. It would be nice if there's a plugin that controls cell execution queue.

[woodrujm]

.

[antoinetroadec]

This would be great. +1

[trivedigaurav]

Agreed. This feature would be really helpful!

[payne911]

Please!

[ismailg]

add my vote! +1

[saforem2]

Agreed. @gnestor I'd be willing to help work on this.

[gnestor]

@saforem2 Great! Are you interested in building a classic notebook extension or JupyterLab extension? I'd recommend lab because that's where almost all development resources are going these days.

[mlaugharn]

Just adding my opinion that an execution queue would be immensely helpful. I am doing some long-running multi-stage deep learning tasks on a remote billed-per-hour compute server; sometimes I realize I forgot to tweak a cell that is about to be executed, and it would be so much better to just preemptively cancel a future cell's execution rather than starting over.

[saforem2]

@gnestor I would prefer to help build a classic notebook extension just because I'm not too familiar with javascript or typescript but you're right that JupyterLab seems to be the future.

Do you think trying to build something for classic notebooks would still be worth it at this point?

[payne911]

To anyone interested, here is a list of features that you might all want to see included : #4191 (comment)

[louis925]

Looking forward to any update of this feature

[alloky]

Would be much thankfull if this feature will be released some day

[Chenart]

Agreed that this would be helpful!

[bobseboy]

Much needed indeed!

[a-dubbel]

+1

[DanMcM-SD]

+1

Sure would be nice to say "take this cell out of the queue". The one idea I had is to comment out the code in the subsequent cells so that when the long-running cell finishes it just goes through the others.

[jasongrout]

For context, for whoever wants to work on this (perhaps as a separate extension) - the way things work right now (IIRC) is that all cell executions are immediately sent to the kernel, so the execute queue is in the kernel, not in the front end. An advantage of this approach is that if there is some problem with your browser, you need to refresh, or you simply want to close your laptop and go home, the code already went to the kernel where it can be executed. This also makes it harder to adjust the queue, since you'd need some other way to communicate with the kernel to adjust the running queue.

[TomNicholas]

I personally think something as commonly useful, and intuitively necessary as this shouldn't be relegated to an extension.

So to implement this might require the kernel itself to have a list of cells to be executed? Then the notebook frontend could send a message like "wait no actually please delete the 3rd element from the cells_to_be_executed list? I know nothing about how the kernel works - is that possible?

[jasongrout]

I know nothing about how the kernel works - is that possible?

There is an interrupt message that can be sent on the control channel to the kernel - that is what is sent (IIRC) when you press the stop button in the notebook. If we keep the execution queue in the kernel, then this involves changing the kernel message protocol.

Each execution request has a message id associated with it. Perhaps one way to do this is to modify the interrupt request to optionally include a message id (or list of message ids?), which signifies that the kernel should cancel execution of those message ids if they are in the queue.

At this point, it might be best to open an issue in https://github.com/jupyter/jupyter_client, which defines the kernel messaging spec.

CC @SylvainCorlay, @afshin, @martinRenou, @JohanMabille - who are working on the kernel message protocol with their debugger work.

[ismael-elatifi]

Indeed this is a very wanted feature ! I very often need it when I want to correct a typo/bug in a cell already planned for execution.
I think a solution could be to be able to choose between 2 modes of execution :

• the classic mode (current one) "send immediately" where a cell is sent to the kernel for execution immediately (when we hit run on that cell). With this mode the queue cannot be edited easily because it would require changing the kernel message protocol as explained by @jasongrout .
• a new mode "send when previous done" where a cell we want to run is sent to the kernel only when the execution of the previous one has finished. This way we can update the code in the cell until it is sent to the kernel. We would need something to warn the user editing the cell when it has been sent to the kernel.
  If execution time of previous cells is long, this new mode allows us to use that time to fix the code in the subsequent cells, even if we already hit run on them.
  I think adding this mode is the easiest solution because it does not require to change the kernel message protocol and it only needs to be able to know when the previous cell execution is done to trigger the sending of the next one to the kernel.

[Edit] for the 2nd wanted mode, if the previous cells are still running, we would rather first cancel the execution of the cell (i.e. remove it from the queue) to have all time needed to edit the code. When the fix is done we can it run again on that cell to send it back to the queue.

[sarakodeiri]

Count my vote! This feature is much needed. +1

[Hydraxx10]

Count me in!!

[neel04]

Sad that this feature is still not implemented... 👎

[edridgedsouza]

Has this feature still not been implemented as an extension?

[opqpop]

how is this still not implemented? I do something expensive, then realize another expensive task cell wasn't right, and I want to stop the cell. But the only thing I can do is kill the kernel and now I have to re-run my first expensive task again...

[roshgor]

This would be really helpful !!!
I am training a model on server. Usually takes around three days to complete. Today is the end of second day. I accidentally ran the model initialization cell again. It has not run yet. So as far as I understand, as of now, there is no way I can save my trained model from being over written. I have to restart kernel and retrain model, and I will lose two days of training. !?

[ilzenkov]

+1 for implementing this natively, this feature would make the life of my team a lot easier.
opqpop sure said it: "I do something expensive, then realize another expensive task cell wasn't right ... and now I have to re-run my first expensive task again..."

[shugybugy-assaf]

+1 +1 ++++

[kevin-bates]

Although this sounds great and appears to be a popular request, it's something that won't be addressed in this repository and this should be re-routed to Jupyter Server (or jupyter_client per @jasongrout's #2340 (comment) above). Practically speaking, this probably requires a Jupyter Enhancement Proposal (JEP).

Perhaps we can move this to Jupyter Server first since that's the "mirror" of this repository, then decide from there?

[cossio]

@kevin-bates There is an issue already in JupyterLab: jupyterlab/jupyterlab#7825

Perhaps this issue here can be closed in favor of the one at JupyterLab?

[kevin-bates]

@cossio - I think this is a good idea. If the Lab effort requires backend tasks (which I'm sure it will), then a sibling issue/PR can be opened in that repo (jupyter_server and/or jupyter_client) when appropriate.

Closing per the latest conversation.

[ismael-elatifi]

For information I suggested this.