-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not getting a CellExecutionError raised with %run commands #48
Comments
What's the specific pattern you used with In this case since you're in a python process already, Why not use https://github.com/jupyter/nbclient/blob/master/nbclient/client.py#L839 directly in your master notebook? Also if you're not aware, papermill is a little more opinionated for pipeline work (has parameterization and input / output notebook isolation / CLI) and uses nbclient under the hood. |
I am doing it like: %run ./folder/notebook.ipynb I'd rather not introduce nbclient into a notebook, where the magic command works just fine. In my opinion, nbclient should "just work" in these circumstances and run all notebooks from the terminal as I would expect them to in the browser. My temporary solution has been to recreate the master notebook's manifest of sub notebooks in my nbclient shell script so that it reproduces what's being done with %run. This requires me to keep the manifest list in two different places, but at least I see the errors now. |
Just to add a bit more context, I tried re-creating and: If I have three noteboks:
So the chain is always triggered by running nbclient from the third notebook. In the second notebook:
|
It's that last bullet point where my use case falls that I think you should address. Here's my workflow where it comes up. We have a closed-source notebook pipeline behind datadesk/california-coronavirus-data that downloads several dozen data sources from Google Sheets, GitHub, open data portals, etc., and then processes it to go on our live page on latimes.com. Each download routine and each process routine is a separate notebook. In total, it's more than 20 notebook files. There are three master notebooks (download, process and optimize) that round all the notebooks into straightforward routines. They include documentation explaining what each notebook does. Those three master notebooks are run manually five times a day (at least) after we finish a run manually collecting data from 61 state agencies across the state. (I just finished the 9 a.m. run myself before writing this message.) To simplify things, the entrant is asked to run a single Without the |
Thanks for the clarification @palewire - two quick thoughts - one on-topic and one off-topic:
I think that's a very reasonable workflow (it sounds quite cool!). However the feeling I get is this is precisely what papermill was designed for. My intuition is that the "magic" commands are all tailored for an interactive session, and so in general they may break in unexpected ways when use programmatically. Is accomplishing the same thing with papermill or with nbclient a non-starter for you? I think in general it is not good practice to include magic commands in production (but that's just my opinion).
Just a friendly open sourcey note here - that "you" can also be you ;-) this is a 100% volunteer project and contributions are most-welcome. I try to use passive tense in these kinds of statements: "this is an issue I think should be addressed" rather than "this is an issue I think you should address". Don't forget we're all doing this in our spare time. (apologies if this comes across as super pedantic and condescending, I promise I don't mean it that way) |
I definitely appreciate your effort here and didn't mean to offend. I am trying to articulate my problem and opinion about the issue as clearly as I can, not attack anyone. So is it the case that the |
No offense taken - and I appreciate you taking the time to explain further, maybe I am just being pedantic after all 😬
Ahh - your point made me try out something new which makes me think that you are right: If I run e.g. hitting restart and run-all with while raising a "proper" error gives this: So this makes me think that IPython is just displaying an error, rather than actually raising an error. |
@palewire I agree that That being said, I definitely second @choldgraf 's point that papermill was made for running these -- it lets you parameterize a single notebook (say with the |
also cc @Carreau who I believe may have more cycles for ipython these days, and may be interested in this issue? |
I started a modest PR on their end. ipython/ipython#12301 |
The patch we devised here has been merged into the IPython trunk. |
yahooo! @palewire thanks so much for upstreaming this one ❤️ |
I follow the practice of creating a master notebook that runs all the notebooks in my pipeline via the
%run
magic command. When I execute that notebook with nbclient, it fails to raise errors when they happen in the sub-notebooks. This is, for obvious reasons, not good. Is there a way to address this?The text was updated successfully, but these errors were encountered: