-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No source is visible in line_profiler output in Jupyter notebook #86
Comments
I don't use notebooks much, so I don't know how much help I can be, but maybe try with a commit close to Apr 8, 2016? @e9t seemed to indicate that the issue did not appear there? Can you try that to check to see if this is a regression, or perhaps this bug just never got fixed? In terms of the bug, it looks like it's not pulling the source file correctly, maybe a change in nbclient or one of the other nb modules? Maybe try using my wrappers around line-profiler in
Results in
That seemed to work for me. (Beside notebook magic is an anti-pattern in most cases anyway, why bother when pure python syntax works just as well, and it can be copied into a proper script?) |
The example you gave didn't work:
|
I upgraded jupyter and it stopped working for me too, so it must have to do with a jupyter update. Unfortunately I was not thinking, and I didn't record the version of juypter where it was working for me. But I suspect bisecting versions of juypter is the first step towards debugging this. |
This bug happened to me too after updating to ipykernel 6 |
Jupyter's ipykernel changed how it represents the filename of code .ipynb cells. Those code fragments don't have a real filename - they don't live in a .py file, but are just a part of a larger .ipynb notebook. Older Jupyter lab versions used See ipykernel/compiler.py. It seems like this ipykernel commit changed the previous functionality. The following workaround fixes the problem. --- line_profiler/line_profiler.orig.py 2021-08-17 15:22:41.321546451 +0200
+++ line_profiler/line_profiler.py 2021-08-17 15:44:50.587317937 +0200
@@ -14,6 +14,7 @@ except ImportError:
import functools
import inspect
import linecache
+import tempfile
import os
import sys
from argparse import ArgumentError, ArgumentParser
@@ -194,6 +195,15 @@ class LineProfiler(CLineProfiler):
return nfuncsadded
+def is_ipython_kernel_cell(filename):
+ """ Return True if a filename corresponds to a Jupyter Notebook cell
+ """
+ return (
+ filename.startswith("<ipython-input-") or
+ filename.startswith(tempfile.gettempdir() + '/ipykernel_')
+ )
+
+
def show_func(filename, start_lineno, func_name, timings, unit,
output_unit=None, stream=None, stripzeros=False):
""" Show results for a single function.
@@ -217,7 +227,7 @@ def show_func(filename, start_lineno, fu
scalar = unit / output_unit
stream.write("Total time: %g s\n" % (total_time * unit))
- if os.path.exists(filename) or filename.startswith("<ipython-input-"):
+ if os.path.exists(filename) or is_ipython_kernel_cell(filename):
stream.write("File: %s\n" % filename)
stream.write("Function: %s at line %s\n" % (func_name, start_lineno))
if os.path.exists(filename): It's not a great long-term solution. Do we even need that if-file-exists check? Instead we could drop the whole if-else, and display the lines if we have them in the line cache. If we don't, we can still display empty lines and include a warning message afterwards? |
@Grk0 It looks like the existing implementation is also a hueristic, so your fix seems to be about as good of a long term solution as the previous solution was (i.e. not good, beause we are having this conversation). I'm not sure about the existence check. Your proposal for dispalying the lines if possible and warning otherwise seems reasonable. I don't think line_profiler even has tests for IPython in it's current suite. It may be worth writing some. I've done a bit of testing with notebooks in xdoctest before. Here are those tests for reference: https://github.com/Erotemic/xdoctest/blob/master/testing/test_notebook.py |
Thanking @Grk0 for the fix. This stumped me an hour back. I'm on Python 3.8 and Linux, I use Thanks to all the team for your support on For reference these are the Jupyter versions that I'm using:
|
@ChrisAichinger will you make a PR for this? Tried some versions and the following seems to work:
Resulting in:
|
I created a PR from the patch I posted above. Sadly, I didn't have the time to write additional tests, or find a more robust fix as discussed above. |
Profiling in Jupyter Notebooks or Jupyter Lab with recent ipykernel versions (>= 6), resulted in the following error message: Could not find file /tmp/ipykernel_8298/3242911465.py Are you sure you are running this program from the same directory that you ran the profiler from? Continuing without the function's contents. The timing output then does not include source code lines. The problem is that line_profiler tries to determine when it runs in side IPython/Jupyter using a simple heuristic based on the reported filename. New ipykernel versions changed this filename pattern and broke our test. As a quick fix, amend the test so it continues working. Longer term, a more robust solution would be desirable. Fixes #86
Would it be possible to get a new release onto pypi? The public 3.3.0 is from June on both pypi and conda forge. I'll be using this in a public talk for https://pydata.org/global2021/ (actually in just a couple of weeks, ahead of the conference as a "briefing" for ticket holders on the state of the art for high performance python) and it'd be lovely to say "it just works in a Notebook". |
@ianozsvald Yes, I'm working on getting that setup. The TravisCI leak means I needed to rotate a bunch of credentials, so all of the pypi packages I manage are behind right now. I've been taking the opportunity to refine my process, which is one reason why it is taking some time. I do expect to be able to start releasing new packages by the end of the week. |
@Erotemic Thanks for the reply - and kudos to you (and colleagues?) for keeping this supported. Are you looking for support with the CI/deployment etc setup? If so I can make a mention in my data science newsletter, I've got a bunch of data engineers/systems devs on there and just possibly they'd see this as a chance to pick-up useful & visible skills. I'd be happy to mention it if that might be useful for the future? |
I think the CI setup is pretty robust at the moment, but I'm always looking to learn better ways provided there exists a path to run everything locally - I want ensure there always exists an easy straightforward path to test everything locally in order to minimize reliance on the cloud. What I mainly need to do is rotate the code-signing GPG keys (previously I was doing it with the primary key: 262A1DF005BE5D2D5210237C85CD61514641325F, but I've learned more GPG best practices, so I've generated a new primary key for the PyUtils CI (2A290272C174D28EA9CA48E9D7224DAF0347B114) and I'll be doing code signing with a subkey, so next time anything gets compromised I just need to generate new subkeys instead of new primary keys). Beyond that there are various other secrets (pypi password / github upload token) that just need to be rotated. This isn't a giant task, but I have to do this for 10+ repos, so I'm looking for better ways to automate credential rotation across github and various gitlab instances. It would also be nice if there was something I could do to trigger a test release on the pypi.test servers without modifying the github actions workflow yaml. I'll probably manually do it for now, but that's another area that I'm looking for better ways of doing things. |
@ianozsvald 3.3.1 was just published to pypi. Please verify that it does "just work in a Notebook". |
@Erotemic what's the mechanism to get I'd hoped it'd be automatic but maybe there's a bot to notify? |
It looks like it's waiting on one of the feedstock maintainers to merge. @jakirkham @grlee |
@CharlesFr Have you tried upgrading to 3.5.1? |
I'm currently having an issue identical to rkern/line_profiler#23. After I use
%lprun
in a Jupyter notebook no source is shown for the profiling.This was in a newly created conda environment, Python 3.8.10. I'm on macOS 11.4, with Google Chrome version 91.0.4472.114. However, all packages were installed via pip. I've tried installing line profiler from source at the main branch of this repository, as well as from PyPI.
!pip freeze
gives:The text was updated successfully, but these errors were encountered: