Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No source is visible in line_profiler output in Jupyter notebook #86

Closed
andyfaff opened this issue Jul 14, 2021 · 20 comments · Fixed by #93
Closed

No source is visible in line_profiler output in Jupyter notebook #86

andyfaff opened this issue Jul 14, 2021 · 20 comments · Fixed by #93

Comments

@andyfaff
Copy link

I'm currently having an issue identical to rkern/line_profiler#23. After I use %lprun in a Jupyter notebook no source is shown for the profiling.

%load_ext line_profiler
import numpy as np

def func(x):
    y = np.cos(x)
    z = np.sin(x)
    return y+z

%lprun -f func func(np.linspace(0, 1, 1001))
Timer unit: 1e-06 s

Total time: 0.000416 s

Could not find file /var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/ipykernel_18955/3242911465.py
Are you sure you are running this program from the same directory
that you ran the profiler from?
Continuing without the function's contents.

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           
     2         1         69.0     69.0     16.6  
     3         1         15.0     15.0      3.6  
     4         1        332.0    332.0     79.8

This was in a newly created conda environment, Python 3.8.10. I'm on macOS 11.4, with Google Chrome version 91.0.4472.114. However, all packages were installed via pip. I've tried installing line profiler from source at the main branch of this repository, as well as from PyPI.
!pip freeze gives:

appnope==0.1.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==21.2.0
backcall==0.2.0
beniget==0.4.0
bleach==3.3.0
certifi==2021.5.30
cffi==1.14.6
cycler==0.10.0
Cython==0.29.24
debugpy==1.3.0
decorator==5.0.9
defusedxml==0.7.1
entrypoints==0.3
gast==0.5.0
ipykernel==6.0.1
ipython==7.25.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==3.0.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kiwisolver==1.3.1
line-profiler @ file:///Users/andrew/Documents/Andy/programming/line_profiler
MarkupSafe==2.0.1
matplotlib==3.4.2
matplotlib-inline==0.1.2
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nest-asyncio==1.5.1
notebook==6.4.0
numpy==1.21.0
packaging==21.0
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.3.1
ply==3.11
prometheus-client==0.11.0
prompt-toolkit==3.0.19
ptyprocess==0.7.0
pybind11==2.6.2
pycparser==2.20
Pygments==2.9.0
pyparsing==2.4.7
PyQt5==5.15.4
PyQt5-Qt5==5.15.2
PyQt5-sip==12.9.0
pyrsistent==0.18.0
python-dateutil==2.8.2
pythran==0.9.12
pyzmq==22.1.0
qtconsole==5.1.1
QtPy==1.9.0
scipy==1.7.0
Send2Trash==1.7.1
six==1.16.0
terminado==0.10.1
testpath==0.5.0
tornado==6.1
traitlets==5.0.5
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
@Erotemic
Copy link
Member

Erotemic commented Jul 16, 2021

I don't use notebooks much, so I don't know how much help I can be, but maybe try with a commit close to Apr 8, 2016? @e9t seemed to indicate that the issue did not appear there? Can you try that to check to see if this is a regression, or perhaps this bug just never got fixed?

In terms of the bug, it looks like it's not pulling the source file correctly, maybe a change in nbclient or one of the other nb modules?

Maybe try using my wrappers around line-profiler in xdev? (will require pip install xdev)

import xdev
def func():
    return 1
xdev.profile_now(func)()

Results in

Timer unit: 1e-06 s

Total time: 4e-06 s
File: <ipython-input-1-b4453c8f0400>
Function: func at line 2

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           def func():
     3         1          4.0      4.0    100.0      return 1

That seemed to work for me. (Beside notebook magic is an anti-pattern in most cases anyway, why bother when pure python syntax works just as well, and it can be copied into a proper script?)

@andyfaff
Copy link
Author

line_profiler was working for me earlier this year.

The example you gave didn't work:

timer unit: 1e-06 s

Total time: 4e-06 s

Could not find file /var/folders/m8/gwtcncws12jf60xw5n6knwnw0000gn/T/ipykernel_52935/1497566524.py
Are you sure you are running this program from the same directory
that you ran the profiler from?
Continuing without the function's contents.

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2          
     3         1          4.0      4.0    100.0                                  
``

@Erotemic
Copy link
Member

I upgraded jupyter and it stopped working for me too, so it must have to do with a jupyter update. Unfortunately I was not thinking, and I didn't record the version of juypter where it was working for me. But I suspect bisecting versions of juypter is the first step towards debugging this.

@chourmo
Copy link

chourmo commented Jul 24, 2021

This bug happened to me too after updating to ipykernel 6

@ChrisAichinger
Copy link
Contributor

Jupyter's ipykernel changed how it represents the filename of code .ipynb cells. Those code fragments don't have a real filename - they don't live in a .py file, but are just a part of a larger .ipynb notebook.

Older Jupyter lab versions used <ipython-input-1234>-like filenames (with a random number). Newer Jupyterlab versions use /tmp/ipykernel_1234 on Linux, or $APPDATA\Local\Temp\ipykernel_1234 on Windows.

See ipykernel/compiler.py. It seems like this ipykernel commit changed the previous functionality.

The following workaround fixes the problem.

--- line_profiler/line_profiler.orig.py 2021-08-17 15:22:41.321546451 +0200
+++ line_profiler/line_profiler.py      2021-08-17 15:44:50.587317937 +0200
@@ -14,6 +14,7 @@ except ImportError:
 import functools
 import inspect
 import linecache
+import tempfile
 import os
 import sys
 from argparse import ArgumentError, ArgumentParser
@@ -194,6 +195,15 @@ class LineProfiler(CLineProfiler):
         return nfuncsadded


+def is_ipython_kernel_cell(filename):
+    """ Return True if a filename corresponds to a Jupyter Notebook cell
+    """
+    return (
+        filename.startswith("<ipython-input-") or
+        filename.startswith(tempfile.gettempdir() + '/ipykernel_')
+    )
+
+
 def show_func(filename, start_lineno, func_name, timings, unit,
     output_unit=None, stream=None, stripzeros=False):
     """ Show results for a single function.
@@ -217,7 +227,7 @@ def show_func(filename, start_lineno, fu
     scalar = unit / output_unit

     stream.write("Total time: %g s\n" % (total_time * unit))
-    if os.path.exists(filename) or filename.startswith("<ipython-input-"):
+    if os.path.exists(filename) or is_ipython_kernel_cell(filename):
         stream.write("File: %s\n" % filename)
         stream.write("Function: %s at line %s\n" % (func_name, start_lineno))
         if os.path.exists(filename):

It's not a great long-term solution. Do we even need that if-file-exists check? Instead we could drop the whole if-else, and display the lines if we have them in the line cache. If we don't, we can still display empty lines and include a warning message afterwards?

@Erotemic
Copy link
Member

@Grk0 It looks like the existing implementation is also a hueristic, so your fix seems to be about as good of a long term solution as the previous solution was (i.e. not good, beause we are having this conversation).

I'm not sure about the existence check. Your proposal for dispalying the lines if possible and warning otherwise seems reasonable.

I don't think line_profiler even has tests for IPython in it's current suite. It may be worth writing some. I've done a bit of testing with notebooks in xdoctest before. Here are those tests for reference:

https://github.com/Erotemic/xdoctest/blob/master/testing/test_notebook.py
https://github.com/Erotemic/xdoctest/blob/master/testing/notebook_with_doctests.ipynb

@ianozsvald
Copy link

Thanking @Grk0 for the fix. This stumped me an hour back.

I'm on Python 3.8 and Linux, I use line_profiler consistently if infrequently and I'll be using it in a course in a couple of months for higher-performance students.

Thanks to all the team for your support on line_profiler, it is appreciated.

For reference these are the Jupyter versions that I'm using:

$ jupyter --version
jupyter core     : 4.7.1
jupyter-notebook : 6.4.0
qtconsole        : not installed
ipython          : 7.25.0
ipykernel        : 6.0.1
jupyter client   : 6.1.12
jupyter lab      : 3.0.16
nbconvert        : 6.1.0
ipywidgets       : not installed
nbformat         : 5.1.3
traitlets        : 5.0.5

@sanderland
Copy link

sanderland commented Sep 2, 2021

@ChrisAichinger will you make a PR for this?

Tried some versions and the following seems to work:

pip install "ipykernel<6"
pip install "jupyterlab<3.1"

Resulting in:

jupyter-notebook : 6.4.3
qtconsole        : not installed
ipython          : 7.27.0
ipykernel        : 5.5.5
jupyter client   : 6.1.12
jupyter lab      : 3.0.17
nbconvert        : 6.1.0
ipywidgets       : 7.6.4
nbformat         : 5.1.3
traitlets        : 5.1.0

@ChrisAichinger
Copy link
Contributor

I created a PR from the patch I posted above.

Sadly, I didn't have the time to write additional tests, or find a more robust fix as discussed above.

Erotemic pushed a commit that referenced this issue Sep 5, 2021
Profiling in Jupyter Notebooks or Jupyter Lab with recent ipykernel
versions (>= 6), resulted in the following error message:

    Could not find file /tmp/ipykernel_8298/3242911465.py
    Are you sure you are running this program from the same
    directory that you ran the profiler from?
    Continuing without the function's contents.

The timing output then does not include source code lines.

The problem is that line_profiler tries to determine when it runs in
side IPython/Jupyter using a simple heuristic based on the reported
filename. New ipykernel versions changed this filename pattern and broke
our test.

As a quick fix, amend the test so it continues working. Longer term, a
more robust solution would be desirable.

Fixes #86
@ianozsvald
Copy link

Would it be possible to get a new release onto pypi? The public 3.3.0 is from June on both pypi and conda forge. I'll be using this in a public talk for https://pydata.org/global2021/ (actually in just a couple of weeks, ahead of the conference as a "briefing" for ticket holders on the state of the art for high performance python) and it'd be lovely to say "it just works in a Notebook".

@Erotemic
Copy link
Member

@ianozsvald Yes, I'm working on getting that setup. The TravisCI leak means I needed to rotate a bunch of credentials, so all of the pypi packages I manage are behind right now. I've been taking the opportunity to refine my process, which is one reason why it is taking some time. I do expect to be able to start releasing new packages by the end of the week.

@ianozsvald
Copy link

@Erotemic Thanks for the reply - and kudos to you (and colleagues?) for keeping this supported. Are you looking for support with the CI/deployment etc setup? If so I can make a mention in my data science newsletter, I've got a bunch of data engineers/systems devs on there and just possibly they'd see this as a chance to pick-up useful & visible skills. I'd be happy to mention it if that might be useful for the future?

@Erotemic
Copy link
Member

I think the CI setup is pretty robust at the moment, but I'm always looking to learn better ways provided there exists a path to run everything locally - I want ensure there always exists an easy straightforward path to test everything locally in order to minimize reliance on the cloud.

What I mainly need to do is rotate the code-signing GPG keys (previously I was doing it with the primary key: 262A1DF005BE5D2D5210237C85CD61514641325F, but I've learned more GPG best practices, so I've generated a new primary key for the PyUtils CI (2A290272C174D28EA9CA48E9D7224DAF0347B114) and I'll be doing code signing with a subkey, so next time anything gets compromised I just need to generate new subkeys instead of new primary keys). Beyond that there are various other secrets (pypi password / github upload token) that just need to be rotated. This isn't a giant task, but I have to do this for 10+ repos, so I'm looking for better ways to automate credential rotation across github and various gitlab instances.

It would also be nice if there was something I could do to trigger a test release on the pypi.test servers without modifying the github actions workflow yaml. I'll probably manually do it for now, but that's another area that I'm looking for better ways of doing things.

@Erotemic
Copy link
Member

@ianozsvald 3.3.1 was just published to pypi. Please verify that it does "just work in a Notebook".

@ianozsvald
Copy link

Using a pip install of 3.3.1 I confirm that %lprun works as expected in a Jupyter Lab session with an up to date installation using Python 3.8:

image

Thank you for the fix!

I'll also mention this in my newsletter - do you have any idea about the installed userbase for line_profiler?

@ianozsvald
Copy link

@Erotemic what's the mechanism to get conda-forge updated to 3.3.1? https://anaconda.org/conda-forge/line_profiler

I'd hoped it'd be automatic but maybe there's a bot to notify?

@andyfaff
Copy link
Author

It looks like it's waiting on one of the feedstock maintainers to merge. @jakirkham @grlee

@CharlesFr
Copy link

It seems this issue is still present, I'm running 3.3.1

image

@Erotemic
Copy link
Member

@CharlesFr Have you tried upgrading to 3.5.1?

@CharlesFr
Copy link

Yes, doesn't work - can someone suggest a fix?

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants