Error in inspect.py

[hmgaudecker]

Running things with debug flag yields an error:

$ pytask -x --pdb
──────────────────────────────────────────────────────────────────── Start pytask session ─────────────────────────────────────────────────────────────────────
Platform: linux -- Python 3.9.13, pytask 0.2.4, pluggy 1.0.0
Root: /mnt/econ/ambig_beliefs
Configuration: /mnt/econ/ambig_beliefs/pyproject.toml
Plugins: latex-0.2.1, parallel-0.2.0
Collected 175 tasks.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py:1006 in getsourcelines          │
│                                                                                                  │
│   1003 │   original source file the first line of code was found.  An OSError is                 │
│   1004 │   raised if the source code cannot be retrieved."""                                     │
│   1005 │   object = unwrap(object)                                                               │
│ ❱ 1006 │   lines, lnum = findsource(object)                                                      │
│   1007 │                                                                                         │
│   1008 │   if istraceback(object):                                                               │
│   1009 │   │   object = object.tb_frame                                                          │
│                                                                                                  │
│ /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py:817 in findsource               │
│                                                                                                  │
│    814 │   in the file and the line number indexes a line in that list.  An OSError              │
│    815 │   is raised if the source code cannot be retrieved."""                                  │
│    816 │                                                                                         │
│ ❱  817 │   file = getsourcefile(object)                                                          │
│    818 │   if file:                                                                              │
│    819 │   │   # Invalidate cache if needed.                                                     │
│    820 │   │   linecache.checkcache(file)                                                        │
│                                                                                                  │
│ /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py:697 in getsourcefile            │
│                                                                                                  │
│    694 │   """Return the filename that can be used to locate an object's source.                 │
│    695 │   Return None if no way can be identified to get the source.                            │
│    696 │   """                                                                                   │
│ ❱  697 │   filename = getfile(object)                                                            │
│    698 │   all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:]                │
│    699 │   all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:]           │
│    700 │   if any(filename.endswith(s) for s in all_bytecode_suffixes):                          │
│                                                                                                  │
│ /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py:677 in getfile                  │
│                                                                                                  │
│    674 │   │   object = object.f_code                                                            │
│    675 │   if iscode(object):                                                                    │
│    676 │   │   return object.co_filename                                                         │
│ ❱  677 │   raise TypeError('module, class, method, function, traceback, frame, or '              │
│    678 │   │   │   │   │   'code object was expected, got {}'.format(                            │
│    679 │   │   │   │   │   type(object).__name__))                                               │
│    680                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: module, class, method, function, traceback, frame, or code object was expected, got partial

Digging a bit deeper:

[26] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py(1011)getsourcelines()
-> if istraceback(object):
   4 frames hidden (try 'help hidden_frames')
(Pdb++) object
functools.partial(<function compile_latex_document at 0x7f1c66a50ee0>, compilation_steps=[<function latexmk.<locals>.run_latexmk at 0x7f1c66a50f70>], path_to_tex=PosixPath('/path/to/ambiguity-attitudes.tex'), path_to_document=PosixPath('/other/path/to/ambiguity-attitudes.pdf'))
[18] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/inspect.py(1011)getsourcelines()
-> if istraceback(object):
   2 frames hidden (try 'help hidden_frames')
(Pdb++) u
[17] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/console.py(220)_get_source_lines()
-> return inspect.getsourcelines(function)[1]
(Pdb++) u
[16] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/console.py(177)create_url_style_for_task()
-> "line_number": _get_source_lines(task_function),
(Pdb++) u
[15] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/console.py(152)format_task_id()
-> url_style = create_url_style_for_task(task.function, editor_url_scheme)
(Pdb++) u
[14] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/live.py(250)_generate_table()
-> format_task_id(
(Pdb++) u
[13] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/live.py(175)pytask_execute_build()
-> table = self._generate_table(
(Pdb++) u
[12] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
(Pdb++) u
[11] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/pluggy/_hooks.py(265)__call__()
-> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
[9] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/pluggy/_manager.py(80)_hookexec()
-> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
   2 frames hidden (try 'help hidden_frames')
(Pdb++) n
UnboundLocalError: local variable 'lines' referenced before assignment
[5] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/click/core.py(760)invoke()
-> return __callback(*args, **kwargs)
   4 frames hidden (try 'help hidden_frames')
(Pdb++) u
[4] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/click/core.py(1404)invoke()
-> return ctx.invoke(self.callback, **ctx.params)
(Pdb++) u
[3] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/click/core.py(1657)invoke()
-> return _process_result(sub_ctx.command.invoke(sub_ctx))
(Pdb++) u
[2] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/click/core.py(1055)main()
-> rv = self.invoke(ctx)
(Pdb++) u
[1] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/click/core.py(1130)__call__()
-> return self.main(*args, **kwargs)
(Pdb++) u
[0] > /home/hmg/miniconda3/envs/ambig_beliefs/bin/pytask(10)<module>()
-> sys.exit(cli())

No big deal in the end, the issue is more that pytask quit without informing the user of anything. I ended up there somewhat by chance and ended up worrying whether recent changes to environment or tex-files had caused this; in the end I think it's about the first time I ran that project with the --pdb flag turned on.

[tobiasraabe]

Are you able to replicate it?

pytask tries to infer the file and line number of the task function which is here the function to compile a latex document that has partialed arguments. You cannot directly get the source of a partialed function, instead, you need to check the underlying function. This should be checked here:

https://github.com/pytask-dev/pytask/blob/22a1178acfd03a20aeee9354b1fd02bc747e9696/src/_pytask/console.py#L215-L220

So, I am curious why this check fails. Maybe in [17] > /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/console.py(220)_get_source_lines() -> return inspect.getsourcelines(function)[1] you can test isinstance(function, functools.partial).

[hmgaudecker]

Ah, maybe it is something introduced during my LaTeX changes after all?

> /home/hmg/miniconda3/envs/ambig_beliefs/lib/python3.9/site-packages/_pytask/console.py(220)

 215     def _get_source_lines(function: Callable[..., Any]) -> int:                                                                                               
 216         """Get the source line number of the function."""                                                                                                     
 217         if isinstance(function, functools.partial):                                                                                                           
 218             return _get_source_lines(function.func)                                                                                                           
 219         else:                                                                                                                                                 
 220  ->         return inspect.getsourcelines(function)[1]                                                                                                        
(Pdb++) function
<function wrap_function_for_post_mortem_debugging.<locals>.wrapper at 0x7f65f04d0d30>