Does not work when running Vim in a Poetry shell

[gegnew]

nvim-tmux-navigation doesn't work at all if the neovim session is run after poetry shell. Would appreciate help debugging, if possible.

:checkhealth outside of poetry shell:

Python 3 provider (optional) ~
- pyenv: Path: /opt/homebrew/Cellar/pyenv/2.3.18/libexec/pyenv
- pyenv: Root: /Users/g/.pyenv
- `g:python3_host_prog` is not set.  Searching for python3 in the environment.
- Executable: /Users/g/.pyenv/versions/3.10.11/bin/python3
- Python version: 3.10.11
- pynvim version: 0.4.3
- OK Latest pynvim is installed.

and inside poetry shell, the same except for a virtualenv error:

Python 3 provider (optional) ~
- pyenv: Path: /opt/homebrew/Cellar/pyenv/2.3.18/libexec/pyenv
- pyenv: Root: /Users/g/.pyenv
- `g:python3_host_prog` is not set.  Searching for python3 in the environment.
- Executable: /Users/g/.pyenv/versions/3.10.11/bin/python3
- Python version: 3.10.11
- pynvim version: 0.4.3
- OK Latest pynvim is installed.

Python virtualenv ~
- WARNING $VIRTUAL_ENV is set to: /Users/g/Desktop/zeit/ml/.venv
  And its /bin directory contains: python, python3, python3.10, pythoni, pythoni1
  But $PATH yields this pythoni executable: Traceback (most recent call last):
  File "/Users/g/Desktop/zeit/ml/.venv/bin/pythoni", line 30, in <module>
  from pyrepl.python_reader import main
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/python_reader.py", line 25, in <module>
  from pyrepl.completing_reader import CompletingReader
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/completing_reader.py", line 22, in <module>
  from pyrepl import commands, reader
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/commands.py", line 376, in <module>
  from pyrepl import input
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/input.py", line 39, in <module>
  from trace import trace
  ImportError: cannot import name 'trace' from 'trace' (/Users/g/.pyenv/versions/3.10.11/lib/python3.10/trace.py)
  
  And $PATH in subshells yields this pythoni executable: Traceback (most recent call last):
  File "/Users/g/Desktop/zeit/ml/.venv/bin/pythoni", line 30, in <module>
  from pyrepl.python_reader import main
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/python_reader.py", line 25, in <module>
  from pyrepl.completing_reader import CompletingReader
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/completing_reader.py", line 22, in <module>
  from pyrepl import commands, reader
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/commands.py", line 376, in <module>
  from pyrepl import input
  File "/Users/g/Desktop/zeit/ml/.venv/lib/python3.10/site-packages/pyrepl/input.py", line 39, in <module>
  from trace import trace
  ImportError: cannot import name 'trace' from 'trace' (/Users/g/.pyenv/versions/3.10.11/lib/python3.10/trace.py)
  
  And $PATH yields this pythoni1 executable:   File "/Users/g/Desktop/zeit/ml/.venv/bin/pythoni1", line 16
  print 'Python', sys.version
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
  
  And $PATH in subshells yields this pythoni1 executable:   File "/Users/g/Desktop/zeit/ml/.venv/bin/pythoni1", line 16
  print 'Python', sys.version
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
  
  So invoking Python may lead to unexpected results.
  - ADVICE:
    - $PATH ambiguities arise if the virtualenv is not properly activated prior to launching Nvim. Close Nvim, activate the virtualenv, check that invoking Python from the command line launches the correct one, then relaunch Nvim.
    - $PATH ambiguities in subshells typically are caused by your shell config overriding the $PATH previously set by the virtualenv. Either prevent them from doing so, or use this workaround: https://vi.stackexchange.com/a/34996

[gegnew]

This thread is relevant, but I didn't find a satisfactory solution: christoomey/vim-tmux-navigator#230

[gegnew]

Here is a workaround involving checking for a poetry environment. It's a little weird because the tty does not show up with poetry in it at all, but rather as a long filepath like:
S<+ /opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python

# add a check for the poetry env
is_poetry="ps -o state= -o comm= -t '#{pane_tty}' | grep -iqE 'Frameworks\/Python.framework'"

# all other lines in tmux config remain the same, but change the "bind-key" lines to:
bind -n C-h run "($is_vim && tmux send-keys C-h) || ($is_poetry && tmux send-keys C-h) || tmux select-pane -L"
bind -n C-j run "($is_vim && tmux send-keys C-j)  || ($is_poetry && tmux send-keys C-j) || tmux select-pane -D"
bind -n C-k run "($is_vim && tmux send-keys C-k) || ($is_poetry && tmux send-keys C-k)  || tmux select-pane -U"
bind -n C-l run  "($is_vim && tmux send-keys C-l) || ($is_poetry && tmux send-keys C-l) || tmux select-pane -R"
bind-key -n 'C-\\' if-shell "$is_vim" 'send-keys C-\\\\'  'select-pane -l'

[gegnew]

This should work for pipenv and other environments, but you'll need to check the process status with ps to see what your env looks like.

1. get the tty of the tmux pane containing the env with `:display-message '#{pane_tty}'
2. then check the process status: ps -o state= -o comm= -t /dev/<tty from previous step>

Edit: nevermind, still broken

[alexghergh]

Hey,

Thanks for reporting the issue! I am aware that some of the functionality of the plugin is broken inside python* shells/environments. It's always Python that's broken, am I right..

My long term idea was to completely change how the plugin detects vim/neovim and tmux, since we cannot always determine them reliably with the current solution.

However, due to life issues, I am currently unable to look into it. I will be able to do so in about 1 to 2 months.

I cannot provide a workaround solution for now, since I don't use any of the tools, however I will keep the thread open, and perhaps someone else would be able to find a fix.

In the meantime, please keep posting if you do find new information that could be helpful for a future fix.

Regards

[pacjac]

The workaround works for me, thanks @gegnew! Would love to see a fix still.

[gegnew]

The problem I have is that, although I can detect the Poetry environment, it cannot detect vim in a pane if it's in the Poetry environment. The is_poetry ps command returns true, but the is_vim poetry command returns false, because the output of ps in the pane with the Poetry env is /opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS.

If anyone has any ideas, I'm all 👂

[akail]

So, I've been coming back to this issue on and off for a while now and found how to get things working right with pipenv at least and also know why is_vim does not work for any future references. Hope it helps.

For is_vim, the ps command looks at the current pane's tty. When poetry shell or pipenv shell are called, it starts a new tty under the pipenv process. See below where you see pts/10 has been started.

akail      30255    2760  0 Dec02 pts/4    00:00:00       -zsh
akail     237624   30255  6 10:54 pts/4    00:00:00         /usr/bin/python /usr/bin/pipenv shell
akail     237626  237624  1 10:54 pts/10   00:00:00           /bin/zsh -i

I found the same experience with poetry shell

akail      30255    2760  0 Dec02 pts/4    00:00:01       -zsh
akail     238893   30255  8 11:05 pts/4    00:00:00         /usr/bin/python /usr/bin/poetry shell
akail     238899  238893  1 11:05 pts/10   00:00:00           /bin/zsh -i

I've been trying to find a way to extract all the children but it gets cumbersome pretty quickly.

Pipenv offers a flag on the shell to command --fancy which changes the behavior and keeps everything under the same tty.

  --fancy             Run in shell in fancy mode. Make sure the shell have no
                      path manipulating scripts. Run $pipenv shell for issues
                      with compatibility mode.

When running in this mode, it does not start a new process in a new tty and only seems to modify the environment variables.

[prurph]

Just wanted to chime in here with another workaround. I've found trying to modify the $is_vim logic to be unsatisfactory, and instead I just use poetry run nvim instead of doing poetry shell and then nvim. The former seems to make LSP references, diagnostics, built-in nvim terminal, etc. "just work". I had also tried launching nvim then using a plugin to choose the virtual env; this also worked but required extra steps to exclude diagnostics from showing on imported modules.

I also tried this workaround but found it to be a bit slow because it needs multiple shell commands to first see if we're inside poetry, then to get its child processes to see if we're inside vim as well. I'm not an expert and I'm working on personal projects so perhaps there are problems with this approach I'm not aware of in larger or professional settings, but afaict poetry run nvim is perfect.

[haakenlid]

One workaround is to activate the poetry virtual env in the current shell instead of of using poetry shell to spawn a subshell.

In bash you can run this command:
source $(poetry env info --path)/bin/activate

https://python-poetry.org/docs/basic-usage#activating-the-virtual-environment

[jakubbortlik]

I've been trying to find a solution for this for a long time. I generally don't use poetry shell but when I do, I'm annoyed that it breaks navigation in nvim. Or I was annoyed until I figured out the following workaround.

As described by akail, poetry shell creates a new tty, that's why ps -o state= -o comm= -t '#{pane_tty}' does not find nvim - #{pane_tty} links to the tty from which poetry shell was run, not the one started by it.

While it's possible to find the pid of the poetry process, and then find the tty of its children, this is kind of slow, as was mentioned by prurph.

I've found a way that is rather naive and may not work for everybody, but it works for me just fine and I find it practically as fast as the default solution with just one ps call.

I use the user options feature of Tmux and the fact that on my system (Manjaro Linux), ttys are numbered consecutively for each newly created shell including the poetry shell. So in my bashrc I've got this code (EDIT: I've had to update the get_next_tty function in order to provide for available intermediate tty indexes, e.g., existing indexes are 1, 3, the next available is 2, my previous code would incorrectly use 4):

get_next_tty() {
  pts_array=($(ls /dev/pts | \grep -E "^[0-9]+$"))
  for ((i=0; i<${#pts_array[@]}; i++)); do
    if [ $i -ne ${pts_array[i]} ]; then
      break
    fi
  done
  echo /dev/pts/$i
}
alias psh="[ ! $POETRY_ACTIVE ] && (tmux set -p @active_tty \$(get_next_tty) &>/dev/null; poetry shell && tmux set -pu @active_tty &>/dev/null)"

EDIT: I've added the [ ! $POETRY_ACTIVE ] check which prevents an incorrect @active_tty when I am already in a Poetry shell.

I can then start a poetry shell by typing psh in the terminal. This first creates the tmux user option @active_tty and sets it to the next available tty. It redirects stderr to /dev/null for the case that tmux is not running. Then it simply starts the poetry shell, and unsets the option after exiting the shell.

In my tmux.conf I use the ternary choice operator in the is_vim string to use the user option @active_tty if set, otherwise to fall back to #{pane_tty}:

is_vim="ps -o state= -o comm= -t '#{?#{@active_tty},#{@active_tty},#{pane_tty}}' \
    | grep -iqE '^([^TXZ ]+ +(\\S+\\/)?g?(view|n?vim?x?)(diff)?)$'"

[verisimillipede]

One workaround is to activate the poetry virtual env in the current shell instead of of using poetry shell to spawn a subshell.

In bash you can run this command: source $(poetry env info --path)/bin/activate

https://python-poetry.org/docs/basic-usage#activating-the-virtual-environment

This works, ty