No such file or directory: 'ninja'

[dclong]

Running pytype on a Python scripts gives me the following error message.

Computing dependencies
Analyzing 1 sources with 2 local dependencies
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workdir/user/chdu/side_projects/dbay/.venv/lib/python3.8/site-packages/pytype/__main__.py", line 10, in <module>
    sys.exit(main())
  File "/workdir/user/chdu/side_projects/dbay/.venv/lib/python3.8/site-packages/pytype/tools/analyze_project/main.py", line 97, in main
    return runner.run()
  File "/workdir/user/chdu/side_projects/dbay/.venv/lib/python3.8/site-packages/pytype/tools/analyze_project/pytype_runner.py", line 355, in run
    ret = self.build()
  File "/workdir/user/chdu/side_projects/dbay/.venv/lib/python3.8/site-packages/pytype/tools/analyze_project/pytype_runner.py", line 346, in build
    return subprocess.call(command)
  File "/usr/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ninja'

[rchen152]

Thanks for the bug report. At a glance, it looks like ninja is not installed. Did you see any error messages about ninja while pytype was installing? Is ninja available outside of pytype? (You can check by running a command such as ninja --version or which ninja on the commandline.)

[dclong]

That's what I thought and I did try to install ninja and it seems to be already installed. Not sure whether it was corrupted. Didn't try to force reinstall. Let me give it a try and get back to you.

[dclong]

I confirm that this issue was due to ninja not available rather than pytype.

[dclong]

Reopening this issue:

As a matter of fact, is it possible for pytype to use Python API of ninja rather than the ninja command? In systems where users uses Python virtual environments, it can be quite tricky to get $PATH configured right especially when users work with multiple Python environments at the same time. If pytype uses the Python API of ninja, then configuring $PATH is less a problem.

[rchen152]

Hmm, I'm skeptical that using the Python API would fix configuration issues. If you install pytype, the ninja dependency should get installed into the same environment, so if the pytype command works, I'd expect ninja to also work - unless the environment already has a corrupt ninja installation, in which case I don't see how using the Python API would save the day, since ultimately, it would still be calling into the corrupt installation.

From a maintenance perspective: ninja isn't a Python project; it's a build system for which a third party created a pypi distribution that includes a small __init__.py file that just uses subprocess to call ninja under the hood. The way things are currently set up, the pytype code does not have any dependency on this particular distribution of ninja. I'm reluctant to introduce such a dependency unless there are clear advantages.

[dclong]

Thank you for the explanation. I will close the issue now.

[adamchainz]

I have hit this bug too when using the path to my venv pytype executable without the venv activated:

$ python -m venv venv
$ source venv/bin/activate
$ pip install pytype
...
Successfully installed attrs-20.1.0 decorator-4.4.2 importlab-0.5.1 networkx-2.5 ninja-1.10.0.post1 pytype-2020.8.28 pyyaml-5.3.1 six-1.15.0 typed-ast-1.4.1
...
$ echo 1 > test.py
$ pytype test.py
Computing dependencies
Analyzing 1 sources with 0 local dependencies
ninja: Entering directory `/Users/chainz/tmp/pytype-example/.pytype'
[1/1] check test
Success: no errors found
$ deactivate
$ ./venv/bin/pytype test.py
Computing dependencies
Analyzing 1 sources with 0 local dependencies
Traceback (most recent call last):
  File "./venv/bin/pytype", line 8, in <module>
    sys.exit(main())
  File "/Users/chainz/tmp/pytype-example/venv/lib/python3.8/site-packages/pytype/tools/analyze_project/main.py", line 97, in main
    return runner.run()
  File "/Users/chainz/tmp/pytype-example/venv/lib/python3.8/site-packages/pytype/tools/analyze_project/pytype_runner.py", line 362, in run
    ret = self.build()
  File "/Users/chainz/tmp/pytype-example/venv/lib/python3.8/site-packages/pytype/tools/analyze_project/pytype_runner.py", line 353, in build
    return subprocess.call(command)
  File "/Users/chainz/.pyenv/versions/3.8.5/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Users/chainz/.pyenv/versions/3.8.5/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/chainz/.pyenv/versions/3.8.5/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ninja'

Using venv executables without the venv active is a common-enough usecase imo, useful for getting e.g. unrelated tools to run the program. The scripts in the venv/bin directory are given a shebang to run with the correct Python executable:

$ head ./venv/bin/pytype
#!/Users/chainz/tmp/pytype-example/venv/bin/python

However running subprocess.call(["something']) won't have the venv bin directory on the path. One coudl temporarily add it.

An alternative is to execute sub-commands with the equivalent of python -m package using the current interpreter: subprocess.call([sys.executable, "-m", "package"]). Unfortunately the Python ninja wrapper doesn't currently support python -m ninja, but there's an open issue: scikit-build/ninja-python-distributions#36.

[nathanielmanistaatgoogle]

I and a few other collaborators have also been hitting this in recent weeks.

I believe that steps to reproduce this are as easy as

temp-temp % python -m venv my_venv
temp-temp % ./my_venv/bin/pip install --upgrade pip pytype
  # <stuff gets installed>
temp-temp % mkdir lol
temp-temp % cd lol
lol % touch lol.py
lol % ../my_venv/bin/pytype .
Computing dependencies
Analyzing 1 sources with 0 local dependencies
Traceback (most recent call last):
  # <stack frames omitted>
FileNotFoundError: [Errno 2] No such file or directory: 'ninja': 'ninja'

. Might this be reopened? If it is reopened and if @jcfr's ninja #37 is integrated into ninja and released, would this issue become fixable according to what @adamchainz described a few days ago?

[rchen152]

Weird, I'll look into this.

[jcfr]

Might this be reopened? If it is reopened and if @jcfr's ninja #37 is integrated into ninja and released, would this issue become fixable according to what @adamchainz described a few days ago?

Thanks for the ping and for the suggestions. As soon as I am done integrating scikit-build/ninja-python-distributions#37, I will create a new release of the wheel.

I anticipate this will be completed before the end of the week.

Thanks for your patience

[rchen152]

Summarizing the issue to the best of my understanding:

pytype does not support the following common workflow: create a virtualenv, pip install X, then run X without activating the virtualenv. To fix this, pytype needs to stop doing subprocess.call(['ninja']) and instead use {path_to_venv_bin}/ninja or {path_to_venv_bin}/python -m ninja. An additional wrinkle is that python -m ninja does not currently work due to scikit-build
/ninja-python-distributions/ninja not having a __main__ file.

Once scikit-build/ninja-python-distributions#37 is merged and released, 3cfc247 appears to fix the issue. Even without that change to ninja-python-distribution, I think we could fix this by looking for a ninja executable in the same directory as the pytype executable, but that seems a little hackier, so if the ninja change will be rolled out this week, I'd prefer to wait for it.

@jcfr Thanks for all your work on ninja-python-distributions! One of the reasons we went with ninja for running pytype is how easy it is to install thanks to this project.

[adamchainz]

Good summary! Yes waiting for ninja.main seems sensible

[rchen152]

Looks like the ninja change is merged and released! :D I'll try to close this issue out tomorrow.