Command '['ninja', '-v']' returned non-zero exit status 1 #104

zsp1993 · 2019-03-15T10:50:37Z

Traceback (most recent call last):
File "/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 576, in _build_extension_module
['ninja', '-v'], stderr=subprocess.STDOUT, cwd=build_directory)
File "/home/peng/anaconda2/envs/python36/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/home/peng/anaconda2/envs/python36/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 35, in
from network import Network
File "/home/peng/pytorch-seg-new/experiments/SimpleMerge_NYU/network.py", line 20, in
import resnet101_dilation
File "../../basemodel/resnet101_dilation.py", line 18, in
from LibInplaceABN.modules import InPlaceABNSync, ABN, GlobalAvgPool2d
File "../../lib/LibInplaceABN/modules/init.py", line 1, in
from .bn import ABN, InPlaceABN, InPlaceABNSync
File "../../lib/LibInplaceABN/modules/bn.py", line 10, in
from .functions import *
File "../../lib/LibInplaceABN/modules/functions.py", line 18, in
extra_cuda_cflags=["--expt-extended-lambda"])
File "/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 501, in load
_build_extension_module(name, build_directory)
File "/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 582, in _build_extension_module
name, error.output.decode()))
RuntimeError: Error building extension 'inplace_abn': [1/5] /usr/local/cuda-9.1/bin/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn_cuda.cu -o inplace_abn_cuda.cuda.o
FAILED: inplace_abn_cuda.cuda.o
/usr/local/cuda-9.1/bin/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn_cuda.cu -o inplace_abn_cuda.cuda.o
/bin/sh: 1: /usr/local/cuda-9.1/bin/bin/nvcc: not found
[2/5] /usr/local/cuda-9.1/bin/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn_cuda_half.cu -o inplace_abn_cuda_half.cuda.o
FAILED: inplace_abn_cuda_half.cuda.o
/usr/local/cuda-9.1/bin/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn_cuda_half.cu -o inplace_abn_cuda_half.cuda.o
/bin/sh: 1: /usr/local/cuda-9.1/bin/bin/nvcc: not found
[3/5] c++ -MMD -MF inplace_abn.o.d -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m -fPIC -std=c++11 -O3 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn.cpp -o inplace_abn.o
FAILED: inplace_abn.o
c++ -MMD -MF inplace_abn.o.d -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m -fPIC -std=c++11 -O3 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn.cpp -o inplace_abn.o
/home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn.cpp:1:29: fatal error: torch/extension.h: No such file or directory
compilation terminated.
[4/5] c++ -MMD -MF inplace_abn_cpu.o.d -DTORCH_EXTENSION_NAME=inplace_abn -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda-9.1/bin/include -I/home/peng/anaconda2/envs/python36/include/python3.6m -fPIC -std=c++11 -O3 -c /home/peng/pytorch-seg-new/lib/LibInplaceABN/modules/src/inplace_abn_cpu.cpp -o inplace_abn_cpu.o
ninja: build stopped: subcommand failed.

zsp1993 · 2019-03-15T10:51:35Z

I don't know why [1/5] /usr/local/cuda-9.1/bin/bin/nvcc has two bin

sunggukcha · 2019-03-18T08:20:38Z

I am also still being suffered from the same problem,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

I am using python3.6, torch>1.0.0, CUDA9.2, and appropriate cudnn.

sbelharbi · 2019-03-20T14:42:55Z

Hi,
I am having a similar issue. Any updates? thanks!

sbelharbi · 2019-03-22T01:19:43Z

Fix: #106 (comment)

charlesCXK · 2019-04-12T17:41:49Z

/bin/sh: 1: /usr/local/cuda-9.1/bin/bin/nvcc: not found
It is because in /home/peng/anaconda2/envs/python36/lib/python3.6/site-packages/torch/utils/cpp_extension.py, the function _find_cuda_home finds the wrong cuda_home variable. So you can modify the py tile, I mean, modify the cuda_home variable it finds, such as delete a '/bin' in it, and you will run Inplace ABN correctly.

rudylyh · 2019-06-13T23:03:53Z

This is because PyTorch failed to build the InplaceABN library. Normally, there should have some files like inplace_abn.so and build.ninja in /tmp/torch_extensions/inplace_abn/, if the just-in-time compile is successful. The failure could be caused by many factors, including gcc, cuda, ninja, pytorch.
It works for me with the combination of gcc7.3+cuda10.1+ninja1.9+pytorch0.4.1.
Took me two days to figure out :D

vakkov · 2019-06-18T16:29:53Z

Try installing ninja in such a way - https://www.claudiokuenzler.com/blog/756/install-newer-ninja-build-tools-ubuntu-14.04-trusty#.WxYrvFMvzJw

ducksoup · 2019-07-10T08:35:35Z

This is fixed since v.1.0.0.

songyuc · 2020-03-10T07:16:07Z

Hi, @ducksoup Lorenzo, I met this problem again today.

Alicegaz · 2020-08-21T14:54:13Z

I was not able to fix the issue however, the way to make command ninja executable globally is (steps 1-4):

git clone git://github.com/ninja-build/ninja.git && cd ninja
./configure.py --bootstrap
cmake -Bbuild-cmake -H.
cmake --build build-cmake

That way ninja is called only from the installation directory but not globally, the hacky way is to alias the call inside of the installation directory

alias ninja='f(){(cd /media/gazizullina/ext/ninja && /usr/bin/ninja "$@");  unset -f f;}; f'

In case you will get CMake errors in the steps above reinstall CMake like bellow and repeat steps 1-4:

1. Follow the instructions for CMake installation from https://stackoverflow.com/a/56690743/8896612
2. export CMAKE_ROOT=<path where Cmake is>

8secz-johndpope · 2021-02-05T19:29:20Z

this fixed things for me - #106

xiaoaoran · 2021-03-02T04:36:37Z

I met this problem again. My code is able to run successfully in pycharm, but meet this problem when runinng in the terminal under the same virtual env.

Anyone knows why? Appreciate.

LightingMc · 2021-06-04T12:35:01Z

@xiaoaoran I also am facing a similar issue. My code runs in a jupyter notebook, but not from a .py file via the terminal.

Jichao-Wang · 2022-08-18T06:17:14Z

I record the solution for your problem in blog.

vTuanpham · 2023-12-25T18:56:41Z

I run this command sudo apt-get update and build ninja from source via python and it suddenly work again

Aukture · 2024-02-11T14:00:37Z

Adding C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx86\x86 to environmental variables solved the problem
Don't forget to restart terminal/system after this.
Note: I'm on Windows 11.

Installing pytorch3d from conda worked but doing so from github/source failed for me (mapillary/inplace_abn#104).

RyanbowZ · 2024-04-07T23:30:57Z

If you already installed ninja on ubuntu and tried other methods but still suffer from this problem, you can check whether you installed ninja both on python pip and ubuntu environment, they will cause conflict!
I tried almost every other approach on the Internet, but finally I resolved this by:
pip uninstall ninja

ducksoup closed this as completed Jul 10, 2019

ThibaultGROUEIX mentioned this issue Dec 9, 2019

Compile error ThibaultGROUEIX/ChamferDistancePytorch#7

Closed

dzyjjpy mentioned this issue Feb 26, 2020

struggle for ninja when running test.py HRNet/HRNet-Semantic-Segmentation#104

Closed

aeoleader mentioned this issue Mar 25, 2022

run error: No module named 'ninja' GoGoDuck912/Self-Correction-Human-Parsing#56

Open

AlexiaJM mentioned this issue Aug 4, 2023

Error in training on MNIST voletiv/mcvd-pytorch#5

Open

dgorissen added a commit to dgorissen/Real3DPortrait that referenced this issue Feb 12, 2024

Update install_guide.md on pytorch3d

0e06d88

Installing pytorch3d from conda worked but doing so from github/source failed for me (mapillary/inplace_abn#104).

dgorissen mentioned this issue Feb 12, 2024

Update install_guide.md on pytorch3d yerfor/Real3DPortrait#11

Merged

balabooooo mentioned this issue Oct 9, 2024

Cannot build MultiScaleDeformableAttention balabooooo/AED#2

Open

NULLPlayer mentioned this issue Jan 28, 2025

build disable ninja open-mmlab/mmcv#3233

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Command '['ninja', '-v']' returned non-zero exit status 1 #104

Command '['ninja', '-v']' returned non-zero exit status 1 #104

zsp1993 commented Mar 15, 2019

zsp1993 commented Mar 15, 2019

sunggukcha commented Mar 18, 2019 •

edited

Loading

sbelharbi commented Mar 20, 2019

sbelharbi commented Mar 22, 2019

charlesCXK commented Apr 12, 2019

rudylyh commented Jun 13, 2019

vakkov commented Jun 18, 2019

ducksoup commented Jul 10, 2019

songyuc commented Mar 10, 2020

Alicegaz commented Aug 21, 2020 •

edited

Loading

8secz-johndpope commented Feb 5, 2021

xiaoaoran commented Mar 2, 2021 •

edited

Loading

LightingMc commented Jun 4, 2021

Jichao-Wang commented Aug 18, 2022

vTuanpham commented Dec 25, 2023

Aukture commented Feb 11, 2024

RyanbowZ commented Apr 7, 2024

Command '['ninja', '-v']' returned non-zero exit status 1 #104

Command '['ninja', '-v']' returned non-zero exit status 1 #104

Comments

zsp1993 commented Mar 15, 2019

zsp1993 commented Mar 15, 2019

sunggukcha commented Mar 18, 2019 • edited Loading

sbelharbi commented Mar 20, 2019

sbelharbi commented Mar 22, 2019

charlesCXK commented Apr 12, 2019

rudylyh commented Jun 13, 2019

vakkov commented Jun 18, 2019

ducksoup commented Jul 10, 2019

songyuc commented Mar 10, 2020

Alicegaz commented Aug 21, 2020 • edited Loading

8secz-johndpope commented Feb 5, 2021

xiaoaoran commented Mar 2, 2021 • edited Loading

LightingMc commented Jun 4, 2021

Jichao-Wang commented Aug 18, 2022

vTuanpham commented Dec 25, 2023

Aukture commented Feb 11, 2024

RyanbowZ commented Apr 7, 2024

sunggukcha commented Mar 18, 2019 •

edited

Loading

Alicegaz commented Aug 21, 2020 •

edited

Loading

xiaoaoran commented Mar 2, 2021 •

edited

Loading