[Build] Unable to suppress unused variable #16000

mc-nv · 2023-05-18T05:36:12Z

Describe the issue

Hi OnnxRuntime team,
I'm trying to build OnnxRuntime on Jetson platform.

But keep failing due to unused variable error:

./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib --build_dir /home/nvidia/triton/onnxruntime/build --update --build --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --use_tensorrt --tensorrt_home /usr/src/tensorrt --cmake_extra_defines CMAKE_CXX_FLAGS=-Wunused-variable --cmake_extra_defines 'CMAKE_CUDA_ARCHITECTURES=53;62;72;87'
...
...
...
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.10") 
Generated: /raid/home/nvidia/triton/onnxruntime/build/Release/_deps/onnx-build/onnx/onnx-ml.proto
Generated: /raid/home/nvidia/triton/onnxruntime/build/Release/_deps/onnx-build/onnx/onnx-operators-ml.proto
Generated: /raid/home/nvidia/triton/onnxruntime/build/Release/_deps/onnx-build/onnx/onnx-data.proto
-- 
-- ******** Summary ********
--   CMake version             : 3.26.3
--   CMake command             : /usr/bin/cmake
--   System                    : Linux
--   C++ compiler              : /usr/bin/c++
--   C++ compiler version      : 9.4.0
--   CXX flags                 : -Wunused-variable -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -Wnon-virtual-dtor
--   Build type                : Release
--   Compile definitions       : ORT_ENABLE_STREAM;EIGEN_MPL2_ONLY;_GNU_SOURCE;__STDC_FORMAT_MACROS
--   CMAKE_PREFIX_PATH         : /home/nvidia/triton/onnxruntime/build/Release/installed
--   CMAKE_INSTALL_PREFIX      : /usr/local
--   CMAKE_MODULE_PATH         : /raid/home/nvidia/triton/onnxruntime/cmake/external
-- 
--   ONNX version              : 1.14.0rc1
--   ONNX NAMESPACE            : onnx
--   ONNX_USE_LITE_PROTO       : OFF
--   USE_PROTOBUF_SHARED_LIBS  : OFF
--   Protobuf_USE_STATIC_LIBS  : ON
--   ONNX_DISABLE_EXCEPTIONS   : OFF
--   ONNX_WERROR               : OFF
--   ONNX_BUILD_TESTS          : OFF
--   ONNX_BUILD_BENCHMARKS     : OFF
-- 
--   Protobuf compiler         : 
--   Protobuf includes         : 
--   Protobuf libraries        : 
--   BUILD_ONNX_PYTHON         : OFF
Finished fetching external dependencies
-- Performing Test HAS_UNUSED_BUT_SET_PARAMETER
-- Performing Test HAS_UNUSED_BUT_SET_PARAMETER - Success
-- Performing Test HAS_UNUSED_BUT_SET_VARIABLE
-- Performing Test HAS_UNUSED_BUT_SET_VARIABLE - Success
-- Performing Test HAS_UNUSED_VARIABLE
-- Performing Test HAS_UNUSED_VARIABLE - Success
...
...
...
[ 56%] Building CXX object CMakeFiles/onnxruntime_providers.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/core/providers/cpu/math/gemm.cc.o
[ 56%] Building CXX object CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc.o
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of ‘onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = onnxruntime::MLFloat16]’:
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:105:8: error: unused variable ‘is_mask_1d_key_seq_len_start’ [-Werror=unused-variable]
  105 |   bool is_mask_1d_key_seq_len_start = parameters.mask_type == AttentionMaskType::MASK_1D_KEY_SEQ_LEN_START;
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of ‘onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = float]’:
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:105:8: error: unused variable ‘is_mask_1d_key_seq_len_start’ [-Werror=unused-variable]
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:2084: CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
...
...
...
[ 66%] Linking CXX static library libonnxruntime_providers.a
[ 66%] Built target onnxruntime_providers
make: *** [Makefile:166: all] Error 2
Traceback (most recent call last):
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 2597, in <module>
    sys.exit(main())
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 2493, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 1432, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 779, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/raid/home/nvidia/triton/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/home/nvidia/triton/onnxruntime/build/Release', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.

Urgency

We are trying to build it against rel-1.15.0 branch and I would like to confirm it before release date.

Target platform

aarch64

Build script

./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib --build_dir /home/nvidia/triton/onnxruntime/build --update --build --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --use_tensorrt --tensorrt_home /usr/src/tensorrt --cmake_extra_defines CMAKE_CXX_FLAGS=-Wunused-variable --cmake_extra_defines 'CMAKE_CUDA_ARCHITECTURES=53;62;72;87'

Error / output

[ 56%] Building CXX object CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc.o
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of ‘onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = onnxruntime::MLFloat16]’:
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:105:8: error: unused variable ‘is_mask_1d_key_seq_len_start’ [-Werror=unused-variable]
  105 |   bool is_mask_1d_key_seq_len_start = parameters.mask_type == AttentionMaskType::MASK_1D_KEY_SEQ_LEN_START;
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of ‘onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = float]’:
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:105:8: error: unused variable ‘is_mask_1d_key_seq_len_start’ [-Werror=unused-variable]
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:2084: CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....

Visual Studio Version

No response

GCC / Compiler Version

No response

The text was updated successfully, but these errors were encountered:

mc-nv · 2023-05-18T05:40:01Z

cc: @pranavsharma

snnn · 2023-05-18T05:46:36Z

Maybe your CUDA version is too old.
Please let us know which version you have.

mc-nv · 2023-05-18T06:01:24Z

The CUDA version installed cuda-driver-dev-11-4/stable,now 11.4.298-1 arm64 [installed]

snnn · 2023-05-18T06:03:15Z

That's the reason. Can you upgrade it to CUDA 11.6 or above? All our CI pipelines are using CUDA 11.8. We support CUDA 11.8 better.

jywu-msft · 2023-05-18T14:41:05Z

i noticed you tried to supress warnings as error via --cmake_extra_defines CMAKE_CXX_FLAGS=-Wunused-variable
try adding --compile_no_warning_as_error option to build.py as a temp workaround instead.
@snnn , it can probably be fixed by adding ORT_UNUSED_VARIABLE to the #else block at

onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc

Line 166 in 648bedf

constexpr bool use_memory_efficient_attention = false;

jywu-msft · 2023-05-18T14:56:29Z

@snnn , it can probably be fixed by adding ORT_UNUSED_VARIABLE to the #else block at

onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc

Line 166 in 648bedf

constexpr bool use_memory_efficient_attention = false;

#16010

snnn · 2023-05-18T17:09:37Z

@mc-nv , could you help verify @jywu-msft's fix?

mc-nv · 2023-05-18T17:12:02Z

I'm in process of verification provided above suppression with --compile_no_warning_as_error flag.

I will keep you posted on my results.

jywu-msft · 2023-05-18T17:13:53Z

I'm in process of verification provided above suppression with --compile_no_warning_as_error flag.

I will keep you posted on my results.

I created a PR for a targeted fix #16010
(so you wouldn't need the --compile_no_warning_as_error if the fix works)

mc-nv · 2023-05-18T18:53:22Z

Given flag allows me to bypass above issue but build itself wasn't successful.
It was triggered as:

./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib --build_dir /home/nvidia/triton/onnxruntime/build --update --build --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --use_tensorrt --tensorrt_home /usr/src/tensorrt --cmake_extra_defines CMAKE_CXX_FLAGS=-Wunused-variable --cmake_extra_defines 'CMAKE_CUDA_ARCHITECTURES=53;62;72;87' --compile_no_warning_as_error

And return error:

[ 80%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/ngram_repeat_block_impl.cu.o
[ 80%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/packed_attention_impl.cu.o
[ 80%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu.o
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu(159): error: identifier "__hmul2_rn" is undefined

1 error detected in the compilation of "/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu".
make[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:5538: CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/relative_attn_bias_impl.cu.o] Error 255
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:1933: CMakeFiles/onnxruntime_providers_cuda.dir/all] Error 2
make: *** [Makefile:166: all] Error 2
Traceback (most recent call last):
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 2597, in <module>
    sys.exit(main())
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 2493, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 1432, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/nvidia/triton/onnxruntime/tools/ci_build/build.py", line 779, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/raid/home/nvidia/triton/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/home/nvidia/triton/onnxruntime/build/Release', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.

jywu-msft · 2023-05-18T19:40:20Z

seems related to #15840 and fixed by #15852
we need that fix cherry picked to rel-1.15 if not already there?
fyi @snnn , @wangyems

mc-nv · 2023-05-18T19:49:57Z

can we have a branch with cherry-pick commits I can refer to in order to confirm the fixes?

jywu-msft · 2023-05-18T19:53:57Z

can we have a branch with cherry-pick commits I can refer to in order to confirm the fixes?

can you test main branch and rel-1.15.0 ? if main branch works and 1.15 does not, it means we need to cherry pick missing fixes into rel branch.

mc-nv · 2023-05-18T22:14:04Z

Working on it...

fix #16000

mc-nv · 2023-05-19T01:49:20Z

I was able able to build the OnnxRuntime on Jetson device using main branch.
Could you please bring appropriate changes to 1.15.0 release.

Thank you

fix #16000

snnn · 2023-05-20T04:50:16Z

Done.

@mc-nv , could you please help verify if the latest rel-1.15.0 branch is good?

mc-nv · 2023-05-20T04:57:52Z

I'm checking.
So far I can confirm that we were able to build ONNXRUNTIME on existing hardware (x84, aarch64 etc), still need some time for testing...

In context of current issue I can confirm that we were able to build build ORT with rel-1.15.0

mc-nv · 2023-05-25T04:54:42Z

@snnn I may have to ask you revert this commit #16010 .
Because it trows the error:

/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In member function 'onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const':
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:167:3: error: there are no arguments to 'ORT_UNUSED_VARIABLE' that depend on a template parameter, so a declaration of 'ORT_UNUSED_VARIABLE' must be available [-fpermissive]
  167 |   ORT_UNUSED_VARIABLE(is_mask_1d_key_seq_len_start);
      |   ^~~~~~~~~~~~~~~~~~~
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:167:3: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of 'onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = onnxruntime::MLFloat16]':
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:167:22: error: 'ORT_UNUSED_VARIABLE' was not declared in this scope; did you mean 'HAS_UNUSED_VARIABLE'?
  167 |   ORT_UNUSED_VARIABLE(is_mask_1d_key_seq_len_start);
      |   ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |   HAS_UNUSED_VARIABLE
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc: In instantiation of 'onnxruntime::common::Status onnxruntime::contrib::cuda::Attention<T>::ComputeInternal(onnxruntime::OpKernelContext*) const [with T = float]':
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.h:21:10:   required from here
/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc:167:22: error: 'ORT_UNUSED_VARIABLE' was not declared in this scope; did you mean 'HAS_UNUSED_VARIABLE'?
  167 |   ORT_UNUSED_VARIABLE(is_mask_1d_key_seq_len_start);
      |   ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |   HAS_UNUSED_VARIABLE
make[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:2084: CMakeFiles/onnxruntime_providers_cuda.dir/raid/home/nvidia/triton/onnxruntime/onnxruntime/contrib_ops/cuda/bert/attention.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....

to bypass the issue for Jetson I had to remove newly added line and build it with flag --compile_no_warning_as_error same as we address it above.

Seems all the time that I built it successfully, that commit wasn't a part of the in OnnxRuntime code.

tianleiwu · 2023-05-25T23:05:20Z

@mc-nv, it has been fixed in main branch by replacing ORT_UNUSED_VARIABLE with ORT_UNUSED_PARAMETER.

snnn · 2023-05-26T05:19:15Z

I will cherry-pick the change to the release branch when we do a patch release.

See microsoft#16000 (comment) for details

mc-nv added the build build issues; typically submitted using template label May 18, 2023

mc-nv changed the title ~~[Build]~~ [Build] Unable to suppress unused variable May 18, 2023

jywu-msft mentioned this issue May 18, 2023

fix unused var warning in contrib_ops/cuda/bert/attention.cc #16010

Merged

jywu-msft closed this as completed in #16010 May 19, 2023

jywu-msft added a commit that referenced this issue May 19, 2023

fix unused var warning in contrib_ops/cuda/bert/attention.cc (#16010)

a74fdeb

fix #16000

snnn pushed a commit that referenced this issue May 19, 2023

fix unused var warning in contrib_ops/cuda/bert/attention.cc (#16010)

f535d5b

fix #16000

snnn mentioned this issue Jun 13, 2023

Build onnxruntime on arm64 linux with CUDA EP #16263

Closed

traversaro mentioned this issue Jun 27, 2023

Enable CUDA (take two) conda-forge/onnxruntime-feedstock#63

Merged

5 tasks

traversaro added a commit to traversaro/onnxruntime that referenced this issue Jun 27, 2023

Fix compilation error due to missing ORT_UNUSED_VARIABLE definition

6f10118

See microsoft#16000 (comment) for details

traversaro mentioned this issue Aug 8, 2023

Enable cuda feature in onnxruntime package iit-danieli-joint-lab/idjl-vision-dependencies-vcpkg#5

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] Unable to suppress unused variable #16000

[Build] Unable to suppress unused variable #16000

mc-nv commented May 18, 2023

mc-nv commented May 18, 2023

snnn commented May 18, 2023

mc-nv commented May 18, 2023

snnn commented May 18, 2023

jywu-msft commented May 18, 2023

jywu-msft commented May 18, 2023

snnn commented May 18, 2023

mc-nv commented May 18, 2023 •

edited

Loading

jywu-msft commented May 18, 2023

mc-nv commented May 18, 2023

jywu-msft commented May 18, 2023 •

edited

Loading

mc-nv commented May 18, 2023

jywu-msft commented May 18, 2023

mc-nv commented May 18, 2023

mc-nv commented May 19, 2023 •

edited

Loading

snnn commented May 20, 2023

mc-nv commented May 20, 2023 •

edited

Loading

mc-nv commented May 25, 2023

tianleiwu commented May 25, 2023

snnn commented May 26, 2023

[Build] Unable to suppress unused variable #16000

[Build] Unable to suppress unused variable #16000

Comments

mc-nv commented May 18, 2023

Describe the issue

Urgency

Target platform

Build script

Error / output

Visual Studio Version

GCC / Compiler Version

mc-nv commented May 18, 2023

snnn commented May 18, 2023

mc-nv commented May 18, 2023

snnn commented May 18, 2023

jywu-msft commented May 18, 2023

jywu-msft commented May 18, 2023

snnn commented May 18, 2023

mc-nv commented May 18, 2023 • edited Loading

jywu-msft commented May 18, 2023

mc-nv commented May 18, 2023

jywu-msft commented May 18, 2023 • edited Loading

mc-nv commented May 18, 2023

jywu-msft commented May 18, 2023

mc-nv commented May 18, 2023

mc-nv commented May 19, 2023 • edited Loading

snnn commented May 20, 2023

mc-nv commented May 20, 2023 • edited Loading

mc-nv commented May 25, 2023

tianleiwu commented May 25, 2023

snnn commented May 26, 2023

mc-nv commented May 18, 2023 •

edited

Loading

jywu-msft commented May 18, 2023 •

edited

Loading

mc-nv commented May 19, 2023 •

edited

Loading

mc-nv commented May 20, 2023 •

edited

Loading