Add Initial Cross-Compilation Documentation for RISC-V Support #19239

phoebesv · 2024-01-23T09:36:07Z

Update the document for PR: Enable Cross-Compiling Support for ONNX Runtime on Linux for RISC-V 64-bit.
#19238

#19238) ### Description This pull request introduces the necessary changes to enable RISC-V 64-bit cross-compiling support for the ONNX Runtime on Linux. The RISC-V architecture has gained popularity as an open standard instruction set architecture, and this contribution aims to extend ONNX Runtime's compatibility to include RISC-V, thereby broadening the reach of ONNX models to a wider range of devices. ### Motivation and Context RISC-V is a free and open-source instruction set architecture (ISA) based on established RISC principles. It is provided under open licenses without fees. Due to its extensibility and freedom in both software and hardware, RISC-V is poised for widespread adoption in the future, especially in applications related to AI, parallel computing, and data centers. ### Example Build Command ``` ./build.sh --parallel --config Debug --rv64 --riscv_toolchain_root=/path/to/toolchain/root --skip_tests ``` ### Documentation Updates Relevant sections of the documentation will be updated to reflect the newly supported RISC-V 64-bit cross-compilation feature. #19239 --------- Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>

yihonglyu

@phoebesv I followed the instructions but encounter the error message

[ 51%] Building CXX object CMakeFiles/onnxruntime_providers_xnnpack.dir/home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc.o
/home/yilyu/risc-v/riscv/bin/riscv64-unknown-linux-gnu-g++ --sysroot=/home/yilyu/risc-v/riscv/sysroot -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DONNX_USE_LITE_PROTO=1 -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_XNNPACK=1 -D_GNU_SOURCE -D__ONNX_NO_DOC_STRINGS -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/utf8_range-src -I/home/yilyu/onnxruntime/include/onnxruntime -I/home/yilyu/onnxruntime/include/onnxruntime/core/session -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/google_nsync-src/public -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo -I/home/yilyu/onnxruntime/onnxruntime -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/abseil_cpp-src -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/safeint-src -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/gsl-src/include -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/onnx-src -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/onnx-build -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/protobuf-src/src -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/googlexnnpack-src/include -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/pthreadpool-src/include -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/flatbuffers-src/include -I/home/yilyu/onnxruntime/build/Linux/RelWithDebInfo/risc-v/RelWithDebInfo/_deps/mp11-src/include -DNDEBUG -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -O3 -pipe -ggdb3 -fstack-clash-protection -ffunction-sections -fdata-sections -Wno-error=attributes -Wno-restrict -O2 -g -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers_xnnpack.dir/home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc.o -MF CMakeFiles/onnxruntime_providers_xnnpack.dir/home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc.o.d -o CMakeFiles/onnxruntime_providers_xnnpack.dir/home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc.o -c /home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc
In file included from /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/string:51,
from /home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.h:6,
from /home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc:4:
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = const onnxruntime::Node* const; _Up = const onnxruntime::Node*; bool _IsMove = false]',
inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = const onnxruntime::Node* const*; _OI = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_algobase.h:506:30,
inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = const onnxruntime::Node* const*; _OI = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_algobase.h:533:42,
inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, vector<const onnxruntime::Node*> >; _OI = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_algobase.h:540:31,
inlined from '_OI std::copy(_II, _II, _OI) [with _II = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, vector<const onnxruntime::Node*> >; _OI = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_algobase.h:633:7,
inlined from 'static _ForwardIterator std::__uninitialized_copy::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, std::vector<const onnxruntime::Node*> >; _ForwardIterator = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_uninitialized.h:147:27,
inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, vector<const onnxruntime::Node*> >; _ForwardIterator = const onnxruntime::Node**]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_uninitialized.h:185:15,
inlined from '_ForwardIterator std::__uninitialized_copy_a(_InputIterator, _InputIterator, _ForwardIterator, allocator<_Tp>&) [with _InputIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, vector<const onnxruntime::Node*> >; _ForwardIterator = const onnxruntime::Node**; _Tp = const onnxruntime::Node*]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_uninitialized.h:373:37,
inlined from 'void std::vector<_Tp, _Alloc>::_M_range_insert(iterator, _ForwardIterator, _ForwardIterator, std::forward_iterator_tag) [with _ForwardIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, std::vector<const onnxruntime::Node*> >; _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/vector.tcc:814:38,
inlined from 'std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::insert(const_iterator, _InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, std::vector<const onnxruntime::Node*> >; = void; _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_vector.h:1483:19,
inlined from 'std::vector<const onnxruntime::Node*> onnxruntime::NodeUnit::GetAllNodesInGroup() const' at /home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc:271:19:
/home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' writing between 9 and 9223372036854775800 bytes into a region of size 0 overflows the destination [-Werror=stringop-overflow=]
437 | __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
| ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/riscv64-unknown-linux-gnu/bits/c++allocator.h:33,
from /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/allocator.h:46,
from /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/string:43:
In member function '_Tp* std::__new_allocator<_Tp>::allocate(size_type, const void*) [with _Tp = const onnxruntime::Node*]',
inlined from 'static _Tp* std::allocator_traits<std::allocator<_CharT> >::allocate(allocator_type&, size_type) [with _Tp = const onnxruntime::Node*]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/alloc_traits.h:482:28,
inlined from 'std::_Vector_base<_Tp, _Alloc>::pointer std::_Vector_base<_Tp, _Alloc>::_M_allocate(std::size_t) [with _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_vector.h:378:33,
inlined from 'std::_Vector_base<_Tp, _Alloc>::pointer std::_Vector_base<_Tp, _Alloc>::_M_allocate(std::size_t) [with _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_vector.h:375:7,
inlined from 'void std::vector<_Tp, _Alloc>::_M_range_insert(iterator, _ForwardIterator, _ForwardIterator, std::forward_iterator_tag) [with _ForwardIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, std::vector<const onnxruntime::Node*> >; _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/vector.tcc:805:40,
inlined from 'std::vector<_Tp, _Alloc>::iterator std::vector<_Tp, _Alloc>::insert(const_iterator, _InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<const onnxruntime::Node* const*, std::vector<const onnxruntime::Node*> >; = void; _Tp = const onnxruntime::Node*; _Alloc = std::allocator<const onnxruntime::Node*>]' at /home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/stl_vector.h:1483:19,
inlined from 'std::vector<const onnxruntime::Node*> onnxruntime::NodeUnit::GetAllNodesInGroup() const' at /home/yilyu/onnxruntime/onnxruntime/core/providers/shared/node_unit/node_unit.cc:271:19:
/home/yilyu/risc-v/riscv/riscv64-unknown-linux-gnu/include/c++/13.2.0/bits/new_allocator.h:147:55: note: at offset [-9223372036854775808, -1] into destination object of size [8, 9223372036854775800] allocated by 'operator new'
147 | return static_cast<_Tp*>(_GLIBCXX_OPERATOR_NEW(__n * sizeof(_Tp)));
| ^
cc1plus: all warnings being treated as errors

phoebesv · 2024-01-25T08:41:01Z

@yihonglyu
This seems to be a flattbuffer issue. Please see google/flatbuffers#8119
Using the default onnxruntime compiler option, I can successfully build ORT.
I also noticed someone set -Wno-error in the ORT flatbuffer.

onnxruntime/cmake/patches/flatbuffers/flatbuffers.patch

Line 10 in 2b87dd3

    
           +    "${CMAKE_CXX_FLAGS} -Wall -pedantic -Werror -Wextra -Werror=shadow -Wno-error=stringop-overflow")

Could you try using the default ORT compile option to build it?

phoebesv · 2024-01-25T09:43:51Z

I also found that there would be no flatbuffer issue by using config debug. I will update the documentation to use the Debug configuration until the FlatBuffers issue is resolved. I think this will provide a more stable build.

yihonglyu · 2024-01-25T21:14:30Z

@yihonglyu This seems to be a flattbuffer issue. Please see google/flatbuffers#8119 Using the default onnxruntime compiler option, I can successfully build ORT. I also noticed someone set -Wno-error in the ORT flatbuffer.

onnxruntime/cmake/patches/flatbuffers/flatbuffers.patch

Line 10 in 2b87dd3

+ "${CMAKE_CXX_FLAGS} -Wall -pedantic -Werror -Wextra -Werror=shadow -Wno-error=stringop-overflow")

Could you try using the default ORT compile option to build it?

That is really weird that it would work with Debug option but fail on other option. Could you possibly explain why this is happening?

phoebesv · 2024-01-26T03:06:40Z

That is really weird that it would work with Debug option but fail on other option. Could you possibly explain why this is happening?

The compiler optimization levels for Debug and RelWithDebInfo are different, so it is possible that this problem only affects Rel.

snnn · 2024-01-27T20:29:41Z

In general we do not treat warnings from third-party libraries as errors, since there is very little thing we can do. Therefore, we may remove "-Werror" from the patch file.

phoebesv · 2024-01-28T13:16:51Z

In general we do not treat warnings from third-party libraries as errors, since there is very little thing we can do. Therefore, we may remove "-Werror" from the patch file.

Thanks for informing me about this. I will submit a separate pull request to address the issue with the flatbuffer.

yihonglyu · 2024-01-30T05:09:50Z

@phoebesv does the PR #19238 only support the XNNPACK provider? From my understanding, it appears that it can be utilized even without XNNPACK. Could you please confirm?

phoebesv · 2024-01-30T05:18:10Z

@phoebesv does the PR #19238 only support the XNNPACK provider? From my understanding, it appears that it can be utilized even without XNNPACK. Could you please confirm?

xnnpack is optional, I added optional annotations in the document, hoping to provide a better understanding that xnnpack is not a mandatory option.

docs/build/inferencing.md

Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>

yihonglyu · 2024-01-30T12:14:25Z

@phoebesv, I’ve checkout out the branch at #19296 and attempted to build and test it using the command ./build.sh --parallel --config RelWithDebInfo --rv64 --riscv_toolchain_root=/home/yilyu/risc-v/riscv --riscv_qemu_path=/home/yilyu/risc-v/riscv/bin/qemu-riscv64 --build_dir /home/yilyu/onnxruntime/build/Linux/risc-v. However, numerous test cases failed. Is this an anticipated outcome? If that’s the case, do you have a subsequent PR planned to resolve the issue?

phoebesv · 2024-01-30T13:52:44Z

@yihonglyu Our initial RISC-V PR focused on adding cross-compilation support to ORT, not all tests are enabled yet, so we've included 'skip_tests' in the build command. Considering the large number of onnx test cases and the complexities of supporting RISC-V, an initial pull request cannot possibly cover every test scenario. By enabling cross-compilation, we hope to encourage contributions to gradually enhance RISC-V support for more scenarios.

### Description Building on g++ 13.2.0 results in -Wstringop-overread errors on Linux. This commit addresses the flatbuffer build issue with the following changes: 1. Remove the Werror flag in the flarbuffer patch. 2. Add a compilation option to suppress the 'stringop-overflow' error in the Flatbuffers within the xnnpack provider. ### Motivation and Context google/flatbuffers#8119 #19239 Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>

csukuangfj · 2024-02-26T03:45:19Z

Our initial RISC-V PR focused on adding cross-compilation support to ORT,

@phoebesv Thanks for supporting compiling onnxruntime for riscv64.

I have successfully compiled onnxruntime for riscv64.
However, when I use it to run my speech recognition model, the recognition results are wrong compared with using x86_64 onnxruntime.

Is that expected? Does it only support cross-compiling onnxruntime for riscv64 but the correctness has not been tested yet?

phoebesv · 2024-02-26T11:10:15Z

@phoebesv Thanks for supporting compiling onnxruntime for riscv64.

I have successfully compiled onnxruntime for riscv64. However, when I use it to run my speech recognition model, the recognition results are wrong compared with using x86_64 onnxruntime.

Is that expected? Does it only support cross-compiling onnxruntime for riscv64 but the correctness has not been tested yet?

@csukuangfj Yes. We currently have test scalar code by UT onnxruntime_test_all on qemu (using xnnpack EP), with multiple failed test cases. Unfortunately, we do not have enough resources to clean up all of the failed test instances.
onnxruntime_test_all riscv testing results 3688 tests from 258 test suites ran.
Failed ratio (NUM_FAILED/NUM_PASSED)~= 0.068
The most of failed test cases are due to mismatches between expected and actual results. As far as I know, a few people are working on addressing it, you are welcome to contribute.

csukuangfj · 2024-02-26T11:20:20Z

using xnnpack EP

Is it possible to use the default CPU provider instead of the xnnpack provider?

csukuangfj · 2024-02-26T11:33:28Z

As far as I know, a few people are working on addressing it, you are welcome to contribute.

Is there a place to find the failed tests and find whom working on which failed test?

phoebesv mentioned this pull request Jan 23, 2024

Enable RISC-V 64-bit Cross-Compiling Support for ONNX Runtime on Linux #19238

Merged

yihonglyu reviewed Jan 25, 2024

View reviewed changes

phoebesv force-pushed the riscv_cross_compile/gh-pages branch from a6dc816 to 9e7be4e Compare January 25, 2024 09:48

phoebesv mentioned this pull request Jan 28, 2024

Fix Flatbuffer build issue. #19296

Merged

phoebesv force-pushed the riscv_cross_compile/gh-pages branch from 9e7be4e to 97f267d Compare January 30, 2024 06:08

phoebesv requested a review from yihonglyu January 30, 2024 06:09

phoebesv force-pushed the riscv_cross_compile/gh-pages branch from 97f267d to ff27131 Compare January 30, 2024 07:08

yihonglyu reviewed Jan 30, 2024

View reviewed changes

docs/build/inferencing.md Outdated Show resolved Hide resolved

Add the initial cross-compilation documentation for RISC-V.

db06b1a

Signed-off-by: Phoebe Chen <phoebe.chen@sifive.com>

phoebesv force-pushed the riscv_cross_compile/gh-pages branch from ff27131 to db06b1a Compare January 30, 2024 11:38

tianleiwu requested a review from yihonglyu April 5, 2024 13:50

tianleiwu approved these changes Apr 5, 2024

View reviewed changes

tianleiwu merged commit e31928e into microsoft:gh-pages Apr 5, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Initial Cross-Compilation Documentation for RISC-V Support #19239

Add Initial Cross-Compilation Documentation for RISC-V Support #19239

phoebesv commented Jan 23, 2024 •

edited

Loading

yihonglyu left a comment

phoebesv commented Jan 25, 2024

phoebesv commented Jan 25, 2024

yihonglyu commented Jan 25, 2024

phoebesv commented Jan 26, 2024

snnn commented Jan 27, 2024

phoebesv commented Jan 28, 2024 •

edited

Loading

yihonglyu commented Jan 30, 2024

phoebesv commented Jan 30, 2024 •

edited

Loading

yihonglyu commented Jan 30, 2024 •

edited

Loading

phoebesv commented Jan 30, 2024

csukuangfj commented Feb 26, 2024

phoebesv commented Feb 26, 2024 •

edited

Loading

csukuangfj commented Feb 26, 2024

csukuangfj commented Feb 26, 2024

Add Initial Cross-Compilation Documentation for RISC-V Support #19239

Add Initial Cross-Compilation Documentation for RISC-V Support #19239

Conversation

phoebesv commented Jan 23, 2024 • edited Loading

yihonglyu left a comment

Choose a reason for hiding this comment

phoebesv commented Jan 25, 2024

phoebesv commented Jan 25, 2024

yihonglyu commented Jan 25, 2024

phoebesv commented Jan 26, 2024

snnn commented Jan 27, 2024

phoebesv commented Jan 28, 2024 • edited Loading

yihonglyu commented Jan 30, 2024

phoebesv commented Jan 30, 2024 • edited Loading

yihonglyu commented Jan 30, 2024 • edited Loading

phoebesv commented Jan 30, 2024

csukuangfj commented Feb 26, 2024

phoebesv commented Feb 26, 2024 • edited Loading

csukuangfj commented Feb 26, 2024

csukuangfj commented Feb 26, 2024

phoebesv commented Jan 23, 2024 •

edited

Loading

phoebesv commented Jan 28, 2024 •

edited

Loading

phoebesv commented Jan 30, 2024 •

edited

Loading

yihonglyu commented Jan 30, 2024 •

edited

Loading

phoebesv commented Feb 26, 2024 •

edited

Loading