Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linux gpu编译错误 #40394

Closed
wuwuwuxxx opened this issue Mar 10, 2022 · 7 comments
Closed

linux gpu编译错误 #40394

wuwuwuxxx opened this issue Mar 10, 2022 · 7 comments
Assignees

Comments

@wuwuwuxxx
Copy link

- 版本、环境信息:
Paddle version: 2.2.2
Paddle With CUDA: True

OS: Ubuntu 18.04
Python version: 3.7.12

CUDA version: 11.2.142
Build cuda_11.2.r11.2/compiler.29558016_0
cuDNN version: None.None.None
Nvidia driver version: 510.47.03

- 安装方式信息:
compiled with docker paddlepaddle/paddle:2.2.2-gpu-cuda11.2-cudnn8

cmake .. -DPY_VERSION=3.7 -DWITH_GPU=ON -DWITH_TESTING=OFF -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=1

[ 54%] Generating libdnnl.so.0
[ 54%] copy_if_different /home/Paddle/paddle/fluid/pybind/op_function_impl.h.tmp to /home/Paddle/paddle/fluid/pybind/op_function_impl.h
op_function_generator: /home/Paddle/paddle/utils/small_vector.h:334: T& paddle::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::at(paddle::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::size_type) [with T = phi::TensorArgDef; <template-parameter-1-2> = void; paddle::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::reference = phi::TensorArgDef&; paddle::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::size_type = long unsigned int]: Assertion `idx < size()' failed.
Child aborted
paddle/fluid/pybind/CMakeFiles/op_function_generator_cmd.dir/build.make:62: recipe for target '../paddle/fluid/pybind/op_function_impl.h' failed
make[2]: *** [../paddle/fluid/pybind/op_function_impl.h] Error 1
CMakeFiles/Makefile2:512240: recipe for target 'paddle/fluid/pybind/CMakeFiles/op_function_generator_cmd.dir/all' failed
make[1]: *** [paddle/fluid/pybind/CMakeFiles/op_function_generator_cmd.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

it seems that op_function_generator failed. Here is the stack

#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007f5f3dc3b7f1 in __GI_abort () at abort.c:79 #2 0x00007f5f3dc2b3fa in __assert_fail_base (fmt=0x7f5f3ddb26c0 "%s%s%s:%u: %s%sAssertion %s' failed.\n%n", assertion=assertion@entry=0xd4e71f7 "idx < size()", file=file@entry=0xd4e6890 "/home/Paddle/paddle/utils/small_vector.h", line=line@entry=334,
function=function@entry=0xd4e7aa0 <paddle::SmallVectorTemplateCommon<phi::TensorArgDef, void>::at(unsigned long)::PRETTY_FUNCTION> "T& paddle::SmallVectorTemplateCommon<T, >::at(paddle::SmallVectorTemplateCommon<T, >::size_type) [with T = phi::TensorArgDef; "...) at assert.c:92
#3 0x00007f5f3dc2b472 in __GI___assert_fail (assertion=0xd4e71f7 "idx < size()", file=0xd4e6890 "/home/Paddle/paddle/utils/small_vector.h", line=334,
function=0xd4e7aa0 <paddle::SmallVectorTemplateCommon<phi::TensorArgDef, void>::at(unsigned long)::PRETTY_FUNCTION> "T& paddle::SmallVectorTemplateCommon<T, >::at(paddle::SmallVectorTemplateCommon<T, >::size_type) [with T = phi::TensorArgDef; "...) at assert.c:101
#4 0x0000000007e6a823 in paddle::SmallVectorTemplateCommon<phi::TensorArgDef, void>::at (this=0x7fff7b12dca8, idx=3) at /home/Paddle/paddle/utils/small_vector.h:334
#5 0x00000000080c4a02 in phi::Kernel::OutputAt (this=0x7fff7b12dc40, idx=3) at /home/Paddle/paddle/phi/core/kernel_factory.h:202
#6 0x000000000868660e in __PD_KERNEL_args_def_FN_batch_norm_grad_GPU_ALL_LAYOUT (kernel_key=..., kernel=0x7fff7b12dc40) at /home/Paddle/paddle/phi/kernels/gpu/batch_norm_grad_kernel.cu:1001
#7 0x00000000080c4d64 in phi::KernelRegistrar::ConstructKernel(phi::RegType, char const*, char const*, paddle::experimental::DataLayout, paddle::experimental::DataType, void ()(phi::KernelKey const&, phi::KernelArgsDef), void ()(phi::KernelKey const&, phi::Kernel), std::function<void (phi::KernelContext*)>, void*) (this=0x21b227da <__reg_pt_kernel_batch_norm_grad_GPU_ALL_LAYOUT_2>, reg_type=phi::RegType::INNER, kernel_name_cstr=0xd7f7ba5 "batch_norm_grad", backend_cstr=0xd7f461f "GPU",
layout=paddle::experimental::DataLayout::UNDEFINED, dtype=paddle::experimental::DataType::FLOAT16,
args_parse_fn=0x86994eb <phi::KernelArgsParseFunctor<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor, phi::DenseTensor*, phi::DenseTensor*)>::Parse(phi::KernelKey const&, phi::KernelArgsDef*)>, args_def_fn=0x868659b <__PD_KERNEL_args_def_FN_batch_norm_grad_GPU_ALL_LAYOUT(phi::KernelKey const&, phi::Kernel*)>, kernel_fn=...,
variadic_kernel_fn=0x869a095 <phi::KernelImpl<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor, phi::DenseTensor*, phi::DenseTensor*), &(void phi::BatchNormGradKernel<phi::dtype::float16, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*))>::VariadicCompute(phi::DeviceContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*)>) at /home/Paddle/paddle/phi/core/kernel_registry.h:209
#8 0x00000000080c4b78 in phi::KernelRegistrar::KernelRegistrar(phi::RegType, char const*, char const*, paddle::experimental::DataLayout, paddle::experimental::DataType, void ()(phi::KernelKey const&, phi::KernelArgsDef), void ()(phi::KernelKey const&, phi::Kernel), std::function<void (phi::KernelContext*)>, void*) (this=0x21b227da <__reg_pt_kernel_batch_norm_grad_GPU_ALL_LAYOUT_2>, reg_type=phi::RegType::INNER, kernel_name_cstr=0xd7f7ba5 "batch_norm_grad", backend_cstr=0xd7f461f "GPU",
layout=paddle::experimental::DataLayout::UNDEFINED, dtype=paddle::experimental::DataType::FLOAT16,
args_parse_fn=0x86994eb <phi::KernelArgsParseFunctor<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor, phi::DenseTensor*, phi::DenseTensor*)>::Parse(phi::KernelKey const&, phi::KernelArgsDef*)>, args_def_fn=0x868659b <__PD_KERNEL_args_def_FN_batch_norm_grad_GPU_ALL_LAYOUT(phi::KernelKey const&, phi::Kernel*)>, kernel_fn=...,
variadic_kernel_fn=0x869a095 <phi::KernelImpl<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor, phi::DenseTensor*, phi::DenseTensor*), &(void phi::BatchNormGradKernel<phi::dtype::float16, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*))>::VariadicCompute(phi::DeviceContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, paddle::optional<phi::DenseTensor const&>, float, float, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool, bool, bool, bool, phi::DenseTensor*, phi::DenseTensor*, phi::DenseTensor*)>) at /home/Paddle/paddle/phi/core/kernel_registry.h:153
#9 0x000000000868f70c in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/Paddle/paddle/phi/kernels/gpu/batch_norm_grad_kernel.cu:991
#10 0x000000000868fd4d in _GLOBAL__sub_I_tmpxft_00006371_00000000_6_batch_norm_grad_kernel.cudafe1.cpp(void) () at /tmp/tmpxft_00006371_00000000-6_batch_norm_grad_kernel.cudafe1.stub.c:228
#11 0x000000000a44fa6d in __libc_csu_init ()
#12 0x00007f5f3dc1cc18 in __libc_start_main (main=0x4a967d <main(int, char**)>, argc=1, argv=0x7fff7b12e398, init=0xa44fa20 <__libc_csu_init>, fini=, rtld_fini=, stack_end=0x7fff7b12e388) at ../csu/libc-start.c:266
#13 0x00000000004a5d9a in _start ()
`

@paddle-bot-old
Copy link

您好,我们已经收到了您的问题,会安排技术人员尽快解答您的问题,请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时,您也可以通过查看官网API文档常见问题历史IssueAI社区来寻求解答。祝您生活愉快~

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the APIFAQGithub Issue and AI community to get the answer.Have a nice day!

@smallv0221
Copy link
Contributor

您好,我们目前不支持debug模式编译哈

@smallv0221
Copy link
Contributor

可以尝试不使用debug编译看看能不能成功

@wuwuwuxxx
Copy link
Author

不用是可以的,cpu版本的debug模式也是可以的,就是gpu不可以。

@yang131313
Copy link
Contributor

你好,现在Paddle已经有编译好的安装包了,请问你在什么任务场景下需要编译Paddle呀?比如是需要新增算子、还是需要修改飞桨源代码,或者有什么其他的用途呢?

@felixhao28
Copy link

原来不支持debug编译呀,这下调试起来麻烦了。

对我来说,我是需要用SetModelBuffer从内存里加载模型文件,而不是模型文件路径。但如果这么用,再用上TensorRT的话,就会报错。我本来想调试一下,看看能不能把这个问题改掉。现在Debug和RelWithDebInfo都编译不了。

@paddle-bot paddle-bot bot closed this as completed Oct 3, 2023
@paddle-bot
Copy link

paddle-bot bot commented Oct 3, 2023

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复,我们将关闭这个issue/pr。
若问题未解决或有后续问题,请随时重新打开,我们会继续跟进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants