【2.0RC】F.grid_sample在特定条件下造成有cuda异常错误

[leeacord]

• 版本、环境信息：
     1）PaddlePaddle版本：2.0.0rc
     2）CPU：aistudio-gpu
     3）GPU：aistudio-gpu
     4）系统环境：aistudio
• 现象：
  错误仅在在gpu动态图下发生，cpu时未发生错误：
  F.grid_sample在特定条件下执行后，函数可以正常执行，但之后对任何变量进行astype()操作均会报错(可能运行其他op也会出错)。
  经检查后发现，gpu下只要这段代码中的use_cudnn=False，则会出现错误。
  具体来说，只有F.grid_sample(..., align_corners=True, mode='bilinear', padding_mode='zeros')才不会报错。
  错误发生时内存和显存占用极低，令人疑惑。

Paddle/python/paddle/nn/functional/vision.py

Lines 305 to 328 in 530deb1

	use_cudnn = False
	if (cudnn_version is not None
	) and align_corners and mode == 'bilinear' and padding_mode == 'zeros':
	use_cudnn = True
	ipts = {'X': x, 'Grid': grid}
	attrs = {
	'mode': mode,
	'padding_mode': padding_mode,
	'align_corners': align_corners,
	'use_cudnn': use_cudnn
	}
	if in_dygraph_mode():
	attrs = ('mode', mode, 'padding_mode', padding_mode, 'align_corners',
	align_corners, 'use_cudnn', use_cudnn)
	out = getattr(core.ops, 'grid_sampler')(x, grid, *attrs)
	else:
	out = helper.create_variable_for_type_inference(x.dtype)
	helper.append_op(
	type='grid_sampler',
	inputs=ipts,
	attrs=attrs,
	outputs={'Output': out})
	return out

• aistudio上复现

import paddle
paddle.set_device("gpu")
a = paddle.rand((10, 3, 256, 256))
b = paddle.rand((10, 256, 256, 2))
c = paddle.rand((2, 2))
print(c.astype('float32'))
d = paddle.nn.functional.grid_sample(a, b, mode='bilinear', padding_mode='zeros', align_corners=True)
print(c.astype('float32')) # 此时可正常输出
# 若此时进入函数，在320行前令use_cudnn=True，则无报错
d = paddle.nn.functional.grid_sample(a, b, mode='bilinear', padding_mode='reflection', align_corners=True)
print(c.astype('float32')) # 此时报错

• 报错提示：

--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::imperative::Tracer::TraceOp(std::string const&, paddle::imperative::NameVarBaseMap const&, paddle::imperative::NameVarBaseMap const&, paddle::framework::AttributeMap)
1   paddle::platform::EnforceNotMet::EnforceNotMet(paddle::platform::ErrorSummary const&, char const*, int)
2   paddle::platform::GetCurrentTraceBackString[abi:cxx11]()

----------------------
Error Message Summary:
----------------------
FatalError: Operator cast raises an thrust::system::system_error exception.
The exception content is
:parallel_for failed: cudaErrorInvalidConfiguration: invalid configuration argument. (at /paddle/paddle/fluid/imperative/tracer.cc:89)

[yinhaofeng]

你好，感谢反馈。已转交给内部相关同学

[paddle-bot-old]

Are you satisfied with the resolution of your issue?

YES
No