VPI_ERROR_OUT_OF_MEMORY: Not enough space for resource allocation

[poett1]

Hello,

after running isaac_ros_image_proc or isaac_ros::image_proc::RectifyNode for a few minutes, it seems to overflow the GPU Memory. In this case, the following error message is shown:

[component_container-2] [ERROR] [1645635774.159060082] [rectify_node]: Error while rectifying image: /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/src/rectify_node.cpp:305: VPI_ERROR_OUT_OF_MEMORY: Not enough space for resource allocation

It is running inside a container created by the latest isaac_ros_common and runs on amd64 with a Titan RTX, Driver Version: 470.103.01 and CUDA Version: 11.4 on host OS: Ubuntu 20.04 LTS with Docker 20.10.10. I could also provide some logs, but i don't know which additional information would be helpful.

Kind regards,
Patrick

[hemalshahNV]

Thanks for the heads up, Patrick. We haven't run into out-of-memory issues with overnight stress tests ourselves yet. Could it be something else eating up GPU memory elsewhere and squeezing the Rectify node? The general architecture of the graph you're running especially if anything else is running on GPU and the size/frequency of the images you're pumping through would be helpful to know to dig in. There is a newer version of this library here you may want to try out as well: https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline

[poett1]

Aaaah, I'm sorry. Actually I'm using the repo from nvidia-isaac-ros, but this is the first to show up in google. Should i rather post this issue there?
No, currently only the isaac_ros::image_proc::RectifyNode is running on the GPU and the total used memory usage is increasing by 128 MiB about once a minute, but the memory usage of the component container itself is not increasing.

[hemalshahNV]

It would help me track the issue better if you posted there but up to you. What does the output of nvidia-smi say on your x86 machine before running?

[poett1]

I tried updating the Nvidia driver to 510 today, unfortunately that did not solve the problem.
This is the Output of nvidia-smi before running the node:

I also tried isaac_ros::image_proc::RectifyNode on a Jetson Xavier NX and it throws a different error message there:
[ERROR] [1645883427.704355052] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8)

And i don't understand where this depth image message is coming from since it just gets a 720p monocular rgb camera input.

The rectifying node is started like this on both machines:

  rectify_node = ComposableNode(
       package='isaac_ros_image_proc',
       plugin='isaac_ros::image_proc::RectifyNode',
       name='rectify_node',
   )

   rectify_container = ComposableNodeContainer(
       name='rectify_container',
       package='rclcpp_components',
       executable='component_container',
       composable_node_descriptions=[rectify_node],
       output='screen',
   )

In addition, i could provide a ros workspace (e.g. as a github repo) with a gstreamer testsource as camera input and the full launchfile if that would help to solve these problems.