runtime error: cudaMemcpyToSymbol failed for Gauss kernel initialization

[hhackbarth]

I have compiled the current version on an NVidia Jetson Orin NX (Jetpack 6, Ubuntu 22.04, CUDA 12.2).
Demo program throws runtime error, aborts (core dumped)

To Reproduce
Steps to reproduce the behavior:

1. Compile / build as described
2. Execute demo application with ./build/Linux-aarch64/popsift-demo --input-file /path/to/your/test-image.jpg
3. See error:

PopSift version: 1.0.0
/path/to/your/test-image.jpg
terminate called after throwing an instance of 'std::runtime_error'
what(): /home/user/popsift/src/popsift/gauss_filter.cu:245
cudaMemcpyToSymbol failed for Gauss kernel initialization: invalid device symbol

Aborted (core dumped)

Expected behavior
Normal output of popsift-demo program.

Desktop (please complete the following and other pertinent information):

• OS: Linux (Jetpack 6.0, Ubuntu 22.04, CUDA 12.2 on an NVidia Jetson Orin NX)
• PopSift version: current version (cloned 2024.08.10, develop tag)
  • Commit reference bdd37e5

[griwodz]

The last commit changed to use CMake's direct support for CUDA as a first-order language instead of the manual hacks that we have used before. Is it possible that CMake makes bad choices for Tegra and Jetson?

Or could this be related to the following report about CUDA not initializing on Tegra:
https://gitlab.kitware.com/cmake/cmake/-/issues/25880 ?

[simogasp]

Isn't it because Jetson and Tegra are not supported? We had to remove the relevant CC before
#74 (comment)

[griwodz]

Well, CUDA on Jetson and Tegra does work. But the CCs are different from the other GPUs. They use 5.3, 6.2, 7.2 and 8.7.

With CMake's native support for CUDA, the variable CMAKE_CUDA_ARCHITECTURES has been introduced. It's possible to give this one an explicit list of CCs, or set it to "all", "all-major" or "native".

I set the default in CMakeLists.txt to "all-major" because that compiles even on computers that don't have a GPU but uses much less time than "all". The problem may be bigger, but I suspect that "all-major" leads to a selection of CC 8.0, while the Jetson Orin NX requires 8.7.

[griwodz]

Hi @hhackbarth ,
Could you try the branch dev/cmake-native-cuda-jetson (PR #163)?
CMake without any parameters works on our AGX Xavier / Ubuntu 20.04 / CUDA 11.4 now.
I would hope that it works for your configuration as well.

[griwodz]

The PR #163 checks if the CMake configuration happens on a computer that has the file /etc/nv_tegra_release. If the file exists, it configures compilation for CCs 5.3, 6.2, 7.2, and 8.7 instead of the default "all-major".

[hhackbarth]

Thank you for your quick reply and PR.

Could you try the branch dev/cmake-native-cuda-jetson (PR #163)? CMake without any parameters works on our AGX Xavier / Ubuntu 20.04 / CUDA 11.4 now. I would hope that it works for your configuration as well.

I can confirm, that - based on your branch - the popsift-demo example now runs without error on my Jetson Orin NX with JetPack 6.0 (Ubuntu 22.04, CUDA 12.2, architecture 87).

cmake shows:
-- CMAKE_CUDA_ARCHITECTURES = 53;62;72;87

So far, so good. But when I try to use it with the pypopsift wrapper from https://github.com/OpenDroneMap/pypopsift and an adapted CMakelists.txt file, also pypopsift builds successfully and can be installed (building wheels) using pip install . in the pypopsift folder. But when I run their example, I still get Segmentation fault (core dumped).

In the CMakelists.txt file I adapted the FetchContent_declare() part to:

FetchContent_Declare(
  popsift
  GIT_REPOSITORY https://github.com/alicevision/popsift/
  GIT_TAG        dev/cmake-native-cuda-jetson
)

This may be an issue related to pypopsift, but as they are only a wrapper, perhaps the popsift API somehow changed meanwhile and you already have an idea, how to fix that or tell me, what I am doing wrong?

[griwodz]

I haven’t tried pypopsift, and I will run out of time if I try to dig into it. It fails on Jetson, right?

The changes were not to the API, they were meant for the CMake config, but a bit of code was too outdated.

From a quick look, I can only imagine that the max number of features filter fails.