Got 'cutlass error' when running script or testbed

[yitongx]

Hi everyone,

I ran python -u scripts/run.py --mode nerf --scene data/nerf/fox on a TITAN X with CUDA=11.2 and got an 'curlass error' as follow:

Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
15:28:12 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
15:28:12 INFO     Color model:   3--[SphericalHarmonics]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
15:28:12 INFO       total_encoding_params=13074912 total_network_params=9728
Training:   0%|                                                                                                                                                  | 0/100000 [00:00<?, ?step/s]
Got cutlass error: Error Internal at: 394

The error occurs at line 394 in /dependencies/tiny-cuda-nn/include/tiny-cuda-nn/cutlass_matmul.h when launching initialized CUTLASS kernel. Using the testbed also produces the same problem.

I would like to know whether this has something to do with the CUTLASS MLP and have you met the same problem on the GPU architecture under 70. Many thanks in advance.

[Tom94]

Is this the Pascal or Maxwell version of the TITAN X? (There are two GPUs with this name.)

In case you are on the Maxwell TITAN X you'll have to use TCNN_CUDA_ARCHITECTURES=52 rather than 61.

(See https://developer.nvidia.com/cuda-gpus)

[yitongx]

Thanks for your instant reply. The problem was caused by the GPU assignment in the cluster (which will assign TITANs of different architecture).

The problem has been solved. Now the sdf,image and volume mode can be run successfully, except that nerf fox reported OutOfMemory. The fox issue was also mentioned in #36.

[Tom94]

Glad to hear!

I'll take this as motivation to add a runtime check for whether the currently operating GPU has lower compute capability than what the framework was compiled for.

[nikhilmakkar]

@yitongx @Tom94 How did you solve this error. I am facing the same problem. I am using Windows 10 and old GPU (1080).