RuntimeError: CUDA error: an illegal memory access was encountered

[AnudhinaDhanabalan]

File "/home/arrstudios/Desktop/AI/3DGStream/3DGStream/gaussian_renderer/init.py", line 100, in render
"visibility_filter" : radii > 0,
RuntimeError: CUDA error: an illegal memory access was encountered

I am facing this error while training my own dataset, this error occurs after the training of the first frame !!
thanks in advance !!

[gaoyuchen2000]

I assume it is because the GPU VRAM isn't enough to satisfy the requirements. What is the GPU you are using?

[AnudhinaDhanabalan]

I am using nvidia rtx 6000 ada generation

[WuJH2001]

I assume it is because the GPU VRAM isn't enough to satisfy the requirements. What is the GPU you are using?

I am also happended to this error. I guess it may be the error of TCNN

[MarcWangzhiru]

I recommend that you choose the frame where all the moving objects appear as the initial frame to initialize the 3D Gaussian distribution. I also found this problem in the process of the experiment, and later I found that when I initialized the first frame, the hand of the dynamic moving person was not present in the whole scene, but appeared in the later frames. This can lead to gradient explosion later in the optimization process. I then initialized it by picking a frame of the full dynamic scene, and the problem was solved. I suspect this is an inherent problem with 3DGStream. Can you choose the frames where the moving object appears completely as the first frame to train? I hope to test my idea.

[pengcanon]

I also have this problem but all moving objects are present in my scenes. Some of them may be absent from a few number of camera views in the initial frame but captured in other camera views in the same frame so that the initial GSs contain all moving parts. I wonder if someone else has better insight into this problem? or has better luck dealing with this problem? It definitely suggests there are some gradient explosion or NaN values occuring during the training.