Issue with new diff_gaussian_rasterization

[svsambandam]

Hello!! I'm super excited to see this new version of D3DG. However, I noticed that there have been a few new changes to the renderer, (means2D_densify has been added to the rasterizer call) and ever since I cloned this current repo, or even the Vanilla D3DG code, I run into this error:

Traceback (most recent call last):
  File "/ubc/cs/research/kmyi/svsamban/mamba/envs/dd3dg/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)

  File "/ubc/cs/research/kmyi/svsamban/mamba/envs/dd3dg/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path
    pkg_name=pkg_name, script_name=fname)

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)

  File "/ubc/cs/home/s/svsamban/.vscode/extensions/ms-python.debugpy-2024.6.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)

  File "train.py", line 275, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations)

  File "train.py", line 107, in training
    Ll1 = l1_loss(image, gt_image)

  File "/ubc/cs/research/kmyi/svsamban/research/D3DG/Deformable-3D-Gaussians/utils/loss_utils.py", line 19, in l1_loss
    return torch.abs((network_output - gt)).mean()
RuntimeError: CUDA error: an illegal memory access was encountered

I notice it is because after render() is called, the image values are of magnitude 1e34. Are yu able to reproduce this error when downloading both repositories and building conda environments from scratch? How do I circumvent this problem?

Thanks in advance for your help :))