Setting near_far when working with ScanNet scenes

[yashbhalgat]

Hi @apchenstu, I am trying to train TensoRF with a ScanNet scene. I calculate the scene bounding box using the scene mesh provided in the dataset. For example, for scene0000_00 in ScanNet, the bounding box can be computed to be [[-1.0176, -1.0018, -1.0003],[11.3742, 9.7380, 4.0293]]. Although, the TensoRF performance on the test set is quite bad. The visualizations look something like this:

You can see that it can render the chair legs (see bottom left), but overall rendering is quite bad. I tried different far values (e.g. 5.0, 10.0, 100.0) but none of them seem to work with TensoRF.

But when I trained a NeRF model with Instant-NGP, far value of 10.0 had worked. So, I am not sure what I am missing here. Can you please advise?

Interestingly, training PSNR reaches a high value of 25, but test PSNR is very low around 9 or 10.

Thanks,
Yash

[apchenstu]

It looks like something is wrong with the camera poses, please note that the coordinate system in tensorf is different from the iNGP, I would suggest you double-check the extrinsic poses.

[yashbhalgat]

Thank you for the prompt response!
Yes, I do realize that TensoRF uses the OpenCV convention, which is also the default convention used by ScanNet. That is why I set the blender2opencv matrix to Identity in the dataloader. So, I am not able to figure what could be wrong

[bo233]

Hi, @yashbhalgat, have you solved this problem? I try to test TensoRF on ScanNet too, but the result looks not so good. I think there are some problems with my coordinate system, bounding box parameter or near_far parameter. And how do you get the bounding box parameter on ScanNet Dataset?

Thank you!

[yashbhalgat]

Hi, yes, I was able to get it working. You can have a look at this repo which uses TensoRF representation with ScanNet. Hope this helps.

[emjay73]

@yashbhalgat Can you share what was the key element to make it work?

[ChenLu-china]

Is there any solution? I have the same problem and got a set of blur test results.

[yangbaoquan]

Hi, yes, I was able to get it working. You can have a look at this repo which uses TensoRF representation with ScanNet. Hope this helps.

Hello, I have met the same problem and have no idea about the solution. Could you please explain the settings more clearly? Thanks a lot!

[ChenLu-china]

Hi, yes, I was able to get it working. You can have a look at this repo which uses TensoRF representation with ScanNet. Hope this helps.

Hello, I have met the same problem and have no idea about the solution. Could you please explain the settings more clearly? Thanks a lot!
I have solved this problem by change the initial learning rate and decay.

[cmh1027]

It seems the problem can be solved by reducing learning rate