-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why using NeRF to model background? #107
Comments
Hi, @AIBluefisher! In the NeuS system, the foreground is modeled as a signed distance field, but the background is often modeled as a volumetric part, so we need a separate NeRF. In other words, the two NeRFs actually model different things (surface for foreground and volume for background). But it's worth mentioning that in BakedSDF the authors model the whole scene (foreground + background) with SDF but I think it could be harder to converge than using volume to model the background. |
Hi, @bennyguo Thanks for your reply! |
Although NeuS still uses volume rendering, it models the signed distance field which is restricted by the eikonal constraint. This makes it much harder to optimize than NeRF (no constraint at all). |
It is a valid point. Actually, I tried to model NeuS without nerf for backgrounds on unbounded scenes, but the reconstruction quality is really poor. |
Hi, since we already have the foreground point cloud, sphere radius, or mask, each of which can define the region we want to reconstruct, why is it necessary to model the background as well? Thanks! |
Because masks are not always available. If we already have masks, it is not necessary to use a background model. |
Hi @AIBluefisher , why it is still necessary to model the background when we have foreground point cloud or sphere radius to tell the ROI. |
Theoretically speaking, it is unnecessary to model the background if we have the foreground/background masks. If we have a foreground point cloud, say it is dense enough, we can actually obtain the foreground masks by projecting the point clouds to the image plane. However, if we have only sphere radius, we have to model the foreground in the sphere and also the background that is out of the sphere - because we do not know which pixels should be modeled inside/outside the sphere. |
Hi, @bennyguo
I wonder why you are using another NeRF to model backgrounds for neural reconstruction. In the original NeRF paper, I think the designation of using NeRF++ to model background is due to it being an earlier work and at that time there are not many works solving for the unbounded scenes. But since we are using NeRFAcc, we can directly model unbounded scenes in a single model.
The text was updated successfully, but these errors were encountered: