the importance of silhouette loss

[Wang-Tianyu]

I went through the pytorch3d tech report and found that the loss function for mesh reconstruction with 2D supervision includes a silhouette loss in addition to RGB reconstruction loss, which is also employed by most differentiable mesh rendering papers.
One exception is
Learning single-image 3D reconstruction by generative modeling of shape, pose, and shading https://arxiv.org/abs/1901.06447
where silhouettes are not required.
However, there is only TensorFlow implementation available.

So my question is why silhouette loss is so important and can current pytorch3d implementation work without silhouette loss in a mesh reconstruction task, i.e. with RGB L2 loss only?

Thanks!

[gkioxari]

There are many points to address here.

a) I haven't read the paper you cite. The authors of the paper don't show what their images are. Do their images have RGB texture or are they actually silhouette-like? They say they use ShapeNet but do they use the textures? Or just renderings of the shape without texture? They certainly don't seem to predict textures. If they don't use textures, and say render the objects with white texture and Gouraud shading, then the notion between RGB texture and silhouette is not that far apart.

b) The silhouette loss is very important because the texture of an object can vary while the shape of it is very distinctive. Using silhouettes makes training more robust and models converge faster, and that's expected because object silhouettes are a good abstraction when capturing object priors. While texture can vary (e.g. color and color patterns of a car) the shape is very distinctive.

c) When wanting to answer such questions, i.e. how important is X vs Y, it's important to ablate everything under the same conditions. It's almost impossible, and that's more pronounced in 3D papers, to draw conclusions when comparing different papers. The conditions under which different authors do experiments can influence the results quite a bit and if these conditions are not revealed properly that makes drawing conclusions impossible. And here the dataset used is an important factor. And it is why in our tech report we chose to use R2N2, which is a fixed dataset from ShapeNet and we didn't do any processing of it whatsoever.