Noisy Renderings on LineMod Dataset

[manasburagohain]

If you do not know the root cause of the problem / bug, and wish someone to help you, please
post according to this template:

🐛 Bugs / Unexpected behaviors

Noisy rendering are generated on trying to render models from LineMod dataset. Observed noise acts as a function of distance between camera and the object - proportionally increases with distance. Noise density varies across
different camera angles and warps around the object render when xy translation is induced in camera position.

Renders at dist = 4

Renders at dist = 8

Renders at dist = 8, x_trans = 2 and y_trans = 2

Instructions To Reproduce the Issue:

Packages Used:
Torch
Pytorch3d
OpenCV
NumPy

1. Any changes you made (git diff) or code you wrote

import cv2
import numpy as np
import torch
from pytorch3d.structures import Meshes
from pytorch3d.io import IO, ply_io
from pytorch3d.renderer import (
    look_at_view_transform,
    look_at_rotation,
    OpenGLPerspectiveCameras,
    PerspectiveCameras, 
    PointLights, 
    DirectionalLights,
    AmbientLights,  
    RasterizationSettings, 
    MeshRenderer, 
    MeshRasterizer,  
    SoftPhongShader,
    SoftSilhouetteShader,
    SoftPhongShader,
    TexturesVertex
)

K = np.array([[572.4114, 0.,         325.2611],
              [0.,        573.57043,  242.04899],
              [0.,        0.,         1.]])

f_x, f_y = K[0,0], K[1,1]
p_x, p_y = K[0,2], K[1,2]
h = 480
w = 640

device = torch.device("cuda:0")


verts, face = ply_io.load_ply("lamp.ply")
verts_rgb = torch.from_numpy(np.load("textures.npy"))
textures = TexturesVertex(verts_features = verts_rgb)
mesh = Meshes(verts.unsqueeze(0), face.unsqueeze(0), textures).to(device)


elev = (360) * torch.rand(10)
azim = (360) * torch.rand(10) - 180

# elev= torch.tensor([219.2172, 196.6848,  26.7611, 241.7745, 288.8531, 347.6933, 150.2406,
#         289.7920, 105.1258, 207.1929])
# azim = torch.tensor([-152.0129,  -87.8176,  121.8020, -127.5354,  135.0699,  103.5086,
#         -162.2670,    4.3773,   77.4665, -100.1230])
print(elev)
print(azim)
lights = AmbientLights(device=device)

for idx in range(10):
    R, T = look_at_view_transform(dist= 8, elev=elev[idx], azim=azim[idx]) #+5*torch.randn(1)

    #Uncomment to induce x-y translation
    # T[:,:2] = 2*torch.ones(list(T.size()[:-1])+[2])
    f = torch.tensor((f_x, f_y), dtype=torch.float32).unsqueeze(0)
    p = torch.tensor((p_x, p_y), dtype=torch.float32).unsqueeze(0)
    img_size= torch.tensor((h, w), dtype=torch.float32).unsqueeze(0)
    
    camera = PerspectiveCameras(
        R=R, T=T,
        focal_length=f,
        principal_point=p,
            image_size=((h, w),),
            device=device,
            in_ndc=False)
        
    
    raster_settings = RasterizationSettings(
        image_size=(h,w), 
        blur_radius=0.0, 
        faces_per_pixel=100, 
    )

    renderer = MeshRenderer(
        rasterizer=MeshRasterizer(
            cameras=camera, 
            raster_settings=raster_settings
        ),
        shader=SoftPhongShader(
            device=device, 
            cameras=camera,
            lights=lights,
        )
    )

    target_images = renderer(mesh, cameras=camera, lights=lights, znear=0.0, zfar = 10000.0)

    
    img = target_images[0, ..., :3]#[ for i in range(num_views)]

    cv2.imwrite('test/'+str(idx)+'.jpg', img.cpu().numpy())

2. The exact command(s) you ran:
   python <file_name>.py

Files Used:
Mesh PLY: https://drive.google.com/file/d/18dNu-AjTdhTtF8Q9uzsBB4kRh5nfxGTa/view
Textures NPY: https://drive.google.com/file/d/1RY5ZmOegAQ97TitBQw6FFRsdMV0E2BdI/view?usp=sharing

[bottler]

Maybe not the whole problem, but the spotty bits in your images are a phenomenon we have seen a few times: an overflow in statically allocated bins in the coarse-to-fine rasterization. This is more likely to happen when the object is further away as the faces would be more concentrated. The explanation would be the same as #348. You can try to fix it by setting max_faces_per_bin in the RasterizationSettings to something large - e.g. something nearly as large as the number of faces in the mesh. Alternatively you can use the naive rasterizer, by adding bin_size=0 to the RasterizationSettings, but that is likely to be slower.

[manasburagohain]

Yes that fixed that!. Thanks!