Getting UVs from sampled surface points of a mesh

[joeylitalien]

❓ Getting UVs from sampled surface points

I'm using ops.sample_points_from_meshes() to sample a point cloud on a mesh surface, along with the flag return_textures=True to get the texture map RGB values for the points (assuming the mesh is properly UV-mapped). This works great but I am wondering if it's possible to also retrieve the UV coordinates of the sampled points. I've looked at the implementation of the sampling function but I am not sure if there's a simple way to hack it, unless this can be done using TexturesUV.sample_textures(). Surely this is implicitly done under the hood to get a colored point cloud, so I am wondering if this is something that can be easily exposed to the user.

Thanks for all the great work.

[bottler]

Here's a fairly easy way to do get this done with the current code. Make a new TexturesUV with two more colors, U and V (e.g. 5 colors if you were using RGB - so you concatenate more values on to the maps) and sample your mesh with its textures replaced by it. Then the textures output you get from sampling points will have both color and UV values for each point.

If you have align_corners being True, then the added maps you need should be torch.linspace(0, 1, W)[None, : None].expand(H, W, 1) and torch.linspace(1, 0, H)[:, None, None].expand(H, W, 1). Otherwise, the same with arguments to linspace replaced with (1.0 / (2 * W), 1 - 1.0 / (2 * W), W) and (1 - 1.0 / (2 * H), 1.0 / (2 * H), H).

[joeylitalien]

Thanks for the quick reply! This looks like complete sorcery to me, but hopefully we can make it work. Is it enough to simply edit the texture map for the mesh, or do I need to clone and replace it? The following code snippet does sample without error, but the UVs are left at zeros. I guess I am doing something wrong or not using the right atrribute for the map (_maps_padded).

mesh = load_objs_as_meshes([obj_fname], load_textures=True)
uv_extra = torch.zeros((*mesh.textures._maps_padded.shape[:-1], 2))
mesh.textures._maps_padded = torch.cat([mesh.textures._maps_padded, uv_extra], dim=3)

n = 10
pts, tex_uv = sample_points_from_meshes(mesh, n, return_textures=True)

Also, I am not quite sure what you mean in your second paragraph; I guess you are describing how to get [1,W]x[1,H] pixel values for the textures?

[bottler]

It's safer in general to create a new TexturesUV object, and then a new Meshes object, because it's easy for things to go wrong elsewhere if you change things in place (especially things which are being optimized). If you never change the map and you never use maps_list, though, then yes, changing _maps_padded as you have done will probably work.

Your problem is that the uv_extra you are adding in that snippet is full of zeros - i.e. the new colors are all dark, which is why you get zeros out. You need to actually make the new colors be equal to u and v, and that's what linspace does. It's like lighting a long room such that brightness is proportional to distance from one edge, and then someone can measure their distance from the edge using a light meter.

You want something like

mesh = load_objs_as_meshes([obj_fname], load_textures=True)
N,H,W, C = mesh.textures._maps_padded.shape
assert N==1 
u_extra = torch.linspace(0, 1, W)[:, None].expand(1, H, W, 1) 
v_extra = torch.linspace(1, 0, H)[:, None, None].expand(1, H, W, 1).
mesh.textures._maps_padded = torch.cat([mesh.textures._maps_padded, u_extra, v_extra], dim=3)

[joeylitalien]

This is fantastic, major props for figuring this out so quickly. I've cross-validated this with grid_sample() to check that the texture values were indeed the same. Solved and closed! 😄