How to add mask supervision for object 3d reconstruction ?

[Jiakui]

Dear author,

How to add mask supervision for object 3d reconstruction ?

Thanks

[yyfz]

Hi, thank you for your interest in our work.

Please refer to this link for guidance on how to read mask images in your dataparser. You can use this as a reference to incorporate the code snippet for reading masks into your own dataparser and set include_foreground_mask to True.

Additionally, in our model, you can find the original mask supervision code belonging to sdfstudio/nerfstudio at this location. Simply set fg_mask_loss_mult to a value greater than 0.

It's worth noting that we have not yet experimented with using masks for supervision. Therefore, if possible, it would be advisable to remove the mask portion from the unbias loss at this location, as this part of the unbias loss is meaningless.

In summary, once your dataparser is ready to read mask images and your model is prepared, you only need to make the following adjustments to your original startup script:

ns-train ... --pipeline.model.fg_mask_loss_mult <Coefficient of mask supervision> ... <your-data-parser (for example `sdfstudio-data`> ... --include_foreground_mask True