Torso not added to final output

[RobinRenggli]

I trained the Obama example and I get expected results from the HeadNeRF, but when I train and render the TorsoNeRF, I get results that look very similar (i.e. a floating head, torso is missing). I only trained for 10'000 iterations, but from my experience with NeRF, the correct shapes should appear rather quickly.

Could it be that there is a setting that needs to be turned on, or a bug in the code, that prevents the Torso from being learned by the TorsoNeRF? Or does it take a large amount of iterations for the Torso to appear?

I used the commands as specified in the README.

Thanks for your help, this is a very cool project!

HeadNeRF output:

TorsoNeRF output:

[YudongGuo]

Hi, actually it's a problem of network initialization. We found that in few cases the torso model could not produce proper output due to bad initialization.

One solution is to run the command multiple times until good initialization (in my experience, the psnr is greater than 21 in 1000 iterations for a good initialization). Another solution is to use a pretrained Torso model (e.g. torso model trained from another person).

[RobinRenggli]

Oh, ok, this is really useful to know. I see now that in my run PSNR was still below 20 after 10'000 iterations.

Thanks for your help.

[RobinRenggli]

I tried several times now with multiple attempts with PSNR over 25 and different input videos and it never worked. Could you maybe provide a pretrained torso model such that I can try if that makes it work/look at how the results are supposed to look?
(The HeadNeRF works very well!)

[YudongGuo]

Hi, the Torso model pretrained on the obama video is here:
https://www.dropbox.com/s/hy99o8jm3iyskur/500000_body.tar?dl=0

It should work.

[RobinRenggli]

Thank you, I'll try it out today.

[RobinRenggli]

I can confirm that it works by using a pretrained torso model, even if the torso in question is very different from the pretrained one.

In my example, I could not get a good initialization even after many tries. Do you maybe have an intuition or explanation on how likely/unlikely a good initialization is? Does it depend on how much of the frame is taken up by the torso?

Feel free to close this issue, as my question is more of a research one.

[YudongGuo]

I can confirm that it works by using a pretrained torso model, even if the torso in question is very different from the pretrained one.

In my example, I could not get a good initialization even after many tries. Do you maybe have an intuition or explanation on how likely/unlikely a good initialization is? Does it depend on how much of the frame is taken up by the torso?

Feel free to close this issue, as my question is more of a research one.

Hi, in my experience, I can get a good initialization with a single attempt in most cases. In all cases, a good initialization is achieved with no more than 5 attempts.

For bad initialization, the PSNR does not show obvious improvement (usually below 20). For good initialization, the PSNR shows obvious improvement in the first 1000 iterations and usually above 21. I didn't notice the correlation with the torso part rate.

It's strange that you didn't get proper results even with PSNR greater than 25. However, you can always use the pretrained torso model for initialization.

[saslamsameja]

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

[YudongGuo]

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

Hi, expected PSNR is over 30. And in rare cases (with lower rate than TorsoNeRF) HeadNeRF suffers from bad initialization too. So the advices for training TorsoNeRF above also apply to HeadNeRF.

[saslamsameja]

Hi @YudongGuo, What is the expected value of PSNR for the training of HeadNeRF? I started a training iteration and completed about 200k iterations but my PSNR value is still at around 19. I used all default settings and commands given in Readme.

Hi, expected PSNR is over 30. And in rare cases (with lower rate than TorsoNeRF) HeadNeRF suffers from bad initialization too. So the advices for training TorsoNeRF above also apply to HeadNeRF.

@YudongGuo, could you please provide the HeadNeRF pre-trained weights for the Obama video as well?

[YudongGuo]

Hi @saslamsameja, I have uploaded both pretrained models in pretrained_models/ folder. Note that the code of face tracking for training the uploaded models is slightly different from this repo.

[saslamsameja]

Thanks, @YudongGuo. I have two questions:

• Even though the code for face tracking is different but using these weights for initialization will fix the problem of training, right?
• And do you plan on publishing the changes related to the face tracking as well?

[saslamsameja]

I got a PSNR of around 30 on the 2nd iteration of HeadNerf training, without loading the pre-trained weights. Thanks, @YudongGuo.

[YudongGuo]

Thanks, @YudongGuo. I have two questions:

• Even though the code for face tracking is different but using these weights for initialization will fix the problem of training, right?
• And do you plan on publishing the changes related to the face tracking as well?

Hi, the weights could be used for good initialization. And there were slight differences in differentiable rendering settings and tracking smoothness settings (in general the tracking code in this repo is better).

[saslamsameja]

adnerfsub.mp4

@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

[YudongGuo]

adnerfsub.mp4
@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

I think it's because the difference between the tracking parameters. You can finetune the models with current parameters, and I will check this part too.

[YudongGuo]

adnerfsub.mp4
@YudongGuo, the output that I generated using the weights that you shared has a lot of flickering between the head and torso. Could this be due to the difference in the tracking code that you mentioned? Or is there any other reason for this. Because I noticed that the demos that you shared do not have this noise.

obama-nerf.mp4

Hi, this is the generated video with the tracking parameters of pretrained models. So it's because the inconsistence between tracking parameters.

[kingsj0405]

I also got the same problem with another identity.
Both trainings from scratch and from Obama are tried.
Can I get any advice to converge?