nerfacto not working with blender-data

[iszihan]

Hello, I am trying to get nerfacto to work with blender-data (ficus more specifically), and here is the image I get

My training command is just ns-train nerfacto --vis tensorboard blender-data --data ../data/blender/ficus and the data directory is downloaded with 'ns-download-data blender'. Is this expected to not work well with blender dataset?

[iszihan]

This issue solves this problem.

[THUROI0787]

This issue solves this problem.

Hi dude, I still can't solve the problem. I was able to get good results with lego using solutions from other issues, but for ficus, I got poor results with all three models (nerfacto, instant-ngp, instant-ngp-bounded), with iterations==10k. Could you please tell me what configuration or flag you used?

[iszihan]

I usually see good results with 200k steps : )

[THUROI0787]

Well, here are still two problems on me :(

1. Using nerfacto with the command below:
   ns-train nerfacto --data="data/blender/lego/" --experiment-name="lego" --max-num-iterations=10000 --pipeline.model.background-color white --pipeline.model.proposal-initial-sampler uniform --pipeline.model.near-plane 2. --pipeline.model.far-plane 6. --pipeline.datamanager.camera-optimizer.mode off --pipeline.model.use-average-appearance-embedding False --pipeline.model.distortion-loss-mult 0 --pipeline.model.disable-scene-contraction True blender-data
   For dataset lego, it converged very fast, actually after just 200 iterations I could already see a good result. But for dataset ficus, during the training until 10k iterations there is still nothing... :( I don't know what happened and why.

2. Using instant-ngp-bounded, both lego and ficus couldn't converge well after 30k iterations (I'm still waiting for the results maybe until 200k iterations), and it looks like there are some individual clouds close to bound.

Do you know how to solve it? I don't think we need 200k steps for nerfacto (almost slower than the original NeRF with a worse result), there must be something we missed.

[THUROI0787]

Hi there! I tried again using the command above on dataset ficus, and it works now. You can also have a try, it can have a good result after just 2k iterations (almost stable after 8k iters).
Last time it worked on lego but failed on ficus, I think the instability of my network should be blame for it. :(

[iszihan]

Gald it worked out! And sorry I was thinking of training neus when I replied, and yes with nerfacto it shouldn't take 200k steps.

[iszihan]

@THUROI0787
Do you see these kind of white fog rendering during training? It doesn't seem to go away in mine.

[THUROI0787]

@THUROI0787 Do you see these kind of white fog rendering during training? It doesn't seem to go away in mine.

Hi, I remember it didn't happen after I trained it for enough epochs (for nerfacto, maybe 5k iterations). Could you show me your command?

[iszihan]

Yes, I tried
ns-train nerfacto --pipeline.model.background-color white --pipeline.model.disable-scene-contraction True --output-dir /scratch/local/2023/zling/project/shellnerf/outputs --vis tensorboard --pipeline.model.proposal-initial-sampler uniform --pipeline.model.near-plane 2. --pipeline.model.far-plane 6. --pipeline.datamanager.camera-optimizer.mode off --pipeline.model.use-average-appearance-embedding False --pipeline.model.distortion-loss-mult 0 blender-data --data ../data/ficus

[iszihan]

this is the rendering after 29500 steps. :/

[iszihan]

If possible, would you mind sharing what your rendering looks like by the end? @THUROI0787 Thank you!

[THUROI0787]

@iszihan Well... The same problem also for me on ficus, but it has a quite good result on lego. I wonder why...

[monlabiss]

@iszihan Well... The same problem also for me on ficus, but it has a quite good result on lego. I wonder why...

Hi, I wonder how can you get this result. Could you show me your command please?

[dubrovin-sudo]

Well, here are still two problems on me :(

1. Using _nerfacto_ with the command below:
   `ns-train nerfacto --data="data/blender/lego/" --experiment-name="lego" --max-num-iterations=10000  --pipeline.model.background-color white --pipeline.model.proposal-initial-sampler uniform --pipeline.model.near-plane 2. --pipeline.model.far-plane 6. --pipeline.datamanager.camera-optimizer.mode off --pipeline.model.use-average-appearance-embedding False --pipeline.model.distortion-loss-mult 0 --pipeline.model.disable-scene-contraction True  blender-data`
   For dataset _lego_, it converged very fast, actually after just **200** iterations I could already see a good result. But for dataset _ficus_, during the training until 10k iterations there is still nothing... :(  I don't know what happened and why.

2. Using _instant-ngp-bounded_, both _lego_ and _ficus_ couldn't converge well after 30k iterations (I'm still waiting for the results maybe until 200k iterations), and it looks like there are some individual clouds close to bound.

Do you know how to solve it? I don't think we need 200k steps for nerfacto (almost slower than the original NeRF with a worse result), there must be something we missed.

This command give an error:
Unrecognized or misplaced arguments: --pipeline.datamanager.camera-optimizer.mode

[AlexanderRitter02]

This command give an error: Unrecognized or misplaced arguments: --pipeline.datamanager.camera-optimizer.mode

@dubrovin-sudo
Use --pipeline.model.camera-optimizer.mode off now. (e.g. model instead of datamanager)