Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FaceSpace dataset training takes too long #12

Open
Volleria opened this issue Mar 22, 2023 · 10 comments
Open

FaceSpace dataset training takes too long #12

Volleria opened this issue Mar 22, 2023 · 10 comments

Comments

@Volleria
Copy link

Hello, dear author of mofanerf:
your paper mentions that it takes about 2 days for training on dual NVIDIA GTX3090 GPUS , however , after l adjusting the chunk、netchunk and N-rand to the recommended minimum, I found that I still need to train nearly 7700 hours on a single NVIDIA GTX 2080ti,are there any configuration errors, how can l reduce training time?

image

@yiyuzhuang
Copy link
Collaborator

Hi! Could you show me your config file?

@Volleria
Copy link
Author

Hi! Could you show me your config file?

image

@gongwkang
Copy link

你好,请问如下报错是因为我数据集下载少了还是代码本身问题?
image

@Volleria
Copy link
Author

Volleria commented Apr 8, 2023

你好,请问如下报错是因为我数据集下载少了还是代码本身问题? image

应该是代码的问题吧, tranforms_train_tranforms_all_16.json.json 这个文件名肯定是有问题的,看起来应该是 transforms_all_16.json 或者是 transforms_train_16.json

@yiyuzhuang
Copy link
Collaborator

yiyuzhuang commented Apr 8, 2023 via email

@yiyuzhuang
Copy link
Collaborator

Hi! Could you show me your config file?

image

I have reviewed the config file and have noticed that the settings for "netchunk" and "chunk" may potentially slow down the performance of the program. It may be worth considering maximizing these settings based on your machine specifications.

@gongwkang
Copy link

作者您好,依然是训练时间过长的问题。我用24G的4卡titan RTX以如下配置训练,依然需要将近2000h
,这与您的训练时间相差过大,请帮我看看是否存在配置问题。
image
image

@gongwkang
Copy link

Hi! Could you show me your config file?

image

请问你通过增加chunk和net_chunk有明显提升训练速度么

@Volleria
Copy link
Author

Hi! Could you show me your config file?

image

请问你通过增加chunk和net_chunk有明显提升训练速度么

还没有试,实验室资源比较紧张,我现在在跑别的东西

@yiyuzhuang
Copy link
Collaborator

作者您好,依然是训练时间过长的问题。我用24G的4卡titan RTX以如下配置训练,依然需要将近2000h ,这与您的训练时间相差过大,请帮我看看是否存在配置问题。 image image

你好,我注意到“i_testset=1,i_weights=1”设置了“验证指标,保存模型的间隔为1”,这两个应该设置大一些,如100,000。每次迭代都渲染一张完整的图像会占用大量运算时间,而和chunk的影响可能不大。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants