Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

loss变为nan #18

Open
KKKSQJ opened this issue May 16, 2023 · 5 comments
Open

loss变为nan #18

KKKSQJ opened this issue May 16, 2023 · 5 comments

Comments

@KKKSQJ
Copy link

KKKSQJ commented May 16, 2023

作者您好，我在训练自己的数据集的时候，训练了一段时间后，loss变为nan。请问有什么解决方法吗？
@Egrt
Copy link
Owner

Egrt commented May 17, 2023

作者您好，我在训练自己的数据集的时候，训练了一段时间后，loss变为nan。请问有什么解决方法吗？

检查数据集格式是否正确，如果你的数据集很小，建议降低学习率。

@KKKSQJ
Copy link
Author

KKKSQJ commented May 17, 2023

您好，降低学习率之后，loss会正常训练一段时间。但是迭代几轮epoch之后，loss还是会变为nan。我的数据集格式是x1,y1,x2,y2,x3,y3,x4,y4的格式。这里的标注顺序是否要求从目标左上角开始，顺时针标注？

@KKKSQJ
Copy link
Author

KKKSQJ commented May 17, 2023

我的训练数据数量有1.4W张图片，每张图片5-10个旋转目标。图片分辨率是1920*1080。

@Egrt
Copy link
Owner

Egrt commented May 18, 2023

我的训练数据数量有1.4W张图片，每张图片5-10个旋转目标。图片分辨率是1920*1080。

大概率数据集标注的问题。

@Egrt
Copy link
Owner

Egrt commented May 26, 2023

已经更新代码，解决了损失值为nan的问题，在kld损失中，分母部分加上了eps值避免loss值为0

@yuxin7 yuxin7 mentioned this issue Aug 17, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@KKKSQJ @Egrt