Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log for val results does not save correctly when IterBasedRunner is used. #1248

Closed
haotian-liu opened this issue Aug 7, 2021 · 3 comments
Closed

Comments

@haotian-liu
Copy link

When I am using IterBasedRunner is used, and the evaluation runs during the training (triggered by evaluation hook evaluation=dict(interval=5000)), the eval results are not logged with val mode, and are logged as train mode. This is different from EpochBasedRunner.

Specifically, for EpochBasedRunner, training losses will be logged in Tensorboard with prefix train/ (e.g., train/loss, train/loss_cls, etc.), and evaluation results will be logged with prefix val/ (e.g., val/table, val/acc, val/miou, etc.). However, in IterBasedRunner, they are all logged with prefix train/ (e.g., train/loss, train/table, train/acc, train/miou), which is weird and causes troubles for visualization.

It seems that when evaluation hook is triggered in after_iter stage, the mode will not be changed to eval and remains as train instead, which might be the cause of the issue.

Please kindly advise if there is anything that I am not doing correctly, thank you so much!

@zhouzaida
Copy link
Collaborator

hi @haotian-liu , thanks for you feedback, we are fixing the bug

@haotian-liu
Copy link
Author

Thank you so much!

@zhouzaida
Copy link
Collaborator

closed by #1252

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants