Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save .hydra/*.yaml before task execution #678

Closed
binshengliu opened this issue Jun 15, 2020 · 4 comments · Fixed by #686
Closed

Save .hydra/*.yaml before task execution #678

binshengliu opened this issue Jun 15, 2020 · 4 comments · Fixed by #686
Labels
bug Something isn't working
Milestone

Comments

@binshengliu
Copy link
Contributor

🐛 Bug

Saving the configurations is a really cool idea that helps me check what I actually did. I noticed the save timing is delayed after task execution in the 1.0.0rc1. I have encountered some scenarios that I think it would be better if the configs are saved earlier.

For example,

  1. Tasks may exit accidentally due to bugs. Knowing detailed configurations would be helpful.
  2. For time-consuming tasks like ML training which may last for several days, it would be helpful to be able to check the config.yaml. This helps me a lot when I have multiple tasks running but I can't remember the real difference. I do diff config.yaml.
  3. Some tasks don't need to be finished. The full training may be designed to run 100 epochs but it also happens it's aborted/killed in the middle.

To reproduce

Default behavior in 1.0.0rc1.

** Minimal Code/Config snippet to reproduce **
NA

** Stack trace/error message **
NA

Expected Behavior

NA

System information

  • Hydra Version : 1.0.0rc1
  • Python version : NA
  • Virtual environment type and version : NA
  • Operating system : NA
@binshengliu binshengliu added the bug Something isn't working label Jun 15, 2020
@omry
Copy link
Collaborator

omry commented Jun 15, 2020

Is this a change in 1.0?

@omry omry added this to the 1.0.0 milestone Jun 15, 2020
@binshengliu
Copy link
Contributor Author

It seems this change was introduced in 98128a4

@omry
Copy link
Collaborator

omry commented Jun 15, 2020

Interesting, not sure why I moved it down.
can you send a PR fixing it?

@binshengliu
Copy link
Contributor Author

Yeah sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants