Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"keep_ckpts" not working #65

Closed
ThrowawayAccount01 opened this issue Mar 22, 2023 · 4 comments · Fixed by #66 or #89
Closed

"keep_ckpts" not working #65

ThrowawayAccount01 opened this issue Mar 22, 2023 · 4 comments · Fixed by #66 or #89
Assignees
@Lordmau5
Labels
bug Something isn't working

Comments

@ThrowawayAccount01
Copy link

In the config file, the default "keep_ckpts": 3 does not work. When training, all checkpoints will be saved. Changing the value to any other number will not work as well.
@34j 34j mentioned this issue Mar 22, 2023
Merged
@34j
Copy link
Collaborator

34j commented Mar 22, 2023

I noticed it earlier too. I am having difficulty correcting it.

@34j 34j added the bug Something isn't working label Mar 22, 2023
@34j 34j closed this as completed in #66 Mar 22, 2023
@ThrowawayAccount01
Copy link
Author

I just updated to 1.2.8 but the bug is still present. All the checkpoints are still saved.

@34j 34j reopened this Mar 23, 2023
@34j
Copy link
Collaborator

34j commented Mar 23, 2023

😭

@34j 34j added the help wanted Extra attention is needed label Mar 23, 2023
@Lordmau5
Copy link
Contributor

After a bit of back and forth with ChatGPT and looking at the documentation for itertools.groupby it is an issue with how the models are sorted.

The resulting array of models_sorted is sorted by the suffix of the G_ or D_ files, but the resulting array alternates between a "G" path and a "D" path.
This, ultimately, means that it will create a group for every G_ and D_ path
image

I fixed this by modifying the sorted method call.

Additionally, I've cleaned up the code a bit more.
Will do a pull request shortly!

Lordmau5 added a commit to Lordmau5/so-vits-svc-fork that referenced this issue Mar 23, 2023
@Lordmau5
Fix old checkpoint deletion by sorting the models properly
c9c1563 
Should fix voicepaw#65 this time around.
The reason why this didn't work before is because it sorted them based on their name or time alone, but not also based on their prefix, so the resulting array looked like "G_", "D_", "G_", "D_", ...

This addresses that by also sorting based on their prefix, resulting in a properly sorted array "G_", "G_", "G_", "D_", "D_", "D_"

This has to be done this way due to how the itertools.groupby method works: https://note.nkmk.me/en/python-itertools-groupby/
@Lordmau5 Lordmau5 mentioned this issue Mar 23, 2023
Merged
@34j 34j removed the help wanted Extra attention is needed label Mar 24, 2023
Lordmau5 added a commit to Lordmau5/so-vits-svc-fork that referenced this issue Mar 24, 2023
@Lordmau5
fix: fix old checkpoint deletion by sorting the models properly (voic…
e6b5bf7 
…epaw#65)
@34j 34j closed this as completed in #89 Mar 24, 2023
34j pushed a commit that referenced this issue Mar 24, 2023
@Lordmau5
fix: fix old checkpoint deletion by sorting the models properly (#65) (…
287dc94 
…#89)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lordmau5 Lordmau5

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@Lordmau5 @34j @ThrowawayAccount01