Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vqgan training #52

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
90aaf6d
Added basic idea
isamu-isozaki Apr 21, 2023
59efb88
Adding basic idea
isamu-isozaki Apr 21, 2023
cc2bc90
First idea for training loop
isamu-isozaki Apr 27, 2023
d6067a4
Added perliminary generation
isamu-isozaki Apr 27, 2023
bbfd757
Merge branch 'main' of https://github.com/huggingface/muse into vqgan…
isamu-isozaki Apr 27, 2023
508a00e
Added ema
isamu-isozaki Apr 28, 2023
f6dd2a1
Making config and removed wandb
isamu-isozaki Apr 30, 2023
08b2f12
Removed folder
isamu-isozaki Apr 30, 2023
fa4dc0d
Fixing configs
isamu-isozaki Apr 30, 2023
cfd0c19
Finished basic vqgan testing
isamu-isozaki Apr 30, 2023
4615048
Removed folders
isamu-isozaki May 8, 2023
026305a
Removed config
isamu-isozaki May 8, 2023
290f287
Adding discriminator warmup
isamu-isozaki May 8, 2023
9997d82
Starting adding projected gan tech
isamu-isozaki May 8, 2023
eb86603
Updated config
isamu-isozaki May 8, 2023
02e5c74
Update docs
isamu-isozaki May 8, 2023
a597b1a
Adding slurm file
isamu-isozaki May 10, 2023
105aac7
Updated config
isamu-isozaki May 15, 2023
6872bba
Moving tqdm to batch
isamu-isozaki May 15, 2023
38346ed
Tried updating webdb
isamu-isozaki May 15, 2023
4f296fc
Fixed config
isamu-isozaki May 15, 2023
75f8631
Fixing fmap
isamu-isozaki May 15, 2023
e29acf3
Fixed zero grad
isamu-isozaki May 15, 2023
7994f4e
Fixed discriminator training
isamu-isozaki May 15, 2023
c3e6139
Increased batch size
isamu-isozaki May 15, 2023
277edf1
Changed mixed precision
isamu-isozaki May 15, 2023
a7035ab
Fixed oom
isamu-isozaki May 15, 2023
2717aa9
Fixed debugging
isamu-isozaki May 15, 2023
d6cb053
sanity check
isamu-isozaki May 15, 2023
74eb612
sanity check
isamu-isozaki May 15, 2023
e800cae
Fixed training
isamu-isozaki May 15, 2023
ca9535e
sanity check
isamu-isozaki May 15, 2023
8e21e99
Fixing logs
isamu-isozaki May 15, 2023
3588307
Fixing logs
isamu-isozaki May 15, 2023
d88c402
Fixing logs
isamu-isozaki May 15, 2023
61d0d69
Fixing logs
isamu-isozaki May 15, 2023
c71255a
Added spectral norm f16 config
isamu-isozaki May 15, 2023
851f583
Fixing logs
isamu-isozaki May 15, 2023
1bb0a65
saving discriminator too
isamu-isozaki May 16, 2023
2f81055
Fixing logs
isamu-isozaki May 16, 2023
9d9b227
Made training distributed
isamu-isozaki May 16, 2023
c23ae1c
Properly running vqgan training
isamu-isozaki May 17, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixing logs
isamu-isozaki committed May 15, 2023
commit 851f5833a3911f87979854b0e16380bacd0e7a69
5 changes: 3 additions & 2 deletions training/train_vqgan.py
Original file line number Diff line number Diff line change
@@ -584,11 +584,12 @@ def main():
logger.info(
f"Data (t): {data_time_m.val:0.4f}, {samples_per_second_per_gpu:0.2f}/s/gpu "
f"Batch (t): {batch_time_m.val:0.4f} "
f"LR: {lr_scheduler.get_last_lr()[0]:0.6f}"
f"LR: {lr_scheduler.get_last_lr()[0]:0.6f} "
f"Step: {global_step + 1} "
f"Discriminator Loss: {avg_discr_loss.item():0.4f} "
"No Generator Loss yet " if avg_gen_loss is None else f"Generator Loss: {avg_gen_loss.item():0.4f} "
)
if avg_gen_loss is not None:
logger.info(f"Generator Loss: {avg_gen_loss.item():0.4f} ")

# resetting batch / data time meters per log window
batch_time_m.reset()