Code Loss Formulation Vs in Paper

[Andrew-Brown1]

Hi,

Thanks for the awesome repo! I had a couple of questions about the implementation of the loss function.

In the paper you multiply the entire GAN loss by the adaptive weight, lambda. The adaptive loss is a function of the entire GAN loss (L_GAN). Two questions:

1. In the code, only the generator loss is multiplied by the adaptive weight (

   taming-transformers/taming/modules/losses/vqperceptual.py

   Line 107 in 9d17ea6

   loss = nll_loss + d_weight * disc_factor * g_loss + self.codebook_weight * codebook_loss.mean()

   ).

• For me, the adaptive weight gets fairly high and so "loss" is far higher than "d_loss". I think this is why some people are not seeing the discriminator loss decrease during training e.g. Has anyone succeeded in reproducing the results? #44)

2. In the code, the adaptive weight, lambda, is only a function of the generator loss (whereas I thought the GAN loss was a function of both the generator and discriminator loss).

Could you offer any advice here?

Thank you