VQ-VAE with discrete flows

[MichelPezzat]

I've been trying this autorregresive model for quantizised vectors tokens. So far, the training has been troublesome. Any suggestions?
Thanks in advance.

[zackziegler95]

Weird model! If you’re doing the usual VQ-VAE thing with the straight through estimator, could there be some kind of accumulation of error? I can’t picture the model exactly, but you’ll probably have problems with any VQ-VAE if you have an auto regressive process in training that can “compound” the error from the ST estimator.

[MichelPezzat]

I'm not familiar with STE concept. The discrete flow is to model the prior of the discrete bottleneck sequence given by the previously trained encoder. So far, the KL term goes to high (around a 2 billion value) so the training goes nowhere. I guess I'll try something else. Thanks for replying anyway.