Add VAE and AE notes

Author: Jonas1312

base/science-tech-maths/machine-learning/algorithms/autoencoder-vae/autoencoder-vae.pdf

@@ -0,0 +1,23 @@
+# Autoencoder
+
+- An auto-encoder is the combination of an function that converts the input data into a different representation, and a decoder function that converts the new representation back into the original format.
+- Can be used for data visualization (compressing data into 2D or 3D).
+- Autoencoders work because data may occupy a manifold of lower dimensionality than n.
+- One class autoencoders are used for anomaly detection:
+  - A well-trained autoencoder will predict any new data that is coming from the normal state of the process (as it will have the same pattern or distribution).
+  - Therefore, the reconstruction error will be small.
+  - However, if we try to reconstruct a data from a rare-event, the Autoencoder will struggle.
+  - This will make the reconstruction error high during the rare-event.
+  - We can catch such high reconstruction errors and label them as **a rare-event prediction**.
+- Is it possible to find a linear boundary with them? Not sure but I guess yes. For example, if you train an autoencoder on really simple images spread in two classes, you might be able to apply just a single threshold to "classify" those images. the decision boundary is nonlinear in the original data space, but linear in the feature space into which the data are mapped
+
+VAE are generative, but what about autoencoders?
+
+- The latent space can be repurposed for something else like interpolation.
+- We usually consider autoencoders not be generative since there are no distributional assumptions of how your data (in latent space) is generated.
+- In probabilistic reasoning lingo, there are no assumptions on the data generation process.
+- (Ian Goodfellow): the autoencoder doesn’t give direct explicit access to an estimate of the density or the ability to sample directly.
+
+Bottleneck dimensionality vs reconstruction loss:
+
+![](./autoencoders.png)

base/science-tech-maths/machine-learning/algorithms/autoencoder-vae/autoencoders/autoencoders.md

@@ -0,0 +1,38 @@
+# Variational Autoencoders
+
+![](vae2.png)
+
+![](vae-gaussian.png)
+
+- We add a constraint on the encoding network, that forces it to generate latent vectors that roughly follow a unit gaussian distribution.
+- Generating new images is now easy: all we need to do is sample a latent vector from the unit gaussian and pass it into the decoder.
+
+```python
+image_loss = mean((generated_image - real_image)**2)
+latent_loss = kl_divergence(latent_variable, unit_gaussian)
+loss = image_loss + latent_loss
+```
+
+![](vae.png)
+
+- the KL divergence of two gaussians is easy to compute in its closed form.
+- In order to optimize the KL divergence, we need to apply a simple reparameterization trick: instead of the encoder generating a vector of real values, it will generate a vector of means and a vector of standard deviations.
+
+```python
+samples = random_normal([batchsize, n_z], mean=0, std=1, dtype=tf.float32)
+sampled_z = z_mean + (z_stddev * samples)
+```
+
+## More
+
+- <https://wiseodd.github.io/techblog/2016/12/10/variational-autoencoder/>
+- <https://hameddaily.blogspot.com/2018/12/yet-another-tutorial-on-variational.html>
+- <https://jaan.io/what-is-variational-autoencoder-vae-tutorial/>
+- <https://towardsdatascience.com/generating-images-with-autoencoders-77fd3a8dd368>
+- <https://wiseodd.github.io/techblog/2017/01/24/vae-pytorch/>
+- <https://jhui.github.io/2017/03/06/Variational-autoencoders/>
+- <https://www.jeremyjordan.me/variational-autoencoders/>
+- <http://kvfrans.com/variational-autoencoders-explained/>
+- <https://miro.medium.com/max/2255/1*ejNnusxYrn1NRDZf4Kg2lw@2x.png>
+- <https://www.assemblyai.com/blog/variational-autoencoders-for-dummies/>
+- Invariant Representations without Adversarial Training : <https://dcmoyer.github.io/selfhosted/blag.html>