Autoencoders to Beta-VAE

Autoencoders to Beta-VAE

An autoencoder is a neural network trained to reproduce its own input, but only after squeezing it through a narrow bottleneck. The encoder maps a high-dimensional input — for example, a 2048-bit Morgan fingerprint describing a molecule, or a microscopy image of a cell culture — into a much smaller vector called the latent code z . The decoder then tries to reconstruct the original input from that code alone. Think of it as forcing the network to write a summary of a drug molecule using only a handful of words: whatever survives the summary must be the chemically essential part. Training minimizes a reconstruction loss — mean squared error for continuous data, or binary cross-entropy for binary fingerprints. Because the bottleneck is smaller than the input, the network cannot simply memorize and copy; it is forced to learn the most salient structure of the data. This makes plain autoencoders useful for dimensionality reduction, denoising, and anomaly detection (for instance, flagging unusual assay readouts that reconstruct poorly). The size of the latent dimension controls how much compression — and therefore how much abstraction — the model must perform. However, a vanilla autoencoder has a critical weakness for generative work: nothing constrains the latent space to be smooth or continuous. The encoder places codes wherever it likes, and the decoder only knows how to decode the exact codes it has seen. If you sample a random point in latent space, it likely falls in a "hole" the decoder has never encountered, and it decodes to meaningless noise. You can compress molecules, but you cannot reliably generate new ones. The Variational Autoencoder (VAE) fixes this with a probabilistic twist. Instead of outputting a single point, the encoder outputs the parameters of a distribution — a mean μ and variance σ² — representing a distribution q(z|x) over latent codes. Training maximizes the evidence lower bound (ELBO) , which balances two terms: a reconstruction term (how well decoded samples match the input) and a KL divergence term that pulls every molecule's latent distribution toward a shared standard normal prior, N(0, I) . The result is a latent space where nearby points decode to similar molecules, and simply sampling z from the prior and decoding produces plausible new candidates — the foundation of generative molecular design. One technical hurdle remains: you cannot backpropagate through a random sampling step. The reparameterization trick solves this by writing z = μ + σ ⊙ ε , where ε ~ N(0, 1) is external noise. The randomness now enters as an