VQGAN: Taming Transformers for High-Resolution Image Synthesis
from my lab in Heidelberg University

Paper explained on my YouTube channel!

Authors introduce VQGAN which combines the efficiency of convolutional approaches with the expressivity of transformers. VQGAN is essentially a GAN that learns a codebook of context-rich visual parts and uses it to quantize the bottleneck representation at every forward pass.
The self-attention model is used to learn a prior distribution of codewords.
Sampling from this model then allows producing plausible constellations of the codewords which are then fed through a decoder to generate realistic images in arbitrary resolution.

📝 Paper
⚙️ Code (with pretrained models)
📓 Colab notebook
📓 Colab notebook to compare the first stage models in VQGAN and in DALL-E

💪🏻🦾🤙🏼
▶️ YouTube Video explanation