Generative Models: Autoencoders, Variational Autoencoders, GANs and Their Applications

Introduction to Generative Models

Generative Models are Artificial Intelligence models that learn the underlying patterns and characteristics of existing data and use this knowledge to generate new data similar to the original data.

Unlike traditional models that mainly classify or predict, generative models can create new content such as images, text, audio, video, and synthetic data.

For example, if a generative model is trained on thousands of images of clothes, it can learn their patterns and generate new clothing images that were not present in the original dataset.

Simple Flow:

Training Data → Learn Patterns → Generative Model → Generate New Data  

Types of Generative Models

There are different types of generative models. In this unit, we focus on three important models:

1. Autoencoder (AE) → Learns a compressed representation of data and reconstructs the original data.

2. Variational Autoencoder (VAE) → Learns a probabilistic representation of data and can generate new samples.

3. Generative Adversarial Network (GAN) → Uses two competing neural networks, a Generator and a Discriminator, to generate realistic new data.

Simple understanding:

AE → Compress and Reconstruct

VAE → Learn Distribution and Generate

GAN → Generate and Compete

Real-Life Examples

1. Autoencoder (AE) – Removing Noise from Images

Suppose an old scanned photograph contains noise, spots, or distortion. An Autoencoder can learn the important features of clean images and reconstruct a cleaner version.

Noisy Image → Autoencoder → Reconstructed/Cleaner Image

Real application: Image denoising and image restoration.

2. Variational Autoencoder (VAE) – Generating New Face Images

Suppose a VAE is trained on thousands of human face images. It learns a probabilistic latent representation of facial characteristics and can sample from that latent space to create new synthetic face images.

Face Dataset → VAE learns distribution → Sample Latent Vector → New Face Image

Real application: Synthetic image generation and data augmentation.

3. Generative Adversarial Network (GAN) – Generating Realistic Synthetic Images

A GAN can learn from a large collection of images and generate new realistic-looking synthetic images. The Generator creates images while the Discriminator evaluates whether they look real or fake.

Random Noise → Generator → Synthetic Image → Discriminator → Real/Fake

Real application: Synthetic face generation, image enhancement, virtual product/design creation, and training-data augmentation.

Important Terms in Generative Models

Encoder → Converts input data into a smaller, compressed representation.

Decoder → Converts the compressed representation back into reconstructed or generated data.

Latent Space → A compressed representation where the model stores the important learned features of the data.

Latent Vector → A particular set of numerical values representing one point in the latent space.

Probability Distribution → Describes how different possible values are distributed and how likely they are.

Sampling → Selecting values from the learned probability distribution to generate new data.

Generator → The part of a GAN that creates new synthetic or fake data.

Discriminator → The part of a GAN that determines whether data is real or generated (fake).

Random Noise → Random numerical values given to the Generator as input for creating new data.

Adversarial Training → Training in which the Generator and Discriminator compete with each other and improve together.

Reconstruction → Recreating the original input from its compressed representation.

Synthetic Data → Artificially generated data that resembles real data.

Data Augmentation → Increasing the amount and variety of training data by creating additional samples.

4.1 Autoencoders (AE)

An Autoencoder (AE) is a neural network that learns to compress input data into a smaller representation and then reconstruct the original data from that representation.

Basic Architecture

Input → Encoder → Latent Space → Decoder → Reconstructed Output

For example, when an image is given to an Autoencoder:

Original Image → Encoder → Compressed Features → Decoder → Reconstructed Image

The Encoder extracts the important features and reduces the data size.

The Latent Space stores the compressed representation containing important features of the input.

The Decoder uses this compressed representation to reconstruct an image as close as possible to the original.

Main Objective

The objective of an Autoencoder is to minimize the difference between the original input and reconstructed output, called reconstruction error.  

Working of an Autoencoder

The working of an Autoencoder can be understood in three main stages:

1. Encoding → The input data is passed through the Encoder, which extracts important features and converts it into a smaller representation.

2. Latent Representation → The compressed information is stored in the Latent Space. It contains the most important features learned from the input.

3. Decoding → The Decoder takes the compressed representation and reconstructs the original data.

Example

Suppose we provide an image of a shoe:

Shoe Image → Encoder → Compressed Shoe Features → Decoder → Reconstructed Shoe Image

During training, the model compares:

Original Image ↔ Reconstructed Image

The difference between them is called Reconstruction Error (or Reconstruction Loss).

The Autoencoder adjusts its weights during training to minimize this reconstruction error. 

Applications of Autoencoders

Autoencoders are used in many applications where feature learning and reconstruction are required.

Image Denoising → Removes unwanted noise from images and reconstructs a cleaner image.

Dimensionality Reduction → Converts high-dimensional data into a smaller representation while preserving important information.

Anomaly Detection → Identifies unusual data by detecting samples that the Autoencoder cannot reconstruct well.

Feature Extraction → Automatically learns important features from the input data.

Image Reconstruction → Reconstructs an input image from its compressed representation.

Data Compression → Represents large input data using fewer features in the latent space.

Example: Anomaly Detection

If an Autoencoder is trained mainly on normal transactions, it learns to reconstruct normal transactions accurately.

Normal Transaction → Small Reconstruction Error

Unusual/Fraudulent Transaction → Large Reconstruction Error  

Variational Autoencoder (VAE)

A Variational Autoencoder (VAE) is an advanced form of Autoencoder that learns a probability distribution in the latent space, instead of learning only a fixed compressed representation.

This allows the VAE not only to reconstruct existing data but also to generate new data similar to the training data.

Basic Architecture

Input → Encoder → Latent Distribution → Sampling → Decoder → Reconstructed/Generated Output

For example, if a VAE is trained on fashion images:

Fashion Images → Encoder → Learn Latent Distribution → Sample New Latent Vector → Decoder → New Fashion Image

Main Difference from AE

AE → Mainly learns to compress and reconstruct the input.

VAE → Learns a distribution of the data, allowing it to reconstruct as well as generate new samples.

How VAE Works

Unlike a normal Autoencoder, a VAE does not directly create one fixed value for each latent feature. Instead, the Encoder learns a probability distribution.

It mainly learns two values:

Mean (μ) → Represents the center of the learned distribution.

Variance (σ²) → Represents how much the values can vary around the mean.

The VAE then samples a latent vector from this distribution and sends it to the Decoder.

Simple Flow

Input Image → Encoder → Mean (μ) + Variance (σ²) → Sampling → Latent Vector → Decoder → Output Image

Simple Example

Suppose the VAE is trained on thousands of shoe images. Instead of memorizing individual shoes, it learns a structured distribution of their important features.

We can then sample a new point from this learned latent space:

New Latent Sample → Decoder → New Synthetic Shoe Image

This ability to sample from the learned distribution and generate new data is the main reason VAE is considered a generative model.

Applications of Variational Autoencoders (VAE)

Image Generation → Generates new images similar to the images used during training.

Synthetic Data Generation → Creates artificial data samples that resemble real data.

Data Augmentation → Generates additional training samples when the available dataset is limited.

Image Reconstruction → Reconstructs images from their latent representations.

Anomaly Detection → Identifies unusual samples based on reconstruction behaviour or how poorly they fit the learned distribution.

Image Denoising → Helps reconstruct cleaner versions of noisy images.

Feature Learning → Learns meaningful compressed features automatically from complex data.

Drug and Molecule Generation → Can generate new candidate molecular structures by learning patterns from existing molecular data.  

4.2 Generative Adversarial Networks (GANs)

A Generative Adversarial Network (GAN) is a generative model that uses two neural networks that compete with each other during training.

The two networks are:

Generator → Creates new synthetic or fake data that tries to look like real data.

Discriminator → Examines the data and predicts whether it is real or fake.

Basic Architecture

Random Noise → Generator → Generated/Fake Image

Then:

Real Image + Generated Image → Discriminator → Real or Fake

Simple Example

Suppose a GAN is trained using thousands of human face images.

The Generator tries to create new face images.

The Discriminator checks whether each image is a real image from the dataset or a fake image created by the Generator.

As training continues, the Generator gradually learns to create more realistic synthetic faces.

Generator tries to fool the Discriminator, while the Discriminator tries to correctly identify the fake images.

This competition between the two networks is called Adversarial Training. 

Working of GAN

GAN training takes place as a continuous competition between the Generator and the Discriminator.

Step 1 – Random Noise: Random numerical values are provided as input to the Generator.

Step 2 – Generate Fake Data: The Generator uses the random noise to create a synthetic image.

Step 3 – Discriminator Checks Images: The Discriminator receives both real images from the dataset and fake images from the Generator.

Step 4 – Real or Fake Prediction: The Discriminator learns to distinguish real images from generated images.

Step 5 – Generator Improves: Based on the feedback from the Discriminator, the Generator gradually learns to create more realistic images.

Simple Flow

Random Noise → Generator → Fake Image → Discriminator → Real/Fake

At the same time:

Real Dataset Images → Discriminator → Real/Fake

This process is repeated many times until the Generator becomes better at producing realistic synthetic data. 

Applications of GANs

Image Generation → Generates new realistic-looking images such as faces, objects, or fashion items.

Synthetic Data Generation → Creates artificial data similar to real data for training AI models.

Data Augmentation → Generates additional samples to increase the size and variety of a training dataset.

Image-to-Image Translation → Converts an image from one form to another, such as a sketch into a realistic image.

Image Super-Resolution → Converts low-resolution images into higher-resolution images.

Image Restoration → Helps reconstruct missing or damaged portions of images.

Face Generation → Generates realistic synthetic human faces that do not correspond to actual people.

Creative Design → Generates new artwork, fashion concepts, product designs, and other visual content. 

4.3 Simple Case Study – Image Generation

Case Study: Generating Fashion Images

Suppose we have a dataset containing thousands of images of shirts, shoes, bags, dresses, and other fashion items.

A generative model such as a VAE or GAN can learn the visual patterns present in these images and then generate new synthetic fashion images.

Simple Flow

Fashion Image Dataset → Train Generative Model → Learn Image Patterns → Generate New Fashion Images

For example, using the Fashion-MNIST dataset, a GAN can learn from existing fashion images and generate new images that resemble fashion items but are synthetically created by the model.

Objective

The objective is to demonstrate how a generative model can learn from existing images and create new images with similar characteristics.

Simple Case Study – Text-to-Image Generation

Case Study: Generating an Image from a Text Prompt

Text-to-Image Generation is a Generative AI technique in which a model creates an image based on a natural-language text description called a prompt.

For example, the user provides:

Prompt: “A red car travelling on a mountain road.”

The generative model understands the description and generates an image representing the given prompt.

Simple Flow

Text Prompt → Text Understanding/Encoding → Generative Model → Generated Image

Objective

The objective is to generate an image that visually represents the objects, properties, and context described in the text prompt.

Real-Life Applications

Text-to-image generation is used for digital artwork, advertising, product-design concepts, educational illustrations, interior-design visualization, and marketing content creation.







Simple Case Study on Image Generation and Text-to-Image Generation

Problem Statement

To demonstrate the use of Generative AI for image generation and text-to-image generation, where a pretrained generative model is used to create new images from text prompts. The practical aims to understand how a simple natural-language description can be transformed into a corresponding generated image.

Text-To-Image Generation

StableDiffusionPipeline 

from the Hugging Face Diffusers library is used to load and run the pretrained Stable Diffusion v1.5 model for text-to-image generation. The Stable Diffusion v1 family learned from very large LAION image–text datasets containing hundreds of millions to billions of image–text pairs, while Stable Diffusion v1.5 was further trained for 595,000 steps at 512 × 512 resolution on a filtered LAION-Aesthetics subset. A normal fixed training/testing split such as 80:20 is not specified for this pretrained model, because it was developed through large-scale pretraining rather than the small-dataset classification workflow we commonly use in practicals. In our notebook, we are not training the model again; we are using its already learned knowledge to generate new images from the text prompts provided by the user.