Introduction to Optimizers

In Deep Learning, an optimizer is an algorithm used to update the weights and biases of a neural network to minimize the loss (error) during training.

After Backpropagation calculates the gradients, the optimizer uses these gradients to determine how the weights should be changed.

Training Flow:

Feed Forward → Prediction → Loss → Backpropagation → Gradient → Optimizer → Weight Update

In simple words: Backpropagation finds how the weights should change, and the optimizer actually updates the weights to reduce the error.

Types of Optimizers

There are several optimization algorithms used in Deep Learning. Common optimizers include:

1. Gradient Descent (GD) – Updates weights using the complete training dataset.
2. Stochastic Gradient Descent (SGD) – Updates weights using individual samples or mini-batches during training.
3. SGD with Momentum – Adds information from previous updates to make learning smoother and faster.
4. AdaGrad – Adapts the learning rate separately for different parameters.
5. RMSProp – Uses a moving average of squared gradients to control weight updates.
6. Adam – Combines momentum-like gradient averaging with RMSProp-like adaptive scaling.

Among these, SGD, RMSProp, and Adam are commonly used for teaching and implementing neural networks.

Practical 

Problem Statement

Using the Breast Cancer Wisconsin Dataset, develop a Deep Feed Forward Neural Network to classify tumors as Benign or Malignant. Train the model separately using SGD, Adam, and RMSProp optimizers and compare their performance based on accuracy and loss using a comparison table and graphs.

Go to Jupyter Notebook