Stochastic Gradient Descent in Deep Neural Networks

Real-Life Example

Consider a hospital where a patient's medical test results, such as radius, texture, perimeter, and area of a breast tumor, are collected. These features are provided as input to the neural network.

Input features → Hidden layers → Benign or Malignant prediction

If the model makes an incorrect prediction, the error is calculated. Backpropagation identifies how the network weights contributed to the error, and Stochastic Gradient Descent (SGD) updates those weights so the model can make more accurate predictions during the next training cycle.

Practical Objectives

The practical will demonstrate how to:

• Load and preprocess the banknote dataset
• Build a Deep Feed Forward Neural Network
• Configure the SGD optimizer
• Train the model using backpropagation
• Demonstrate pure SGD using batch_size=1
• Demonstrate mini-batch gradient descent
• Compare different learning rates
• Plot training and validation loss
• Evaluate the trained network   
  
  

  The correct term is Stochastic Gradient Descent, commonly written as SGD.

  It is different from Gradient Boosting.

  • Stochastic Gradient Descent is an optimization algorithm used to update the weights of a neural network.
  • Gradient Boosting is a machine learning technique that combines multiple weak models, usually decision trees.

  For Deep Learning and Backpropagation, the required topic is Stochastic Gradient Descent.

  ---

  1. Why Does a Neural Network Need Learning?

  A neural network initially starts with random weights.

  Because the weights are random, the first prediction may be incorrect.

  For example, consider a student placement model.

  Actual Result	Predicted Result
  Placed	Not Placed

  The network must improve its prediction.

  To improve, it must change its weights.

  The process of finding better weights is called optimization.

  ---

  2. What Is an Optimizer?

  An optimizer is an algorithm that adjusts the weights and biases of a neural network to reduce prediction error.

  Examples of optimizers include:

  • Gradient Descent
  • Stochastic Gradient Descent
  • Mini-Batch Gradient Descent
  • Adam
  • RMSprop
  • Adagrad

  SGD is one of the simplest and most important optimization algorithms.

  ---

  3. Complete Neural Network Learning Process

  The learning process follows this sequence:

  Input Data → Feed Forward → Prediction → Loss Calculation → Backpropagation → Weight Update

  Each component performs a different task.

  Component	Role
  Feed Forward	Generates the prediction
  Loss Function	Calculates the prediction error
  Backpropagation	Calculates how much each weight contributed to the error
  Optimizer	Updates the weights
  SGD	One method used to update the weights

  Backpropagation does not update the weights by itself.

  Backpropagation calculates the gradients. The optimizer uses those gradients to update the weights.  Gradient measures how much the loss changes when a weight changes.

4. What Is a Loss Function?

A loss function measures the difference between the actual output and the predicted output.

Suppose:

• Actual output = 1
• Predicted output = 0.30

The prediction is not close to the actual output, so the loss will be high.

After training:

• Actual output = 1
• Predicted output = 0.95

The prediction is close to the actual output, so the loss will be low.

The main objective of neural network training is:

Minimize the loss. 

5. Real-World Example of Loss

Suppose a student aims to score 90 marks but scores only 60 marks.

$$Error = Expected\ Marks - Obtained\ Marks$$$$Error = 90-60=30$$

The difference of 30 represents the student's error.

After additional preparation, the student scores 80 marks.

$$Error = 90-80=10$$

The error decreases from 30 to 10.

Similarly, a neural network repeatedly changes its weights to reduce prediction error. 

6. What Is Gradient?

A gradient indicates:

• The direction in which the loss is increasing.
• How quickly the loss is changing.
• How much a weight should be modified.

In simple words:

The gradient tells the network which direction is wrong and how strongly the weight is affecting the error.

---

7. Real-World Example of Gradient

Imagine standing on a hill and trying to reach the lowest point.

The slope under your feet tells you which direction goes downward.

• A steep slope indicates a large change.
• A gentle slope indicates a small change.
• A flat surface indicates that the minimum may have been reached.

In neural networks:

• Hill height represents loss.
• Current position represents current weights.
• Slope represents gradient.
• Lowest point represents minimum loss.

---

8. What Is Gradient Descent?

Gradient Descent is an optimization algorithm that moves the weights in the direction that reduces the loss.

It is called:

• Gradient because it uses the slope of the loss function.
• Descent because it moves downward toward minimum loss.

The basic objective is:

Move from high error toward low error.

---

9. Weight Update Formula

The basic weight update formula is:

$$w_{new}=w_{old}-\eta \frac{\partial L}{\partial w}$$

Where:

Symbol	Meaning
$w_{new}$	Updated weight
$w_{old}$	Current weight
$\eta$	Learning rate
$L$	Loss function
$\frac{\partial L}{\partial w}$	Gradient of loss with respect to weight

The minus sign is used because the network moves in the opposite direction of increasing loss.

10. Simple Weight Update Example

Suppose:

$$w_{old}=0.50$$
$$Learning\ Rate=0.10$$
$$Gradient=0.20$$

Then:

$$w_{new}=0.50-(0.10\times0.20)$$
$$w_{new}=0.50-0.02$$
$$w_{new}=0.48$$

The weight changes from 0.50 to 0.48.

The updated weight is used during the next Feed Forward pass.

11. What Is the Learning Rate?

The learning rate determines the size of each weight update.

It controls how quickly the model learns.

Common learning-rate values include:

$$0.1,\ 0.01,\ 0.001,\ 0.0001$$

The learning rate is represented by:

$$\eta$$

12. Effect of Learning Rate

Very Large Learning Rate

A very large learning rate causes large weight changes.

The model may:

• Skip the minimum-loss point.
• Become unstable.
• Show increasing and decreasing loss.
• Fail to converge.

Very Small Learning Rate

A very small learning rate causes tiny weight changes.

The model may:

• Learn very slowly.
• Require many epochs.
• Take excessive training time.

Appropriate Learning Rate

An appropriate learning rate provides:

• Stable learning.
• Gradual loss reduction.
• Faster convergence.
• Better model performance.  

13. Real-World Example of Learning Rate

Imagine walking down a mountain.

• Very large steps may cause a person to jump across the lowest point.
• Very small steps will eventually reach the bottom but require more time.
• Moderate steps help reach the bottom safely and efficiently.

The step size represents the learning rate.

---

14. What Is Batch Gradient Descent?

In Batch Gradient Descent, the entire training dataset is processed before one weight update is performed.

Suppose the dataset contains 1,000 records.

The process is:

1. Process all 1,000 records.
2. Calculate the average loss.
3. Calculate gradients.
4. Update the weights once.

Therefore:

One complete dataset produces one weight update.  

15. Batch Gradient Descent Example

Suppose the dataset contains five students.

Student	Prediction Error
Student  1	0.30
Student 2	0.20
Student 3	0.40
Student 4	0.10
Student 5	0.25

Batch Gradient Descent processes all five students, calculates the combined gradient, and then updates the weights once. 

 

16. Advantages of Batch Gradient Descent

• Produces stable gradient calculations.
• Provides smooth loss reduction.
• Uses the complete dataset for each update.
• Works well with small datasets.

---

17. Limitations of Batch Gradient Descent

• Requires more memory.
• Can be slow for large datasets.
• Waits for all records before updating weights.
• One epoch may take a long time.

---

18. What Is Stochastic Gradient Descent?

The word stochastic means random.

Stochastic Gradient Descent updates the network weights after processing only one training record.

Suppose the dataset contains 1,000 records.

The process is:

1. Select one record.
2. Perform Feed Forward.
3. Calculate loss.
4. Perform Backpropagation.
5. Update weights.
6. Select the next record.
7. Repeat the process.

Therefore:

One training record produces one weight update.

---

19. Why Is It Called Stochastic?

Training records are generally shuffled and processed in a random order.

Because the weight update is based on one randomly selected record at a time, the method is called Stochastic Gradient Descent.

It does not use the complete dataset to calculate every update.

---

20. Real-World Example of SGD

Suppose a student is solving ten practice questions.

Batch Learning Approach

The student completes all ten questions and receives feedback only after finishing all of them.

Stochastic Learning Approach

The student solves the first question, checks the mistake, and immediately corrects the learning approach.

Then the student solves the second question, receives feedback, and improves again.

This immediate correction after every question is similar to Stochastic Gradient Descent.  

21. SGD Example with Four Records

Suppose there are four training records.

Record	Actual Output
Record	1
Record 2	0
Record 3	1
Record 4	0

 The SGD process is:

Record 1

• Feed Forward
• Calculate loss
• Backpropagation
• Update weights

Record 2

• Feed Forward using updated weights
• Calculate loss
• Backpropagation
• Update weights again

The process continues for every record.

Therefore, four records produce four weight updates in one epoch. 

22. Advantages of Stochastic Gradient Descent

• Updates weights immediately.
• Requires less memory.
• Suitable for large datasets.
• Can begin learning without processing the complete dataset.
• The random movement may help escape some poor local regions.
• Useful for online and continuously arriving data.

---

23. Limitations of Stochastic Gradient Descent

• Loss may fluctuate frequently.
• Training can appear unstable.
• The path toward minimum loss is noisy.
• It may take many updates to settle near the minimum.
• The final solution may continuously move around the minimum.

---

24. What Is Mini-Batch Gradient Descent?

Mini-Batch Gradient Descent combines the advantages of Batch Gradient Descent and Stochastic Gradient Descent.

Instead of using:

• The entire dataset, or
• Only one record,

it uses a small group of records.

Common batch sizes include:

• 16
• 32
• 64
• 128  

Suppose the dataset contains 1,000 records and batch size is 100.

$$Number\ of\ batches=\frac{1000}{100}=10$$

Therefore, the weights are updated ten times during one epoch.  

25. Real-World Example of Mini-Batch Learning

Suppose a student solves 100 questions.

Instead of checking:

• All 100 questions together, or
• Every single question separately,

the teacher checks ten questions at a time.

After each group of ten questions, feedback is provided.

This represents Mini-Batch Gradient Descent.   

Problem Statement

Breast cancer is one of the most common types of cancer affecting women worldwide. Early and accurate diagnosis plays a vital role in improving treatment outcomes and increasing survival rates. In this practical, a Deep Feed Forward Neural Network (DFFNN) is developed to classify breast tumors as Benign (non-cancerous) or Malignant (cancerous) based on various medical features. The model learns through Feed Forward, Backpropagation, and Stochastic Gradient Descent (SGD) to make accurate predictions.

Go to Jupyter Notebook

Dataset Description

The Breast Cancer Wisconsin Diagnostic Dataset, available in the Scikit-learn library, is used for this practical. It contains 569 patient records, each described by 30 numerical features computed from digitized images of fine needle aspirate (FNA) of breast masses. The target variable has two classes:

• 0 – Malignant (Cancerous)
• 1 – Benign (Non-Cancerous)

The dataset is widely used for binary classification problems and is well suited for demonstrating Deep Learning concepts such as Feed Forward Neural Networks, Backpropagation, and Stochastic Gradient Descent.