Long Short-Term Memory (LSTM) Networks

1. Introduction

Recurrent Neural Networks (RNNs) are designed to process sequential or time-series data by remembering information from previous time steps. They work well for short sequences but struggle when the sequence becomes longer.

Long Short-Term Memory (LSTM) is an advanced type of RNN that overcomes this limitation by learning what information should be remembered, updated, and forgotten over long periods.

LSTM is a specialized Recurrent Neural Network that can capture both short-term and long-term dependencies in sequential data. 

2. Why Do We Need LSTM?

Suppose we want to predict today's minimum temperature.

The previous few days influence today's temperature.

Day	Temperature (°C)
Monday	18.2
Tuesday	19.1
Wednesday	20.5
Thursday	21.0
Friday	?

An RNN uses previous temperatures, but as the sequence becomes longer, it gradually forgets older information.

Real-life Examples

• Weather forecasting

• Stock price prediction

• Electricity demand forecasting

• Speech recognition

• Language translation

• Next-word prediction

These problems require remembering information from many previous time steps, making LSTM more suitable than a basic RNN. 

3. Limitation of RNN

RNN stores information in its hidden state.

As sequences become longer, the hidden state gradually loses important information.

The main problem is called the Vanishing Gradient Problem.

During backpropagation, gradients become extremely small, so the network cannot effectively learn long-term relationships.

RNN remembers recent information but gradually forgets distant information.

4. What is LSTM?

LSTM introduces a new component called the Cell State, which acts as a long-term memory.

Instead of remembering everything, LSTM intelligently decides:

• What should be remembered?

• What should be forgotten?

• What new information should be stored?

• What should be sent to the next time step?

How LSTM Works

Step 1: Forget Gate

The forget gate decides what information should be discarded from the previous memory.

Example:

If very old temperature values are no longer useful, they receive less importance.

Purpose: Remove irrelevant historical information.

Step 2: Input Gate

The input gate decides what new information should be stored.

Example:

Today's temperature becomes important for predicting tomorrow.

Purpose: Store useful new information.

Step 3: Cell State Update

The previous memory and new information are combined to create an updated long-term memory.

Purpose: Maintain important historical knowledge.

Step 4: Output Gate

The output gate determines what information should be sent to the next LSTM cell.

This hidden output is also used for prediction.

Purpose: Generate the prediction while preserving memory.

Why LSTM is Better for Temperature Prediction?

In the Daily Minimum Temperature dataset:

• Tomorrow's temperature depends on previous days.

• Some previous days are more important than others.

• LSTM automatically learns which historical values should influence the prediction.

Example

Previous 7 days:

18.2, 19.1, 20.5, 18.9, 17.8, 19.6, 21.0

LSTM learns:

• Which temperatures are important

• Which values can be ignored

• How historical patterns affect the next day 

Output: 

Predicted Temperature = 20.8°C

Applications of LSTM

Application	Prediction
Weather Forecasting	Next day's temperature
Stock Market	Future stock price
Electricity	Future demand
Healthcare	Patient vital-sign trends
Finance	Revenue forecasting
NLP	Next-word prediction
Speech	Speech recognition

 

Advantages of LSTM

• Captures long-term dependencies

• Solves the vanishing gradient problem better than RNN

• Suitable for sequential and time-series data

• Provides better forecasting accuracy

• Widely used in weather, finance, healthcare, and NLP  

Download Dataset 

Minimum Temperature Prediction Using LSTM 
Go Through Notebook  

Do We Use Gradient Clipping?

During neural-network training, the model follows this process:

Forward Pass → Prediction → Loss → Backpropagation → Gradient Calculation → Weight Update

The gradient tells the optimizer how much and in which direction the weights should change to reduce the loss.

Normal Situation

Suppose:

• Weight = 0.5
• Learning rate = 0.001
• Gradient = 2

Then:

$$W_{new}=W_{old}-(LearningRate \times Gradient)$$ $$W_{new}=0.5-(0.001\times2)=0.498$$

The weight changes only slightly:

0.500 → 0.498

This is a controlled update.

---

What is the Exploding Gradient Problem?

In RNN/LSTM, gradients are propagated backward through multiple time steps during Backpropagation Through Time (BPTT).

Sometimes these gradients can become extremely large.

For example:

Gradient = 2
Gradient = 20
Gradient = 200
Gradient = 1000

Now suppose:

• Weight = 0.5
• Learning rate = 0.001
• Gradient = 1000

Then:

$$W_{new}=0.5-(0.001\times1000)$$ $$W_{new}=-0.5$$

The weight suddenly changes:

0.5 → -0.5

Such very large weight updates can make training unstable.

This is called the:

Exploding Gradient Problem

---

What Does Gradient Clipping Do?

Gradient Clipping places a limit on the magnitude of the gradients before the optimizer updates the weights.

LSTM calculates prediction

          ↓

Calculate Loss

          ↓

Backpropagation

          ↓

Calculate Gradients

          ↓

Are gradients too large?

          ↓

   Gradient Clipping

    (clipnorm = 1.0)

          ↓

Adam Optimizer

          ↓

Controlled Weight Update   

Gradient Clipping is mainly used to prevent excessively large gradients and make neural-network training more stable. If the original model is already stable, clipping may produce little or no improvement in prediction accuracy.

Vanishing → Gradient too small → difficult learning
Exploding → Gradient too large → unstable learning
Clipping → Controls excessively large gradients.