Understanding the Foundation of Deep Learning

 

Artificial Neural Networks (ANN) Understanding the Foundation of Deep Learning

Artificial Intelligence (AI) has transformed the way machines solve real-world problems. Applications such as ChatGPT, facial recognition, speech assistants, self-driving cars, medical diagnosis, recommendation systems, and fraud detection are all powered by Deep Learning. At the heart of Deep Learning lies one of its most fundamental building blocks—the Artificial Neural Network (ANN).

An Artificial Neural Network is a computational model inspired by the human brain. Just as the human brain consists of billions of interconnected neurons that receive information, process it, and produce a response, an ANN is composed of interconnected artificial neurons that learn patterns from data and make predictions. Rather than being explicitly programmed with rules, an ANN learns by analyzing examples and continuously improving its performance during the training process.

Unlike traditional Machine Learning algorithms, which often require manual feature engineering, Artificial Neural Networks automatically discover complex relationships within data. This ability makes ANNs highly effective for solving classification, regression, image recognition, speech processing, and natural language understanding problems.

Before understanding the mathematical working of an Artificial Neural Network, it is important to understand its basic architecture. An ANN consists of three main layers:

• Input Layer – Receives the input features from the dataset.
• Hidden Layer(s) – Performs computations and learns hidden patterns by applying weights, biases, and activation functions.
• Output Layer – Produces the final prediction or classification.

The learning process of an ANN can be summarized as follows:

1. Input data is fed into the network.
2. Each input is multiplied by its corresponding weight.
3. A bias is added to the weighted sum.
4. An activation function determines whether the neuron should activate.
5. The output is compared with the actual value.
6. The error is calculated.
7. The weights are updated using the Backpropagation algorithm and Gradient Descent.
8. The process is repeated until the model achieves satisfactory accuracy.

In this tutorial, rather than directly using a large real-world dataset, we will first understand the mathematical foundation of an Artificial Neural Network using a simple example. Once the concepts of weights, biases, activation functions, forward propagation, and prediction are clear, we will implement the same workflow on a practical dataset for Hotel Booking Cancellation Prediction using Python and TensorFlow.

Simple Dataset

We will use a small Hotel Booking Cancellation Dataset with only 10 records and 3 input features. This will make the Artificial Neural Network calculation easy to understand.

Problem

Predict whether a hotel booking will be:

• 0 = Not Cancelled
• 1 = Cancelled
  
  

• Sample Dataset for ANN Demonstration

Meaning of the Columns

Column	Symbol	Meaning
Lead Time	$x_1$	Number of days between booking and arrival
Average Room Price	$x_2$	Average room price paid by the customer
Special Requests	$x_{}$	Number of special requests made by the customer
Booking Cancelled	$y$	Target output: 0 = Not Cancelled, 1 = Cancelled

ANN Input and Output

The three input features are:

$${\mathrm{<br>}}_{}$$x1​=Lead Time$${\mathrm{<br>}}_{}$$x2​=Average Room Price$${\mathrm{<br>}}_{}$$x3​=Number of Special Requests

The output is: 

        y=Booking Cancellation

Simple ANN Structure

Input Layer                      Output Layer

Lead Time (x₁)          ───────┐

                              │

Room Price (x₂)         ───────┼──► Artificial Neuron ──► Cancellation

                              │                          0 or 1

Special Requests (x₃)   ───────┘

For the first mathematical calculation, we can select Booking ID 2:

x1​=90

x2​=4500 $${\mathrm{<br>}}_{}$$

Actual output:

y=1

This means the booking was cancelled.

Understanding the Building Blocks of an Artificial Neural Network (ANN)

In the previous section, we prepared a simple hotel booking dataset containing three input features and one target variable. Before performing any mathematical calculations, it is important to understand how an Artificial Neural Network processes these inputs internally.

Every Artificial Neural Network consists of three fundamental components:

• Input Features

• Weights and Bias

• Activation Function

These components work together to help the network learn patterns and make predictions.

What are Weights?

A weight represents the importance given to each input feature.

Just like teachers give different importance to different exam components, an ANN assigns different weights to different features.

For example,

Suppose a company hires students based on:

• CGPA

• Technical Skills

• Communication Skills

Feature	Weight
Lead Time	0.8
Room Price	0.3
Special Requests	0.1

• Lead Time has the highest influence.

• Room Price has moderate influence.

• Special Requests have the least influence.

The company may value Technical Skills more than Communication Skills.

Similarly, an ANN assigns different weights to different inputs.

Example:

Interpretation:

Initially, these weights are assigned randomly.

During training, the ANN automatically adjusts these weights to improve prediction accuracy.

Real-Life Analogy of Weights

Imagine you are selecting the "Best Student of the Year."

You decide the following importance:

Parameter	Importance
Academic Performance	50%
Attendance	30%
Sports	20%

• Higher weight → Greater influence

• Lower weight → Lesser influence 

Here,

50%, 30%, and 20% act like weights.

They indicate how much each factor contributes to the final decision.

Similarly, in an ANN:

What is Bias?

Many beginners think weights alone are enough.

Actually, they are not.

The ANN also uses a Bias.

Definition

Bias is an additional value added to the weighted sum before applying the activation function.

It allows the neuron to shift its decision boundary and improves the flexibility of the model.

---

Real-Life Analogy of Bias

Suppose a university admission rule is:

Admission if marks are 75 or above.

Now the university decides to give 5 grace marks.

A student scoring 72 becomes: 
 72+5 = 77

Now the student gets admission.

Those 5 grace marks act exactly like a Bias.

Bias shifts the decision boundary.

Another example:

Suppose a bank approves loans only if the credit score is 700.

Later, during a festival offer, the bank reduces the eligibility to 680.

The bank changed the decision boundary.

That adjustment is similar to Bias in an ANN.

Difference Between Weight and Bias

Weight	Bias
Determines the importance of each input feature	Shifts the decision boundary
One weight for each input	Usually one bias per neuron
Learns from training	Also learns during training
Multiplies the input	Added after multiplication

 

Why Do We Need an Activation Function?

Suppose the neuron performs the calculation: 

z = 45 

What does 45 mean?

Is the booking cancelled?

Is it not cancelled?

The neuron cannot decide.

It needs a function to convert this score into a meaningful output.

That function is called the Activation Function.

Definition

An Activation Function converts the weighted sum into a meaningful output that helps the neuron make a decision.

Without an activation function, an ANN behaves like a simple linear model and cannot learn complex relationships.

Real-Life Analogy

Suppose a teacher evaluates a student.

The student receives:

• Assignment Marks
• Practical Marks
• Viva Marks

The total is: 77
Now Teacher will decide

Above 75 → Pass 

 Below 75 → Fail

This final decision is similar to the job performed by the activation function.

Why is an Activation Function Important?

It helps the network:

• Learn complex (non-linear) patterns.

• Decide whether a neuron should activate.

• Convert outputs into probabilities or categories.

• Improve prediction accuracy.

Without an activation function, even a deep neural network behaves like a single-layer linear model. 

Types of Activation Functions

Activation Function	Output Range	Common Use
Binary Step Function	0 or 1	Simple threshold-based decisions
Sigmoid	0 to 1	Binary Classification
Tanh (Hyperbolic Tangent)	-1 to +1	Hidden Layers (older networks)
ReLU (Rectified Linear Unit)	0 to ∞	Most commonly used in hidden layers
Leaky ReLU	Small negative value to ∞	Solves the "dying ReLU" problem
Softmax	Probability distribution (sum = 1)	Multi-class Classification
Linear    any real value   Regression

For the first mathematical calculation, we can select Booking ID 2:

x1​=90

x2​=4500

x3​=0

Actual output:

y=1

This means the booking was cancelled.

These three input values are sent to the Artificial Neuron.

Lead Time (x₁ = 90)

            │

Room Price (x₂ = 4500)

            │

Special Requests (x₃ = 0)

            │

            ▼

      Artificial Neuron

At this stage, the neuron has received the inputs, but it cannot make a prediction yet.

The first task is to determine the importance of each input using weights.

Assign Initial Weights

When an Artificial Neural Network starts training, it does not know which feature is more important.

Therefore, it assigns random initial weights to every input.

Suppose the initial weights are: 

Input Feature	Symbol	Initial Weight
Lead Time	(w_1)	0.80
Average Room Price	(w_2)	0.30
Special Requests	(w_3)	0.10

Also, let the neuron have a Bias:

$$b = 0.50$$ Important Note: These weights are not final. They are initialized randomly and will be updated during training using Backpropagation and Gradient Descent.

Multiply Inputs by Their Weights

Each input is multiplied by its corresponding weight. The mathematical operation is: $$x_1 \times w_1$$ $$x_2 \times w_2$$ $$x_3 \times w_3$$ Input Weight Multiplication $90$ $0.80$                         90×0.80    =   72.0 $4500$ $0.30$ $4500 \times 0.30 = 1350.0$ $0$ $0.10$ $0 \times 0.10 = 0$ At this point, the neuron has calculated the contribution of each feature. Notice that the Average Room Price contributes the largest value because its input value is much larger. In practice, this is why feature scaling (normalization or standardization) is important before training an ANN.

Calculate the Weighted Sum

The Artificial Neuron now adds all the weighted values and the bias. The mathematical formula is:  For three input features, the formula becomes:  Substituting the values:  This value $z$ is called the Weighted Sum (also known as the Net Input).

Why Can't We Use $z$ Directly?

The value $$z=1422.50$$ does not tell us whether the booking is Cancelled or Not Cancelled. It is simply a score calculated by the neuron. To convert this score into a meaningful prediction, the neuron applies an Activation Function.  Applying the Activation Function

What is an Activation Function?

An Activation Function is a mathematical function that converts the weighted sum (z) into an output that can be interpreted as a prediction. It determines whether the neuron should become active and pass information to the next layer. Without an activation function, an Artificial Neural Network behaves like a simple linear model and cannot learn complex patterns.

Why Do We Need an Activation Function?

Suppose the neuron calculates $$z=1422.5$$ Can we directly conclude Booking Cancelled? Booking Not Cancelled? No. The value 1422.5 is simply a mathematical score. We need to convert this score into a probability between 0 and 1.

Why is Probability Important?

Consider the following probabilities. Probability Interpretation 0.05 Very unlikely to cancel 0.25 Low chance of cancellation 0.50 Uncertain 0.82 High chance of cancellation 0.99 Almost certain cancellation

The Sigmoid Activation Function

For binary classification problems like Hotel Booking Cancellation Prediction, the most commonly used activation function is the Sigmoid Function. Its mathematical formula is where Symbol Meaning $\sigma(z)$                Sigmoid Output $z$ Weighted Sum $e$ Euler's Number (approximately 2.71828)

Step-by-Step Calculation

From the previous step, $$z=1422.5$$ Substitute this value into the Sigmoid function.

Step 1

Calculate $$-1422.5$$

Step 2

Calculate $$e^{-1422.5}$$ Since this is a very large negative exponent, $${\mathrm{<span\; class="base"><span\; class="mord"><span\; class="mord\; mathnormal">e</span><span\; class="msupsub"><span\; class="vlist-t"><span\; class="vlist-r"><span\; class="vlist"><span\; class="pstrut"></span><span\; class="sizing\; reset-size6\; size3\; mtight"><span\; class="mord\; mtight"><span\; class="mord\; mtight">-</span><span\; class="mord\; mtight">1422.5</span></span></span></span></span></span></span></span><span\; class="mspace"></span><span\; class="mrel">\approx </span>0</span>}}^{}$$

Step 3

Substitute into the formula. $$=\frac{1}{1+0}$$

Step 4

Final Answer $$\sigma(1422.5)=1$$ or approximately $$0.999999999$$

Interpretation

The Artificial Neuron predicts

Probability = 99.99%

This means

The model is almost certain that the booking belongs to Class 1. 

Final Decision

The ANN now applies a decision rule.

For binary classification,

If

$$\sigma(z)\ge0.5$$

then

Prediction = 1

Otherwise

Prediction = 0

Since
$$0.999999>0.5$$
the prediction becomes
Booking Cancelled

Prediction = 1

Evaluation Metric Table
Confusion Matrix