Linear and Non-Linear Problems Using Neural Networks

 From Single-Layer Perceptron to Multi-Layer Perceptron: Solving Linear and Non-Linear Problems Using Neural Networks

Introduction

Artificial Neural Networks (ANNs) are inspired by the structure and functioning of the human brain. At the core of every ANN lies a simple computational unit called the Perceptron. A perceptron receives inputs, processes them mathematically using weights and bias, and produces an output.

However, not all problems can be solved using a single perceptron. Some problems follow a linear decision boundary, while others require non-linear decision boundaries. Understanding this distinction is essential before learning Deep Neural Networks.

In this article, we will explore:

• What is a Perceptron?
• Single-Layer Perceptron (SLP)
• Multi-Layer Perceptron (MLP)
• Linear vs Non-Linear Classification
• XOR and XNOR Problems
• Why Hidden Layers are Required
• Real-world applications  

What is a Perceptron?

A Perceptron is the simplest form of an Artificial Neural Network.

It consists of:

• Input neurons
• Weights
• Bias
• Activation Function
• Output neuron

Its purpose is to classify data into different categories.

Basic Structure

Input x₁ ──►

Input x₂ ──►      Perceptron ──► Output
Input x₃ ──►

Mathematically,


Then,
$$Output=f(z)$$
where
w = Weight
b = Bias
$<span\; class="katex"\; style="text-align:\; justify;"><span\; style="font-family:\; arial;"><span\; aria-hidden="true"\; class="katex-html"><span\; class="base"><span\; class="mord\; mathnormal">f</span><span\; class="mopen">(</span><span\; class="mord\; mathnormal">z</span><span\; class="mclose">)</span></span></span>\; =\; Activation\; Function\; </span></span>$
What is a Single-Layer Perceptron (SLP)?
A Single-Layer Perceptron contains only one computational layer between the 
input and output.
Structure
Input Layer

↓

Output Layer
There are no hidden layers.  

Architecture

x₁ ───────┐

x₂ ───────┼────► Output
x₃ ───────┘    

Characteristics of SLP
One output layer
No hidden layer
Uses weights and bias
Suitable only for linear classification
Fast and simple
Limited learning capability   
Real-World Example of Linear Classification
Problem: Student Pass or Fail
Suppose a college decides:
If Marks ≥ 40
Pass
Else
Fail
Dataset
Marks
Result
25
Fail
30
Fail
35
Fail
42
Pass
50
Pass
70
Pass
Plotting these points gives a simple straight-line separation.  
Fail | ● ● ●
---------------------- Decision Boundary
Pass | ▲ ▲ ▲    
A Single-Layer Perceptron can easily solve this problem.
 
Another Linear Example
Loan Approval
Input Features
Salary
Credit Score
Output
Approved
Rejected
If applicants with higher salary and better credit scores consistently receive approval,
 a straight line can separate the two classes.  
What is Linear Classification?
Linear Classification means:
A single straight line can separate the classes.
Mathematically
$$ax+by+c=0$$
Examples of Linear Problems
Pass / Fail
Loan Approval
Spam Detection (simple cases)
Employee Selection
Credit Approval
Customer Purchase Prediction (simple datasets)  
Limitation of Single-Layer Perceptron
Question:
Can every problem be separated using a straight line?
Answer
No.
Some datasets require curved or multiple decision boundaries.
These are called
Non-Linear Problems.   
Example of Non-Linear Classification
Imagine a security system.
Door opens only when
Fingerprint is correct
AND
Face Recognition is correct.
Simple rules are insufficient because the relationship between inputs and outputs is
 more complex.

XOR Problem
The Exclusive OR (XOR) gate is the classic example that demonstrates the limitation of the
 Single-Layer Perceptron.
XOR Truth Table

x₁
x₂
  Output
   0          
0      
0
  0
1
1
  1
0
1
  1
1
0  
 

Why XOR Cannot Be Solved by SLP
A Single-Layer Perceptron can learn only
Linear Decision Boundaries.
XOR requires
Non-Linear Decision Boundaries.
Therefore
Single-Layer Perceptron
Fails.   

Real-Life XOR Example
Suppose a Smart Locker opens only when exactly one of the two authentication methods 
is successful.
XOR means "Only one should be TRUE."
If exactly one input is TRUE, the output is TRUE.
If both inputs are the same (both TRUE or both FALSE), the output is FALSE.

Fingerprint
Face Recognition
Locker
No            
No
   Closed
No
Yes
   Open
Yes
No
    Open
Yes
Yes
   Closed
This is an XOR relationship.
A Single-Layer Perceptron cannot model this behavior.  
University Scholarship (Best Example)
A university has two scholarship schemes:
Sports Scholarship

Cultural Scholarship
Rule:
A student can receive only one scholarship.
Sports Scholarship
Cultural Scholarship
Scholarship Awarded
No
No
No
No
Yes
Yes
Yes
No
Yes
Yes
Yes
No (Cannot receive both scholarships)

XNOR Problem
XNOR means
Outputs are the opposite of XOR. 
XNOR means "Both should be the same."
If both inputs are the same, the output is TRUE.
If the inputs are different, the output is FALSE.

XNOR Truth Table

x₁
      x₂
  Output
0
      0
   1
0
      1
   0
1
      0
   0
1
      1
   1
 

Real-Life XNOR Example
A smart home system controls a light based on two switches.
The light turns ON only when both switches are in the same state.
Switch A
  Switch B
   Light
OFF
   OFF
   ON
OFF
   ON 
   OFF
ON
   OFF
   OFF
ON
  ON
   ON
Again,
A Single-Layer Perceptron cannot solve this problem.
Solution: Multi-Layer Perceptron (MLP)
Scientists realized that one layer was insufficient.
They introduced
Hidden Layers.
Architecture
Input Layer

↓
Hidden Layer
↓
Output Layer   

or
Input

↓
Hidden Layer 1
↓
Hidden Layer 2
↓
Output  


Why Hidden Layers?
Hidden layers allow the network to
Learn complex relationships
Extract useful features
Create non-linear decision boundaries
Solve XOR and XNOR
Recognize images
Understand speech
Process natural language   

Real-Life Example of MLP
Face Recognition
Input
Image
Hidden Layer 1
Learns edges.
Hidden Layer 2
Learns eyes, nose, ears.
Hidden Layer 3
Learns the complete face.
Output
Person identified.
A Single-Layer Perceptron cannot perform this task.   
Comparison: SLP vs MLP

Feature
Single-Layer Perceptron (SLP)
Multi-Layer Perceptron (MLP)
Hidden Layers
No
Yes
Decision Boundary
Linear
Linear and Non-Linear
XOR/XNOR
Cannot Solve
Can Solve
Learning Capability
Limited
High
Complexity
Simple
Complex
Accuracy
Lower
Higher
Applications
Simple Classification
Image Recognition, NLP, Speech Recognition, Medical Diagnosis, Autonomous Vehicles               
          
Applications of Single-Layer Perceptron
Simple Pass/Fail Prediction
Basic Loan Approval
Simple Spam Classification
Binary Decision Systems
Threshold-Based Sensors


Applications of Multi-Layer Perceptron
Face Recognition
Handwritten Digit Recognition
Breast Cancer Prediction
Hotel Booking Cancellation Prediction
Customer Churn Prediction
Speech Recognition
Natural Language Processing
Sentiment Analysis
Recommendation Systems
Autonomous Driving