Introduction to Support Vector Machines (SVMs)

• Definition: A Support Vector Machine is a supervised machine learning algorithm that is used for both classification and regression tasks. However, it is primarily used for classification problems.

• Purpose: The main goal of SVM is to find a hyperplane that best separates data into classes. It works well for high-dimensional spaces and is effective when the number of dimensions exceeds the number of samples.

The above image illustrates the concept of Support Vector Machines (SVM), a machine learning algorithm used for classification. Here's a brief explanation:

• Support Vectors: These are the data points (marked in blue squares and green circles) that lie closest to the decision boundary. These points are critical in determining the position of the optimal hyperplane.

• Optimal Hyperplane: This is the straight line (or boundary) that separates the two classes (blue squares and green circles) while maximizing the distance (margin) between the closest points of each class (support vectors).

• Maximized Margin: The distance between the support vectors and the hyperplane is maximized. The goal of SVM is to find the hyperplane that offers the largest possible margin, ensuring a better generalization to unseen data.

Key Concepts

Hyperplane

• Definition: A hyperplane is a decision boundary that separates the data points of different classes. In a 2D space, the hyperplane is a line, and in a 3D space, it's a plane.

• Optimal Hyperplane: SVM finds the hyperplane that maximizes the margin between the two classes, i.e., the distance from the closest points (support vectors) to the hyperplane.

 Margin

• Definition: The margin is the distance between the hyperplane and the nearest data point from each class.

• Maximizing the Margin: SVM selects the hyperplane with the largest margin because this increases the model's ability to generalize to new data.

Support Vectors

• Definition: These are the data points that are closest to the hyperplane. They are critical in determining the position and orientation of the hyperplane.

• Importance: The SVM algorithm focuses on these support vectors because they are the key elements that influence the decision boundary. Other data points do not affect the model as directly.

Types of SVM

Linear SVM

• Used for linearly separable data: Data that can be perfectly separated by a straight line (or a hyperplane in higher dimensions).

• How it works: The algorithm draws the best possible straight line (or hyperplane) that separates the two classes.

Non-Linear SVM

• Used for non-linearly separable data: In real-world problems, most data are not linearly separable. In such cases, a non-linear SVM is used.

• How it works: SVM uses the kernel trick to transform the data into a higher-dimensional space where it becomes linearly separable.

Kernel Functions

Linear Kernel

• Explanation: The simplest kernel, used when the data is linearly separable. It creates a straight line (or hyperplane) to separate the classes.

• Formula:

  $$K(x_i, x_j) = x_i \cdot x_j$$
  

  This is just the dot product of two input vectors

Polynomial Kernel

• Explanation: Useful when the data is not linearly separable but follows a polynomial relationship. It transforms the data into a higher-dimensional polynomial space.

• Formula:

  $$K(x_i, x_j) = (x_i \cdot x_j + c)^d$$
  

  Where $c$ is a constant (usually 1), and $d$ is the degree of the polynomial.

Radial Basis Function (RBF) Kernel (Gaussian Kernel)

• Explanation: A popular kernel for non-linear data. It transforms the data into an infinite-dimensional space, allowing non-linear separations. It’s commonly used when there's no prior knowledge about the data's distribution.

• Formula:

  $$K(x_i, x_j) = \exp\left(-\gamma ||x_i - x_j||^2\right)$$
  

  Where $\gamma$ controls the spread of the kernel.

Sigmoid Kernel

• Explanation: Similar to the activation function in neural networks, this kernel can model complex relationships between inputs. It is less commonly used but can still be effective in specific cases.

• Formula:

  $$K(x_i, x_j) = \tanh(\alpha x_i \cdot x_j + c)$$
  

  Where $\alpha$ and $c$ are constants.

Download Dataset(pima-indians-diabetes.data.csv)