Support Vector Machines Explained

By

What is a Support Vector Machine?
In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. Given a set of training examples, each marked as belonging to one or the other of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier.

 SVM performs classification by finding the hyperplane that maximizes the margin between two classes. The vectors that define the hyperplane are called "support vectors".

 Why is it important to find the optimal hyperplane?
There can be many possibilities when it comes to finding the hyperplane that separates the different classes.
If we compare those 3 hyperplanes in the graph, hyperplanes 1 & 2 are very close the points and the margin/gap between the points are very less. In that case, points can be easily misclassified. Whereas the optimal hyperplane is positioned right at the middle with a larger margin from the support vectors both sides.
It is important to have the largest margin/gap possible when it comes to separating different groups of data.

 NOTE: Here, only the support vectors are important, as these points decides where the hyperplane sits. Other points are ignorable.
Non-Linear Support Vector Machines
Lets look at the example below:
Lets say we are presented with data that is in a straight line (1-dimensional), it is impossible to separate the 2 classes. We can apply a function that separates our data into higher dimensional space (2-dimensional) that makes it easier to draw our hyperplane.

 Lets look at another example:

 The data in 2-dimensional space is inseparable, whereas when it is projected in 3D feature space, we see that there is a clear separation between the groups.
And thus, these 2 groups are separated as shown below.

 Problem:
The only problem in transformation to a higher dimensional feature space is that it is computationally very expensive.

 Solution:
"KERNEL TRICK" : this reduces the computational cost.

 A function that takes input vectors in original space and returns the dot products of the vectors in feature space is called a "kernel function".
Using this kernel function we can apply the dot products between vectors so that every point is mapped to a higher dimensional space via some transformation.

 Advantages:
1. effective in higher dimensional space.
2. different kernels for different decision functions.
3. can add kernel functions together to achieve more complex hyperplanes.

 Disadvantages:
1. if #features > #samples, this can lead to poor performance.
2. do not provide probability estimates.

Comments

Post a Comment

Popular posts from this blog

A/B Testing: Bernoulli Model VS the Beta-Binomial Hierarchical Model

By
Image
What is A/B Testing? A/B testing  is a method of comparing two versions of a webpage or app against each other to determine which one performs better.  AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and based on the data it receives, statistical analysis is used to determine which variation performs better for a given conversion goal. How A/B Testing works? Here, 2 variations of the same webpage or app are created and are randomly assigned to the users (say 50% to Variation A and 50% to Variation B).  As users are served either the A or B, their engagement with each variation is measured and collected and analyzed through a statistical engine. You can then determine whether changing the webpage or app experience had a positive, negative, or no effect on user behavior. Why A/B Testing should be used? A/B testing allows individuals, teams, and companies to make careful changes to their user experie
Continue Reading »

Recurrent Neural Networks and LSTM explained

By
Image
What are Recurrent Neural Networks ? Recurrent Neural Networks are the state of the art algorithm for sequential data  This is because it is the first algorithm that remembers its input, due to an internal memory, which makes it perfectly suited for Machine Learning problems that involve sequential data.   In simple terms, they are networks with loops in them, allowing information to be saved. Here, A is the network, Xt is the input with output  ht.  Although, these loops makes the RNN kind of hard to interpret, but in reality it is very simple. This is how the RNN looks when we unroll them. A RNN can be thought of as multiple copies of the same network , each passing message to  the next. Because of their internal memory, RNN’s are able to remember important things about the input they received, which enables them to be very precise in predicting what’s coming next. This is the reason why they are the preferred algorithm for sequential data like time
Continue Reading »

Decision Tree Explained

By
Image
What is a Decision Tree? A decision tree is a  flowchart -like structure in which each internal node represents a "test" on an attribute (e.g. whether a coin flip comes up heads or tails), each branch represents the outcome of the test, and each leaf node represents a class label (decision taken after computing all attributes).  The paths from root to leaf represent classification rules . Decision tree builds regression or classification models in the form of a tree structure. It divides a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with  decision nodes and leaf nodes. A decision node (e.g., Outlook) has two or more branches (e.g., Sunny, Overcast and Rainy). Leaf node (e.g., Play) represents a classification or decision. The topmost decision node in a tree which corresponds to the best predictor called  root node .  NOTE: Decision trees can handle both catego
Continue Reading »