k-Nearest Neighbors Algorithm Explained

By


What is k-Nearest Neighbors?
K-Nearest Neighbours is one of the most basic yet essential classification algorithms in Machine Learning. It belongs to the supervised learning domain and finds intense application in pattern recognition, data mining and intrusion detection.
KNN is a very simple algorithm that is based on similarity measures such as distance functions.
"k" refers to the number of nearest neighbors to the datapoint.
Algorithm:
A case is classified by a majority vote of its neighbors, with the case being assigned to the
class most common amongst its K nearest neighbors measured by a distance function.
If K = 1, then the case is simply assigned to the class of its nearest neighbor. 
It should also be noted that all three distance measures are only valid for continuous
variables. In the case of categorical variables, the Hamming distance must be used. 
If the groups are similar, then the distance between them will be 0, whereas 1 if not similar.
Example:
Lets consider the plot below, where we need to classify datapoints between 2 classes, i.e Class A and Class B. 
Now, lets find out whether the red star will be classified as Class A or Class B.
1. when 'k' = 3: when the number of nearest neighbors is 3, then the datapoint will be classified to the class with the maximum number of votes, in this case 2 votes. Now, since there are 2 purple and 1 yellow circles inside the 3 nearest neighbors radius, so red star will be classified as Class B.
2. when 'k' = 6: in this case, there are 4 yellow circles and only 2 purple circles so the red star will be classified as Class A.
Note: Sometimes considering an even number for 'k' can cause tie situations, as both the groups will have 50% chances. So, it is better to stick to odd numbers while considering 'k'.

Advantages:
1. robust to noisy data.
2. learns complex models easily.
Disadvantages:
1.need to determine value to 'k'.
2. not clear about which tpe of distance measure to use.
3. computational cost is high.

Comments

Post a Comment

Popular posts from this blog

A/B Testing: Bernoulli Model VS the Beta-Binomial Hierarchical Model

By
Image
What is A/B Testing? A/B testing  is a method of comparing two versions of a webpage or app against each other to determine which one performs better.  AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and based on the data it receives, statistical analysis is used to determine which variation performs better for a given conversion goal. How A/B Testing works? Here, 2 variations of the same webpage or app are created and are randomly assigned to the users (say 50% to Variation A and 50% to Variation B).  As users are served either the A or B, their engagement with each variation is measured and collected and analyzed through a statistical engine. You can then determine whether changing the webpage or app experience had a positive, negative, or no effect on user behavior. Why A/B Testing should be used? A/B testing allows individuals, teams, and companies to make careful changes to their user experie
Continue Reading »

Exploratory Data Analysis and Hypothesis Testing - Loan Prediction

By
Image
Problem Statement: About Company Dream Housing Finance company deals in all home loans. They have presence across all urban, semi urban and rural areas. Customer first apply for home loan after that company validates the customer eligibility for loan. Problem Company wants to automate the loan eligibility process (real time) based on customer detail provided while filling online application form. These details are Gender, Marital Status, Education, Number of Dependents, Income, Loan Amount, Credit History and others. To automate this process, they have given a problem to identify the customers segments, those are eligible for loan amount so that they can specifically target these customers.  Introduction The objective of this project is to do “Exploratory Analysis” and “Hypothesis Testing” on the features of this dataset to find various insights as to how each feature affects the chances of getting a loan. Dataset The Loan Prediction dataset consists of 613 cust
Continue Reading »

Recurrent Neural Networks and LSTM explained

By
Image
What are Recurrent Neural Networks ? Recurrent Neural Networks are the state of the art algorithm for sequential data  This is because it is the first algorithm that remembers its input, due to an internal memory, which makes it perfectly suited for Machine Learning problems that involve sequential data.   In simple terms, they are networks with loops in them, allowing information to be saved. Here, A is the network, Xt is the input with output  ht.  Although, these loops makes the RNN kind of hard to interpret, but in reality it is very simple. This is how the RNN looks when we unroll them. A RNN can be thought of as multiple copies of the same network , each passing message to  the next. Because of their internal memory, RNN’s are able to remember important things about the input they received, which enables them to be very precise in predicting what’s coming next. This is the reason why they are the preferred algorithm for sequential data like time
Continue Reading »