KNN algo in matlab

Question

I am working on thumb recognition system. I need to implement KNN algorithm to classify my images. according to this, it has only 2 measurements, through which it is calculating the distance to find the nearest neighbour but in my case I have 400 images of 25 X 42, in which 200 are for training and 200 for testing. I am searching for few hours but I am not finding the way to find the distance between the points.

EDIT: I have reshaped 1st 200 images in to 1 X 1050 and stored them in a matrix trainingData of 200 X 1050. similarly I made testingData.

Answer 1

Here is an illustration code for k-nearest neighbor classification (some functions used require the Statistics toolbox):

%# image size
sz = [25,42];

%# training images
numTrain = 200;
trainData = zeros(numTrain,prod(sz));
for i=1:numTrain
    img = imread( sprintf('train/image_%03d.jpg',i) );
    trainData(i,:) = img(:);
end

%# testing images
numTest = 200;
testData = zeros(numTest,prod(sz));
for i=1:numTest
    img = imread( sprintf('test/image_%03d.jpg',i) );
    testData(i,:) = img(:);
end

%# target class (I'm just using random values. Load your actual values instead)
trainClass = randi([1 5], [numTrain 1]);
testClass = randi([1 5], [numTest 1]);

%# compute pairwise distances between each test instance vs. all training data
D = pdist2(testData, trainData, 'euclidean');
[D,idx] = sort(D, 2, 'ascend');

%# K nearest neighbors
K = 5;
D = D(:,1:K);
idx = idx(:,1:K);

%# majority vote
prediction = mode(trainClass(idx),2);

%# performance (confusion matrix and classification error)
C = confusionmat(testClass, prediction);
err = sum(C(:)) - sum(diag(C))

Answer 2

If you want to compute the Euclidean distance between vectors a and b, just use Pythagoras. In Matlab:

dist = sqrt(sum((a-b).^2));

However, you might want to use pdist to compute it for all combinations of vectors in your matrix at once.

dist = squareform(pdist(myVectors, 'euclidean'));

I'm interpreting columns as instances to classify and rows as potential neighbors. This is arbitrary though and you could switch them around.

If have a separate test set, you can compute the distance to the instances in the training set with pdist2:

dist = pdist2(trainingSet, testSet, 'euclidean')

You can use this distance matrix to knn-classify your vectors as follows. I'll generate some random data to serve as example, which will result in low (around chance level) accuracy. But of course you should plug in your actual data and results will probably be better.

m = rand(nrOfVectors,nrOfFeatures); % random example data
classes = randi(nrOfClasses, 1, nrOfVectors); % random true classes
k = 3;  % number of neighbors to consider, 3 is a common value

d = squareform(pdist(m, 'euclidean')); % distance matrix
[neighborvals, neighborindex] = sort(d,1); % get sorted distances

Take a look at the neighborvals and neighborindex matrices and see if they make sense to you. The first is a sorted version of the earlier d matrix, and the latter gives the corresponding instance numbers. Note that the self-distances (on the diagonal in d) have floated to the top. We're not interested in this (always zero), so we'll skip the top row in the next step.

assignedClasses = mode(neighborclasses(2:1+k,:),1);

So we assign the most common class among the k nearest neighbors!

You can compare the assigned classes with the actual classes to get an accuracy score:

accuracy = 100 *  sum(classes == assignedClasses)/length(classes);
fprintf('KNN Classifier Accuracy: %.2f%%\n', 100*accuracy)

Or make a confusion matrix to see the distribution of classifications:

confusionmat(classes, assignedClasses)

Answer 3

yes, there is a function for knn : knnclassify

Play around with the number of neighbors you want to keep in order to get the best result (use a confusion matrix). This function takes care of the distance, of course.