Shiou-Shiou Deng This homework only uses the MNIST …gchen/Math285S16/HW1-Shioushiou.pdf · This homework only uses the MNIST Digits dataset (both training and test data). ... 8

Shiou-Shiou Deng

This homework only uses the MNIST Digits dataset (both training and test data).

Question 1: Implement the plain kNN classifier and apply it with 6-fold cross validation to the training set to select the best k in the range 1:10.

The curve of validation error versus k From the plot, we could find

the best k is 3 because there is a lowest error rate on both validation error and test error. ( The test error is 0.0294. The validation error is 0.0273.)

Comparing to the test error curve, the curve of validation error is consistent with the test error curve in most of k situations.

In general, the test errors are all higher than the validation errors on every k situation.

Question 2: For the optimal k found above, perform kNNclassification on the test set.? Display the error rate and the

confusion matrix.

0 1 2 3 4 5 6 7 8 9

0 974 1 1 0 0 1 2 1 0 0

1 0 1133 2 0 0 0 0 0 0 0

2 10 9 993 2 1 0 0 15 2 0

3 0 2 4 974 1 15 1 7 3 3

4 0 6 0 0 951 0 4 2 0 19

5 4 1 0 9 2 863 5 1 3 4

6 4 3 0 0 4 3 944 0 0 0

7 0 21 4 0 1 0 0 992 0 10

8 5 3 4 11 8 15 6 4 914 4

9 3 5 1 6 9 5 1 9 2 968

The test error is 0.0294. The left is the confusion

matrix. The classifier would tell the correct answers on most cases.

The classifier would be confused when classifying 2 to 7, 3 to 5, 4 to 9, 7 to 1, and 8 to 5.

Question 3: apply the kNN classifier with the city-block distance metric to the test set. Show the test errors curve corresponding to the different k.

The curve of validation error versus k From the plot, applying the kNN classifier with the city-block distance, we found there is the same and the lowest test error on k = 1 and k = 3. (The test error is 0.0369.)

Comparing to the test error curve, the curve of the city-block distance is consistent with the curve of the Euclidean distancein most of k situations.

In general, the test errors of the city-block distance are higher than the test errors of the Euclidean distance.

Question 4: Implement the weighted kNN classifier with linearly decreasing weights, for each k = 3, . . . , 12 and apply it directly to the test set.

The curve of test error versus k Use Linear Weight = /

in order to make the k-th nearest point has weight.

From the plot, applying the weighted kNN classifier on the test set, we found the lowest test error is 2.76% when k = 7.

Comparing to the test errors of other weighted kNN for k = 3, the Linear weighted kNN is the highest error. the test error is 3.02%. (Inverse: 2.83%, Squared inverse: 2.83%)

Comparing to the test error curve, the curve of weighted kNN is not consistent with the curve of the kNN classifier. The curve of weighted kNN is not consistent with the curve of local kmeans, either.

In general, the test errors of the local kmeans classifier are lower than the test errors of other classifiers.

Method: The local kmeans classifier Step1: Use the k nearest points in each class to represent the

corresponding class.for i = 0:9

labelIndex = find(trainLabels==i);Indices = knnsearch(trainImagesT(labelIndex,:), testImagesT,'k', k);Indices = labelIndex(Indices);trainImages_reduced = reshape(trainImagesT(Indices', :)', 784, k, 10000);center(:, i+1, :) = squeeze(mean(trainImages_reduced, 2))';

end

Step2: Perform kmeans classification based on the reduced data.for k = 1:10

for m = 1:10000pred_local = knnsearch(squeeze(center(m, :, :)), testImagesT(m, :)) - 1;LocaltestError(k) = LocaltestError(k) + (testLabels(m) ~= pred_local);

endend

Question 5: apply the local kmeans classifier on the test set. Show the test errors curve corresponding to

the different k.

Question 5: apply the local kmeans classifier on the test set. Show the test errors curve corresponding to

the different k.

The curve of test error versus k From the plot, applying the kmeans classifier on the test set, we found there is apparently decreasing error rate when k = 3. (test error = 0.0235) The lowest test error is 0.0176 when k = 9.

Comparing to the test error curve, the curve of Local kmeansis not consistent with the curve of the kNN classifier in most of k situations. ( Only 1NN = 1means.)

In general, the test errors of the kNN classifier are higher than the test errors of the Local kmeans classifier.