Nearest Neighbor Regression - Data Science Primer

Nearest neighbor

Learning from training data¶

A key concept in machine learning is using a subset of a dataset to train an algorithm to make estimates on a separate set of test data. The quality of the machine learning and algorithm can be assesed based on the accuracy of the predictions made on test data. Many times there are also parameters sometimes termed hyper-parameters which can be optimized through an iterative approach on test or validation data. In practice a dataset is randomly split into training and test sets using sampling.

k nearest neighbor¶

We will examine one machine learning algorithm in the laboratory, k nearest neighbor. Many of the concepts are applicable to the broad range of machine learning algorithms available.

Nearest neighbor concept¶

The training examines the characteristics of k nearest neighbors to the data point for which a prediction will be made. Nearness is measured using several different metrics with Euclidean distance being a common one for numerical attributes.
Euclidean distance:

\begin{align} d(p,q) = \sqrt{(p-q)^{2}} \tag{1} \end{align}

(1)

\begin{align} d(p,q) = \sqrt{(p_1-q_1)^{2}+(p_2-q_2)^{2}} \tag{2} \end{align}

(2)

\begin{align} d(p_i,q_i) = \sqrt{\sum_{i}{((p_i-q_i)^{2})}} \tag{3} \end{align}

(3)

import numpy as np
from datascience import *
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('ggplot')

Distance function inspired from above equations (1-3)¶

def distance(pt1, pt2):
    """The distance between two points, represented as arrays."""
    return np.sqrt(np.sum((pt2-pt1)**2))

Nearest neighbor Functions¶

These cells create the complete algorithm and use as part of a nearest neighbor toolbox

def row_distance(row1, row2):
    """The distance between two rows of a table."""
    return distance(np.array(row1), np.array(row2)) # Need to convert rows into arrays

def distances(training, test, target, features):
    """Compute the distance from test for each row in training."""
    dists = []
    attributes = training.select(features)
    for row in attributes.rows:
        dists.append(row_distance(row, test))
    return training.with_column('Distance', dists)

def closest(training, test, k, target, features):
    """Return a table of the k closest neighbors to example row from test data."""
    return distances(training, test, target, features).sort('Distance').take(np.arange(k))

Prediction Functions¶

def predict_knn(row, train, test, k=5, pr=False):
    """Return the predicting value or class among the 
     k nearest neighbors, pr=1 prints"""
    predict = np.average(closest(train, test.select(features).row(row), k , target, features).column(target[0]))
    if pr:
            print(f'Predicting target value, {target[0]}, for row = {row} using k={k} with features: {features}')
            print(f'Actual value: {test.select(target).take(row)[0][0]:.2f}')
            print(f'Predicted value: {predict:.2f}')
            print(f'Closest neighbor values: {closest(train, test.select(features).row(row), k , target, features).column(target[0])}')
    return predict

def predict_knn_class(row, train, test, k=5, pr=False):
    """Return the predicting value or class among the 
     k nearest neighbors, pr=1 prints"""
    closestclass = list(closest(train, test.select(features).row(row), k , target, features).column(target[0]))
    if pr:
            print(f'Predicting target value, {target[0]}, for row = {row} using k={k} with features: {features}')
            print(f'Actual classification: {test.select(target).take(row)[0][0]}')
            print(f'Predicted classification: {max(closestclass, key=closestclass.count)}')
            print(f'Closest classifications: {closestclass}')
    return max(closestclass, key=closestclass.count)

Regression Functions¶

Use as part of a toolbox for later analysis and the project

def standard_units(any_array):
    "Convert any array of numbers to standard units."
    return (any_array - np.mean(any_array))/np.std(any_array)  
    
def correlation(t, label_x, label_y):
    """Compute the correlation between two variables from a Table with column label_x and label_y.."""
    return np.mean(standard_units(t.column(label_x))*standard_units(t.column(label_y)))

def slope(t, label_x, label_y):
    """Compute the slope between two variables from a Table with column label_x and label_y."""
    r = correlation(t, label_x, label_y)
    return r*np.std(t.column(label_y))/np.std(t.column(label_x))

def intercept(t, label_x, label_y):
    """Compute the slope between two variables from a Table with column label_x and label_y."""
    return np.mean(t.column(label_y)) - slope(t, label_x, label_y)*np.mean(t.column(label_x))

Nearest neighbor regression example¶

We will look at home sales in Ames, Iowa from 2006-2010. The dataset is described here¶

HOUSE=Table().read_table('data/house.csv')
HOUSE

House price prediction¶

Let’s try to predict house price using features available in the extensive dataset.

Define target and features¶

target = ['SalePrice']
features = ['1st Flr SF','Full Bath', '2nd Flr SF','TotRms AbvGrd' ]

sHOUSE = HOUSE.select(target[0])

Standardize¶

for label in features:
    print('Standardizing: ',label)
    sHOUSE = sHOUSE.with_columns(label,standard_units(HOUSE[label]))
sHOUSE

Standardizing:  1st Flr SF
Standardizing:  Full Bath
Standardizing:  2nd Flr SF
Standardizing:  TotRms AbvGrd

Train, test split¶

trainH, testH = sHOUSE.split(int(0.8*sHOUSE.num_rows))
print(trainH.num_rows, 'training and', testH.num_rows, 'test instances.')

trainH.show(3)

2344 training and 586 test instances.

predict_knn(16, trainH, testH, k=8, pr=True)

Predicting target value, SalePrice, for row = 16 using k=8 with features: ['1st Flr SF', 'Full Bath', '2nd Flr SF', 'TotRms AbvGrd']
Actual value: 165250.00
Predicted value: 118412.50
Closest neighbor values: [130000  97000  89000 112000 128900 152400 130500 107500]

118412.5

Test prediction accuracy using specified features¶

k = 5

error = []
for i in np.arange(testH.num_rows):
    predict = predict_knn(i, trainH, testH, k=8, pr=False)
    error.append( predict - testH[target[0]][i])
print(f'Mean signed error: {np.mean(np.array(error)):.2f}')
print(f'Root mean squared error (RMSE): {np.sqrt(np.mean((np.array(error))**2)):.2f}')

Mean signed error: 1971.58
Root mean squared error (RMSE): 50341.06

Pretty good prediction, let’s see if we can do better with additional features

Add additional features¶

target = ['SalePrice']
features = ['Lot Area','1st Flr SF','2nd Flr SF','Full Bath','TotRms AbvGrd', 'Overall Qual' ]

sHOUSE = HOUSE.select(target[0])

for label in features:
    print('Standardizing: ',label)
    sHOUSE = sHOUSE.with_columns(label,standard_units(HOUSE[label]))
sHOUSE

Standardizing:  Lot Area
Standardizing:  1st Flr SF
Standardizing:  2nd Flr SF
Standardizing:  Full Bath
Standardizing:  TotRms AbvGrd
Standardizing:  Overall Qual

Train, test split¶

trainH, testH = sHOUSE.split(int(0.8*sHOUSE.num_rows))
print(trainH.num_rows, 'training and', testH.num_rows, 'test instances.')

trainH.show(3)

2344 training and 586 test instances.

predict_knn(16, trainH, testH, k=8, pr=True)

Predicting target value, SalePrice, for row = 16 using k=8 with features: ['Lot Area', '1st Flr SF', '2nd Flr SF', 'Full Bath', 'TotRms AbvGrd', 'Overall Qual']
Actual value: 187000.00
Predicted value: 166862.50
Closest neighbor values: [143000 163000 187000 180000 173000 192000 136900 160000]

166862.5

Test prediction accuracy using specified features¶

k=5

error = []
for i in np.arange(testH.num_rows):
        predict = predict_knn(i, trainH, testH, k, pr=False)
        error.append( predict - testH[target[0]][i])
print(f'Mean signed error: {np.mean(np.array(error)):.2f}')
print(f'Root mean squared error (RMSE): {np.sqrt(np.mean((np.array(error))**2)):.2f}')

Mean signed error: 226.88
Root mean squared error (RMSE): 29610.18

Improved...