Supervised Learning: Classification

In this lecture we will look at what machine learning is, and look at some examples of its application in our everyday life.

In this lecture, we will define what the classification problem is within Machine Learning, see some examples and briefly outline the models we will be looking at.

In this lecture, we review the metrics that are commonly used for measuring model performance, and provide pointers to additional resources to complement the course material.

In this lecture, we will get familiar with the programming language and environment we will be using, as well as some of the data sets we will use to illustrate the classification models we will be exploring in this course.

Observe the relationships between the features in the Seeds data and identify two features that you consider most likely to provide good separation between the classes.

In this lecture, we will explore the K Nearest Neighbor classifier. We look at the underlying intuition, and the parameters needed to obtain the model. We present the classification algorithm and illustrate this approach using the Iris data set with different parameters and observe the results. We will then review the pros and cons of using this approach.

Create test data by using a portion of the Iris data set for testing. Use the remaining data for training. Measure test accuracy for the models created in class and compare them to the training accuracy.

In this lecture we explore another classification model – the Decision Tree. Decision trees are used for regression as well as classification. You will learn the basic characteristics of decision trees and how they may be applied to classification problems.

We now move to exploring a number of parametric models for classification. First we will review Bayes’ theorem and how it is applied in arriving at the conditional probabilities we are attempting to estimate for our classifiers. We will then look at the Naive Bayes classifier, reviewing the underlying assumptions, the model, its application to the Iris dataset.

In this lecture, we will look at two parametric models - Linear Discriminant Analysis and Quadratic Discriminant Analysis. We will look at how these models differ from Naive Bayes by reviewing their underlying assumptions. We will then apply these models to a sample dataset and review their pros and cons.

In this exercise, we will compare the performance of the 3 probabilistic classifiers introduced in this module - NB, LDA and QDA - on the Seeds data set.

In this section we will explore Logistic Regression, another probabilistic classification model that is widely applied to binary classification problems. We will learn about the Logistic Function and the role it plays in the Logistic Regression Model. We will examine what is involved in learning the Logistic Regression Model from training data and apply it to a sample dataset.

In this exercise, you will build and evaluate a logistic regression model for the seeds dataset. We combine two of the target classes reducing the problem to one of binary classification. You will build and evaluate the logistic regression model for different values of the parameter C.

We will look at three closely related classifier models – the maximum margin classifier, the support vector classifier, and support vector machines. We will examine the underlying concepts and how the classifier models are learnt from training data. We discuss basis expansion and Kernel functions and how they are used to extend the model to handle non-linear decision boundaries.

In this final lecture, we will look at what it means to arrive at the best model for the problem at hand. Specifically, we will review the concepts of model capacity, generalization, capacity control, overfitting, underfitting, bias, variance, and the bias-variance tradeoff. We then examine hyperparameters and regularization, and their role in capacity control.

In this short project, you will work with the Income dataset. The dataset contains information about adults including age, level of education, class of work, marital status, occupation, and nationality. The objective is to predict the income category as either > $50K or <= $50K. You will look at 3 different classifier models and compare them in terms of their test accuracy, training time, and test time.