Model Assessment in Machine Learning

An introduction to the instructor and an overview of the course.

Overview of what to expect from module one. You will know why model assessment is important.

Although you may already know the difference between supervised and unsupervised learning, this lecture will refresh your memory by discussing types of models used for each learning methodology.

Learn how to recognize bias in your machine learning models.

Learn the signs of variance and what it means to have over fitting in your model.

Quiz yourself on the model assessment terms learned in this module.

Understand the reasons for validating your model. Also, you will be introduced to some common methods of validating your model.

Learn the most basic type of validation and the benefits of splitting your data in two.

Learn how to validate your training using K-Fold Cross Validation. The demonstration will use 10-Fold cross validation. A common use of K-Fold cross validation.

Another method of validation you can add to your tool belt. You will learn how to achieve this type of validation using Python.

Sometimes you may not have enough data to create a proper model. Bootstrapping is one method you can utilize to attack this problem.

Answer different questions pertaining to model validation.

The Jupyter notebooks used in the demonstrations for the validation demonstrations.

Perform 10-Fold cross validation on the HR data set and Leave One Out Cross Validation for the HR data set already imported. Which type of validation provides better performance during training?

You will be introduced to the metrics and methods used to evaluate unsupervised learning models.

Learn how the distance properties of cluster nodes can be a sign of how well you are clustering giving the correct hyperparameters.

Learn how to calculate this metric in Python that can additionally be used to determine how well you are separating your data types into different clusters.

Using this measure you can evaluate your unsupervised learning model in a manner similar to validation of supervised learning models. This lecture uses a Latent Dirichlet Allocation (LDA) model for the demonstration.

Test your knowledge on assessing unsupervised learning models.

The Jupyter notebooks with the code for the unsupervised learning assessment demonstrations.

For the diabetes data found in the Jupyter notebook, calculate the silhouette scores. What value for k would you recommend?

These are the assessment methods that will be discussed for supervised learning methods in this model.

You will get an overview the metrics that you can use to evaluate regression models. In this lecture we will begin with a demonstration of the Aiken Information Criteria (AIC).

Although similar to the AIC, you learn the difference between the AIC and BIC. You also learn how to calculate this metric using Python.

Find the least complex regression model by translating your model complexity into a length.

Create a visual representation of your models performance results using tabular data.

Probably the most common method of determining how well your supervised learning model performs. Learn to calculate this by hand in Python using the confusion matrix data.

Also know as the true positive rate. Learn how to calculate the metric that will give you some insight on how well your supervised learning model predicts the expected true values.

Also know as the true negative rate. Learn how to calculate the metric that will give you some insight on how well your supervised learning model predicts the "False" class.

We demonstrate how to calculate the positive predicted value. This will give you a measure that pertains to predicted true values.

Also known as the harmonic mean. In addition to the prediction accuracy, this will indicate how well you model performs. In this lecture you will be able to calculate the metric with the confusion matrix values.

You will learn how to create this powerful visualization of your classifier performance. In conjunction with the confusion matrix this tool will be a necessity in evaluating your classifier's performance. Plotting the curve is one of the tasks you will be performing in this lecture.

This is an additional metric will help you evaluate how well you classifier compares to the observed/labelled data. In this demonstration you will be able to calculate this metric by hand.

Learn how skewed data can make for a deceiving prediction accuracy. This metric will take imbalanced data sets into consideration. You will be able to calculate this metric using Python in this lecture.

Test your skills on the metrics and tools used for the evaluation of regression and classifier models.

The supervised learning Jupyter notebook demonstrations.

Create a confusion matrix for the Diabetes data in the attached homework for a Random Forest Classifier. In addition calculate the Matthew's Correlation Coefficient and Prediction Accuracy. Calculate the Prediction Accuracy for the included SVM. Compare the metrics between the SVM and the Random Forest Classifier. Using your judgment, which model performs better?