Model Tuning for Machine Learning

Students will learn about model tuning - why it is important, and how it separates real machine learning engineers from beginners, or those just implementing out-of-box libraries.

This lecture will introduce the first topic in model tuning - hyper-parameters. We will go in detail about what they are, how they're used, and how understanding them can result in early, easy wins for the model.

Bayesian reasoning is a powerful tool, especially when it comes to tuning machine learning models in complicated, incomplete environments. In this module we will review bayesian statistics, and discuss how it can be applied to ML models.

Bayesian model averaging is a powerful tool that can be used to drastically increase the performance of ML models. We'll give a theoretical framework for it in this module.

Continuing our delve into Bayesian Model Averaging, we'll go over a practical example, and show how it might be applied to problems you encounter.

Discussing the theoretical difficulties surrounding BMA, we'll show some tricks used to overcome these difficulties, including Occam's Window.

This module finishing up our discussion of the theoretical difficulties with BMA, discussing the difficulties in computing complex integrals, and reviewing some techniques used to either compute or approximate it.

Finally, we'll delve into a real example of BMA and show how it can deliver great results - and set you up to try one yourself!

Bootstrap Aggregation is a key concept in the construction of more sophisticated and powerful machine learning models. We'll discuss it's concept and implementation here.

We'll work an example of bagging using it's application to classification and regression trees, or CARTs. This is a good intro to how bagging is used in the real world.

Why not be content with bagged decision trees? We'll discuss problems with that framework, and suggest a simple way they can be fixed with the intro to boosting.

Random Forests by themselves can be powerful ML tools, so let's first understand how they work, and then show how boosting can make them truly powerful!

One concept to always keep in mine when tuning a model is the time-accuracy tradeoff. At this point in the lecture series, we know enough to illustrate how if we used every tool exhaustively, this can be a prohibitive time investment. Let's review it here!

Building on the framework we have for understanding random forests, let's now delve deeply into how this basic setup can be used to accommodate boosting!

Adaboost is very close to being best-in-class, and it's a simple implementation of boosted trees. This module clearly explains the whats and hows of Adaboost.

The final piece of the puzzle needed to get us to the best-in-class XGBoost algorithm is gradient boosting. We'll explore that concept here, and start to see how it can apply to our algorithm.

Apply everything we've learned, let's see how XGBoost and boosted trees can work on real-world problems.

To truly understand a model, you must know which parts of the algorithm are tuned by the model itself, and which you must adjust. Let's review hyper-parameters, and how important they are to model tuning.

Decision trees are an easily-understandable ML model (and the basis of much of what we've discussed!). Let's look at hyper-parameters in the context of these models.

Understanding tradeoffs is a major part of model tuning. Explore how adjusting a hyper-parameter gives us very different results in a decision tree.

With so many potential values for hyper-parameters, how do we know which to pick? Let's investigate AUC as a measure of model fitness.

How to find the optimal hyper-parameter? Sometimes the best ideas to start with are the simplest - let's consider the pros and cons of the brute force method.

Understanding tradeoffs is a recurring theme. Let's understand the choices we're making, and think how to best find the happy medium that makes for good hyper-parameters.

Model stacking is a competition-winner, suitable for problems that would otherwise be intractable. Let's lay down the foundation and understand the concepts behind it.

The best way to understand is to see the basic concepts in action - here we'll review those concepts, and begin to see their application.

Why should model stacking be able to deliver such dramatic results? Some examples show precisely why this technique is so powerful.

Now that we understand the theory, let's see some of the real-world results model stacking has been able to accomplish.

Let's remember the basics of model tuning - why we need to tune, the first steps, and how it sets us apart from those who don't fully understand it.

Even the most complicated models we saw started with a comprehensible framework. Let's discuss the solid foundation required for effective model tuning.

One of the most important mantras for any data scientist, "know your data" is a crucial step before tuning the data - models should fit the data!

We review the tradeoff that we've been discussing for this lecture series. How do we intelligently decide when the model is fit enough, and the rewards are diminishing?

Finally, let's end with another comprehensive example - this should tie together the course and illustrate our points!