Statistical Learning

Module 2: Linear Regression Methods

Welcome to Module 2 of Math 569: Statistical Learning. Here, we explore what is arguably the foundational model of the field: linear regression. This simple yet highly useful model helps us better understand the statistical learning problem discussed in Module 1. In Lesson 1, we'll carefully review what linear regression aims to do, how we construct the model's parameters with a given dataset, and what kinds of statistical tests we can perform on our estimated coefficients. In Lesson 2, we’ll cover a method known as Subset Selection, which aims to improve linear regression by eliminating unimpactful independent variables. In Lesson 3, we explore introducing bias into the linear regression model with two regularization methods: Ridge Regression and LASSO. These methods utilize a hyperparameter, a key concept in this course, to limit the growth of the coefficients. This is the source of the bias and will help us understand why a biased estimator can outperform our unbiased estimator for the coefficients of linear regression in Lesson 1. Finally, Lesson 4 introduces the concept of data transformations, which allow one to address complexities within a dataset. It also provides a simple way of converting a linear model to a nonlinear model.

Module 3: Linear Classification Methods

Welcome to Module 3 of Math 569: Statistical Learning, where we delve into linear classification. In Lesson 1, we explore how linear regression, typically used for predicting continuous outcomes, can be adapted for classification tasks-predicting discrete categories. We'll cover the conversion of categorical data into a numerical format suitable for classification and introduce essential classification metrics such as accuracy, precision, and recall. In Lesson 2, we'll explore Linear Discriminant Analysis (LDA) as an alternative method for constructing linear classifications. This method introduces the notion that classification maximizes the probability of a category given a data point, a framing we will revisit later in the course. Maximizing the likelihood of classification, given some simplifying assumptions, leads to a linear model that can also reduce the dimensionality of the problem. Finally, in Lesson 3, we will cover logistic regression, which is constructed by assuming the log-likelihood odds are linear models. The outcome, similar to LDA, produces a linear decision boundary.

Module 4: Basis Expansion Methods

Welcome to Module 4 of Math 569: Statistical Learning, focusing on advanced methods in statistical modeling. This module starts with an introduction to Basis Expansion Methods, exploring how these techniques enhance linear models by incorporating non-linear relationships. We then delve into Piecewise Polynomials, discussing their utility in capturing varying trends across different segments of data. In Lesson 2, we explore Smoothing Splines, emphasizing their role in effectively balancing model fit and complexity. Lastly, Lesson 3 covers Regularization and Kernel Functions, elaborating on how these concepts contribute to constructing more complex models without significantly increasing computational complexity.

Module 5: Kernel Smoothing Methods

Welcome to Module 5 of Math 569: Statistical Learning, dedicated to advanced techniques in non-linear data modeling. In Lesson 1, we delve into Kernel Smoothers, exploring how they make predictions based on local data and their comparison to k-Nearest Neighbors (kNN) models. Lesson 2 focuses on Local Regression, particularly Local Linear Regression (LLR) and Local Polynomial Regression (LPR). We'll examine how LLR overcomes some kernel smoothing limitations and how LPR provides flexibility in capturing local data structure. The module emphasizes the adaptiveness of these techniques for complex data relationships and addresses the challenges in selecting hyperparameters and computational demands, especially for large datasets.

Module 6: Model Assessment and Selection

Module 6 of Math 569: Statistical Learning delves into model evaluation and model selection via hyperparameter choice. It begins with an understanding of Bias-Variance Decomposition, highlighting the trade-off between model simplicity and accuracy. The module then explores model complexity, offering strategies for balancing this complexity with predictive performance. Building on the importance of balancing model complexity with performance, we move on to cover model selection metrics, namely: AIC, BIC, and MDL. These are information-theoretic metrics that balance error with model complexity, such as the number of parameters. Finally, the module concludes with lessons on estimating test error without a testing set, using concepts like VC Dimension, Cross-Validation, and Bootstrapping. This module is pivotal for mastering model evaluation and selection in statistical learning.

Module 7: Maximum Likelihood Inference

Module 7 of Math 569: Statistical Learning introduces advanced inferential techniques. Lesson 1 focuses on Maximum Likelihood Inference, explaining how to find optimal model parameters by maximizing the likelihood function. This method is pivotal in estimating parameters for which a dataset is most likely. Lesson 2 dives into Bayesian Inference, contrasting it with frequentist approaches. It covers Bayes' Theorem, which integrates prior beliefs with new evidence to update beliefs dynamically. The module thoroughly discusses the process of Bayesian modeling, including the construction and updating of models using prior and posterior distributions. This module is crucial for understanding complex inference methods in statistical learning.

Module 8: Advanced Topics

Module 8 of Math 569: Statistical Learning covers diverse advanced machine learning techniques. It begins with Decision Trees, focusing on their structure and application in both classification and regression tasks. Next, it explores Support Vector Machines (SVM), detailing their function in creating optimal decision boundaries. The module then examines k-Means Clustering, an unsupervised learning method for data grouping. Finally, it concludes with Neural Networks, discussing their architecture and role in complex pattern recognition. Each lesson offers a deep dive into these techniques, showcasing their unique advantages and applications in statistical learning.