W43L09 - Lecture 9 notes

Machine Learning Model Evaluation and Generalization

Goal: Learn a mapping function F from input features X (p-features) to target variables Y (labels).
Joint Distribution:
- Real-world data comes from a joint distribution P(X, Y), representing all possible data in the universe.
- P(X, Y) decomposes into P(X) (how common an input is) and P(Y|X) (how the label depends on the input).
- Sample Data: We only possess a limited subset (training set) drawn from this joint distribution, not the full distribution.
Expected Risk vs. Empirical Risk:
- Expected Risk: The true error of the model averaged across all possible samples in the universe (the joint distribution). This is theoretically ideal but practically impossible to calculate because P(X, Y) is unknown.
- Empirical Risk: An approximation of expected risk calculated using the error on the available sample training data.
Approximation Quality: The ability of Empirical Risk to approximate Expected Risk depends on:
1. Sampling Quality: The sample must be representative and diverse (covering the whole space). If the sample is biased (e.g., only patients from one country), the empirical risk is a biased estimation.
2. Model Complexity: Higher complexity reduces training error but may increase generalization error (overfitting).

The expected error can be decomposed into three components:

Bias:
- Measures how far the average model predictions are from true values.
- Represents systematic error due to oversimplified assumptions (underfitting).
- High Bias means the model is consistently wrong (e.g., a shooter consistently missing the target in the same spot).
Variance:
- Measures how much the model's predictions vary across different training datasets.
- High Variance implies the model is too sensitive to the specific training noise (overfitting).
- High Variance means the model is inconsistent (e.g., a shooter hitting random spots all over the board).
Irreducible Error: Noise in the data that cannot be removed by modeling.
The Trade-off:
- Low Complexity: High Bias, Low Variance.
- High Complexity: Low Bias, High Variance.
- Goal: Find the optimal complexity where the sum of Bias and Variance is minimized.

Model selection involves choosing the right algorithm type (e.g., Random Forest vs. Logistic Regression) and the best hyperparameters.

Method: Split data into training and testing sets. Train on one, evaluate on the other.
Pros: Simple, fast, suitable for very large datasets (millions of samples).
Cons: High variance in performance estimates (results depend heavily on the specific random split); not suitable for small datasets.

Method: Split data into K folds. Iteratively use K-1 folds for training and the remaining fold for testing. Repeat K times and average the results.
Pros: Maximizes data utilization (good for small datasets); provides an estimate of model variance (stability).
Cons: Computationally expensive (K training runs).
Shuffling: Essential to shuffle data before splitting to ensure classes are distributed and to meet IID assumptions.

The Problem: Using the test set to select hyperparameters causes data leakage. The model overfits to the test set, making the evaluation metric invalid for unseen data.
Solution: Nested Cross-Validation:
- Outer Loop: Splits data into Training and Test sets for final evaluation.
- Inner Loop: Splits the Outer Training set into Training and Validation sets for hyperparameter tuning.
- Process: Optimize parameters in the inner loop, retrain on the full outer training set with best parameters, and evaluate once on the outer test set.

Standard Cross-Validation assumes data is IID (Independent and Identically Distributed).

Problem: Standard random splits may result in folds with zero or few minority class samples.
Solution: Stratified K-Fold:
- Ensures the proportion of target classes (e.g., 80% class 0, 20% class 1) is maintained in every fold.
- Mandatory for imbalanced classification tasks.

Problem: Violates IID. Data within a group (e.g., multiple readings from the same patient) are dependent.
Solution: Group K-Fold (Leave-One-Group-Out):
- Ensures all samples from a specific group (e.g., Patient A) are either entirely in the training set or entirely in the test set.
- Prevents the model from "memorizing" a subject's specific characteristics.

Problem: Violates IID. Data is temporally dependent (today depends on yesterday). Shuffling destroys this structure.
Solution: Time Series Split:
- Uses a sliding window approach.
- Train on historical data (Time 0 to t), test on future data (Time t+1).
- Window expands or slides forward; never shuffle time series data.

Mean Squared Error (MSE): Average of squared differences. Heavily penalizes large errors. Used often as a loss function.
Root Mean Squared Error (RMSE): Square root of MSE. Preferred for interpretation because it shares the same unit as the target variable.
Mean Absolute Error (MAE): Average of absolute differences. More robust to outliers than MSE.
R-Squared (R²): Proportion of variance explained by the model. Range 0 to 1. Should be combined with error metrics for context.

Accuracy: Correct predictions / Total samples. Misleading for imbalanced data (e.g., a trivial model predicting the majority class achieves high accuracy).
Precision: True Positives / (True Positives + False Positives). Relevance of positive predictions.
Recall (Sensitivity): True Positives / (True Positives + False Negatives). Ability to find all positive instances.
F1 Score: Harmonic mean of Precision and Recall.
Balanced Accuracy: Average of Sensitivity (Recall) and Specificity (True Negative Rate).
ROC-AUC (Area Under Receiver Operating Characteristic Curve):
- Performance metric independent of a specific decision threshold.
- Plots True Positive Rate vs. False Positive Rate at all possible thresholds.
- 0.5 = Random guessing; 1.0 = Perfect model.
- Preferred over threshold-dependent metrics (like F1/Accuracy) when the optimal decision threshold is unknown.

Macro Averaging: Calculate metric for each class, then average them. Treats all classes equally (good for highlighting performance on minority classes).
Micro Averaging: Aggregate true positives/negatives globally, then calculate metric. Biased towards majority classes.
Weighted Averaging: Calculate metric for each class, then average them weighted by the number of samples (support) in each class. Preferred for imbalanced datasets.