Introduction to Machine Learning

by Albert Major

Introduction to Ensemble Models

• Definition: Ensemble models combine multiple different machine learning models to reach a prediction outcome that is better than any of the individual models alone.
• Goal: To create a stronger, more robust model by aggregating the results of several "weaker" or simpler base models.

Bias and Variance Trade-off

Understanding error is crucial for ensemble methods. Error consists of Bias + Variance + Irreducible Error (noise in the data).

• Bias (Underfitting):
  • Occurs when a model is too simple to capture the underlying structure of the data (e.g., fitting a straight line to a complex curve).
  • High Bias: The model misses relevant relations between features and target outputs. It is consistent but consistently wrong.
• Variance (Overfitting):
  • Occurs when a model is overly complex and captures noise in the training data along with the signal (e.g., a "jacket" fitted so tightly to one person it fits no one else).
  • High Variance: The model fluctuates drastically depending on the specific dataset used for training.
• The Trade-off:
  • Simple models generally have high bias and low variance.
  • Complex models generally have low bias and high variance.
  • Ensemble Goal: To combine models to achieve both low bias and low variance simultaneously.

Methods of Combining Models

1. Voting (Parallel Combination)

Used when running different models (e.g., SVM, Logistic Regression, KNN) independently on the same data.

• Hard Voting (Majority Rule):
  • Used for classification.
  • The final prediction is the class chosen by the majority of the models.
  • Example: If SVM and KNN predict "Spam" and Logistic Regression predicts "Not Spam," the result is "Spam."
• Soft Voting (Probability/Confidence):
  • Takes into account how confident each model is in its prediction (probability estimates).
  • Higher weight is given to models with higher confidence (e.g., a 95% probability carries more weight than a 51% probability).
• Averaging:
  • Used for regression tasks.
  • Calculates the numerical average of the outputs from all models.

2. Bagging (Bootstrap Aggregating)

Bagging aims to reduce variance. It involves training multiple models of the same type in parallel on different versions of the dataset.

• Bootstrapping:
  • The process of creating subsets of the original dataset by random sampling with replacement.
  • This creates statistically similar but distinct datasets.
• Process:
  • Train independent models on these bootstrap samples.
  • Aggregate the results (averaging for regression, voting for classification).
• Independence: The models do not learn from each other; they run completely separately.

3. Random Forest (Specific Bagging Method)

Image of random forest structure

Random Forest is an extension of bagging applied specifically to decision trees.

• Mechanism: It builds a "forest" of many decision trees.
• Double Randomization:
  1. Data: Each tree is trained on a random subset of data (bootstrapping).
  2. Features: At each split in the tree, only a random subset of features is considered (not all features).
• Benefit: This decorrelates the trees. Individual trees might overfit, but when averaged, the variance is drastically reduced without increasing bias.
• Scaling: No feature scaling is required for decision trees (distance doesn't matter, only thresholds).

4. Boosting

Image of boosting vs bagging

Boosting aims to reduce bias. It involves training models sequentially, where each new model attempts to correct the errors of the previous one.

• Sequential Process: Models are not independent. The output of Model A influences the training of Model B.
• AdaBoost (Adaptive Boosting):
  • Reweighting: Focuses on data points that were misclassified by the previous model.
  • Mechanism: It increases the weight of wrongly classified instances so the next model is forced to focus on them.
  • Learners: typically uses very simple models called "weak learners" (e.g., stumps/shallow trees).
• Gradient Boosting:
  • Residual Fitting: Instead of reweighting data, the new model is trained on the residuals (errors) of the previous model.
  • Logic:
    1. Make an initial prediction (e.g., the average).
    2. Calculate the error (Residual = Actual - Predicted).
    3. Train a new model to predict this error.
    4. Update the prediction by adding the new model's output (scaled by a learning rate).
  • Learning Rate: Controls the size of the steps taken toward minimizing error. Large steps risk oscillation; small steps require more iterations (trees) but are more precise.
  • Analogy: Similar to Gradient Descent, but in function space rather than parameter space.

5. Stacking

Stacking uses a "meta-learner" to combine predictions.

• Architecture:
  1. Base Learners: Train different models (Level 0) on the data.
  2. Meta Learner: Use the predictions of the base learners as the input features for a final model (Level 1).
• Learning: The meta-model learns how to best combine the outputs of the base models to predict the final target.
• Data Splitting: Requires careful data management (e.g., holding out data) to ensure the meta-model isn't trained on the same data used to train the base models, avoiding data leakage.