Introduction to Machine Learning

by Albert Major

Part 1: Data Scaling and Transformation Review

The lecture began by reviewing why we scale data. Machine learning models often use distance-based optimization. If features have different units (e.g., age vs. salary), the feature with the larger range will dominate the model. Scaling standardizes these features.

Data scaling transforms your features into a shared numerical range so that variables with large values do not unfairly dominate the model's learning.

• Standard Scaler

  

  • Mechanism: Subtracts the mean and divides by the standard deviation.

  • Formula: z = (x - u) / s

    

  • Effect: Centers the data around 0 with a standard deviation of 1.

  • Use Case: Good for data that follows a normal distribution (Gaussian).

  • Weakness: Sensitive to outliers because the mean is affected by outliers.

• Robust Scaler

  

  • Mechanism: Uses the median and the Interquartile Range (IQR).
  • Effect: Centers data around the median rather than the mean.
  • Use Case: Better for datasets containing significant outliers or skewed data, as the median is not skewed by extreme values.

• Min-Max Scaler

  

  • Mechanism: Scales data to a fixed range, usually between 0 and 1.
  • Use Case: When you need bounded values (e.g., for neural networks or image data).

• Normalizer

  • Mechanism: Scales individual samples (rows) so that their vector norm (length) equals 1.
  • Use Case: Used when the direction of the data vector matters more than the magnitude.

• Univariate Transformations

  • Goal: Most machine learning models perform best with Gaussian distributed data (bell curve), so we transform skewed data to fit this shape.
  • Examples: Simple methods include logarithmic, geometric, or power transformations.
  • Automated Methods:
    • Box-Cox & Yeo-Johnson Transforms: These are advanced methods that automatically estimate parameters to minimize skewness and stabilize variance.
    • Visual Effect: These transforms take skewed histograms (tails to the left or right) and make them centered and bell-shaped.

• Binning

  • Definition: Separating feature values into discrete categories (bins), often equally spaced over the range of values.

  • Process: Replace all values within a bin with a single value, such as the mean of that bin.

  • Use Case: Effective for simple linear models (like regression) but generally not useful for complex models like decision trees.

    

Part 2: Generalization and Model Validation

The core goal of machine learning is generalization: learning patterns from training data to predict accurately on new, unseen data.

• Overfitting vs. Underfitting

  • Underfitting: The model is too simple to capture the underlying structure of the data. It performs poorly on both training and test sets.

  • Overfitting: The model memorizes the training data, including noise. It has high accuracy on the training set but performs poorly on the test set.

  • The Goal: Find the "sweet spot" between these two extremes where the model generalizes well.

    

    The image compares a "Raw data" set on the left, which has scattered values spanning a wide range, against the same data on the right after it has been transformed to cluster tightly around the center point of zero. This demonstrates the effect of scaling (likely the Standard Scaler), which condenses the data into a standardized range to prevent large values from dominating the model while preserving the original shape of the cluster.

• The Complexity Curve:

  • As model complexity increases, training accuracy always continues to rise.
  • However, generalization (test) accuracy only rises to a "sweet spot" before dropping due to overfitting.
  • Cross Validation: Used to detect overfitting or selection bias by evaluating the model on multiple different splits of the data.

• Data Splitting

  • Train/Test Split: You must split data to evaluate performance. The model learns from the Training Set and is evaluated on the Test Set.

• Subject-wise Splitting (Crucial Example):
  • If you have a medical dataset (e.g., knee X-rays) with multiple visits (Visit 0, Visit 1) for the same subjects (A, B, C, D, E).
  • Correct: Put all images from subjects A, C, D in Training and all images from B, E in Testing.
  • Wrong (Random Splitting): Mixing different images from Subject A into both Training and Testing creates data leakage and false confidence.

This image contrasts two data splitting methods: "Subject Wise Splitting," which keeps all data from specific individuals (like subjects B and E) exclusively in the test set, versus "Random Splitting," which mixes samples from the same individuals across both training and testing. The visual demonstrates that subject-wise splitting is necessary to prevent data leakage, ensuring the model is evaluated on completely new subjects rather than recognizing parts of patients it already memorized during training.

This diagram outlines the complete machine learning lifecycle, progressing linearly from data ingestion and cleaning to model training, testing, and final deployment. It emphasizes the iterative nature of the process through specific feedback loops, where testing results refine the model building and real-world user feedback improves future data collection.