Introduction to Machine Learning

by Albert Major

Introduction to Neural Networks and Multi-Layer Perceptrons

1. Recap: Classification and Regression Foundations

• Classification vs. Regression:
  • Classification: Predicts discrete labels (e.g., Cats vs. Dogs, 0 or 1). The model learns a decision boundary to separate classes in feature space.
  • Regression: Predicts continuous values (e.g., house prices). The model fits a function (line or curve) through data points to estimate values.
• The Curse of Dimensionality:
  • While features generally represent information, adding infinite features is not beneficial.
  • Drastically increasing dimensionality increases complexity and can degrade performance (the "curse"). Performance does not scale linearly with the number of features.
• Logistic Regression Limitations & Solutions:
  • Linear regression is poor for classification because outliers can skew the decision boundary, preventing a clean binary separation.
  • Sigmoid Activation: Logistic regression uses the sigmoid function (1 / (1 + e^-z)) to map input values to a probability between 0 and 1.
  • Thresholding: A standard threshold of 0.5 is used; values above are Class 1, values below are Class 0. This creates a stable "S-shaped" curve resilient to outliers.
• Cost Function (Binary Cross-Entropy):
  • The goal of optimization is to minimize cost.
  • If the target is 1 and prediction is 1, cost approaches 0. If prediction is 0, cost is high.
  • The two separate cost cases (for y=1 and y=0) are combined into a single formula: -y * log(y_hat) - (1-y) * log(1-y_hat). This allows for a continuous optimization landscape.

2. Introduction to Neural Networks

• Motivation (Feature Extraction):
  • Traditional Machine Learning (e.g., SVM) relies on manual feature extraction (SIFT, HOG, etc.), which requires deep domain knowledge.
  • If features are noisy or irrelevant (e.g., house color for house price prediction), traditional models fail.
  • Deep Learning Advantage: Neural networks perform automatic, end-to-end feature extraction. They learn which features are relevant directly from the raw data (pixels, text).
• Biological Inspiration:
  • Networks are inspired by the primate visual cortex structure (V1, V2, V4).
  • Hierarchical Processing: The brain processes visual data in layers:
    • V1: Detects simple forms (edges, corners).
    • V2/V4: Recognizes geometric shapes.
    • Inferior Temporal Cortex: Recognizes full objects (faces, trees).
  • Neural networks mimic this by stacking layers to learn increasingly abstract representations.

3. The Multi-Layer Perceptron (MLP)

• Neuron Structure (The Perceptron):
  • Input: Receives data features.
  • Weights & Biases: Inputs are multiplied by weights, summed, and a bias term is added.
  • Activation: The weighted sum is passed through a non-linear activation function to determine the output (firing) state.
• Network Architecture:
  • Input Layer: Defined by the dimensionality of the data (not controllable by the designer).
  • Hidden Layers: The "magic" layers where feature extraction occurs. You can have infinitely many layers and neurons (hyperparameters).
  • Output Layer: Makes the final prediction. The size and activation depend on the task (e.g., 1 neuron for binary classification, N neurons for multi-class).
• Fully Connected: In an MLP, every neuron in one layer is connected to every single neuron in the subsequent layer.

4. Training and Optimization

• Hyperparameter Tuning:
  • The number of hidden layers and neurons per layer determines complexity.
  • Validation Set: Architectural decisions must be made based on performance on a validation set, not the test set. Tuning on the test set leads to overfitting to that specific unseen data.
• Overfitting vs. Underfitting:
  • Underfitting: The model is too simple (e.g., 1 hidden layer with few neurons). Both training and test accuracy are low.
  • Overfitting: The model is too complex. Training accuracy is very high (near 100%), but test/validation accuracy drops significantly because the model memorized the noise rather than generalizing.
• The Training Loop:
  • 1. Feedforward: Data flows through the network (Input -> Hidden -> Output). Predictions are made, and the Loss (Error) is calculated based on the difference between prediction and target.
  • 2. Backpropagation: The learning phase. The network calculates the gradient of the loss function with respect to the weights.
  • 3. Weight Update (Gradient Descent): Weights and biases are adjusted in the opposite direction of the gradient to minimize loss. This is visualized as moving down the slope of a "loss bowl" to find the global minimum.
  • Epoch: One complete cycle of Feedforward and Backpropagation over the entire training dataset.

5. Activation Functions

Activation functions introduce non-linearity. Without them, many layers would mathematically collapse into a single linear model, making it impossible to solve complex problems like XOR gates.

• Sigmoid:
  • Range: [0, 1].
  • Usage: Typically for binary classification output.
• Hyperbolic Tangent (Tanh):
  • Range: [-1, 1].
  • Usage: Often used in recurrent networks (LSTMs).
• ReLU (Rectified Linear Unit):
  • Formula: max(0, z).
  • Behavior: Outputs the input if positive; outputs 0 if negative.
  • Usage: The standard choice for hidden layers. It is computationally efficient and helps with gradient flow.
• Output Layer Rules:
  • Binary Classification: Sigmoid.
  • Multi-class Classification: Softmax.
  • Regression: Linear (or ReLU if output cannot be negative, e.g., prices).

6. Deep Learning and Convolutional Neural Networks (CNNs)

• Hierarchical Learning:
  • A network is considered "Deep" when it has multiple hidden layers.
  • Shallow layers learn low-level features (edges); deep layers learn high-level semantic concepts (objects). In very deep layers, representations become highly abstract and may not look like the original image to humans.
• CNN Components (Computer Vision):
  • Designed specifically for image data to handle spatial relationships.
  • Convolutional Layers: Use "kernels" or "filters" that slide (sweep) across the image. They detect specific patterns (like vertical edges). If a pattern matches the image patch, it yields a high activation score.
  • Pooling Layers (Max Pooling): Reduce the spatial resolution (dimensionality) of the feature maps. They look at a local patch (e.g., 2x2) and keep only the maximum value. This preserves the most dominant features while discarding unnecessary detail.
  • Structure: A typical CNN consists of alternating Convolution and Pooling layers (feature extraction), followed by a Flattening step and Fully Connected MLP layers (classification).

7. AI Safety and Cybersecurity (Brief Mention)

• The field is moving toward interdisciplinary approaches combining AI and Cybersecurity.
• Key concerns include detecting deep fakes, securing software against AI attacks, and complying with regulations like the GDPR and the AI Act (effective August 2024).