Introduction to Machine Learning

by Albert Major

1. Core idea

• Logistic regression is a classification algorithm used in supervised learning.
• It predicts categorical outcomes (e.g., yes/no, spam/not spam).
• It is a linear model that estimates probabilities of belonging to each class using the sigmoid function.
• Logistic regression is transparent and interpretable, unlike “black box” models like neural networks.

2. Why it matters

• Forms the foundation for neural networks. Each neuron in a neural network acts like a logistic regression unit.
• Understanding it builds the conceptual and mathematical base for more complex models.

SUPERVISED LEARNING CONTEXT

• In supervised learning, we have labeled data: input features (X) and known outputs (y).
• Goal: learn a function f(X) → y.
• Two main task types:
  • Regression: output y is continuous (e.g., predicting house prices).
  • Classification: output y is categorical (e.g., spam or not spam).
• Logistic regression is for classification, especially binary classification (two classes).

3. LINEAR MODEL STRUCTURE

• Logistic regression uses a linear combination of features:

  z = wᵀx + b

  • x: vector of input features
  • w: weights (coefficients)
  • b: bias (intercept)

• z is called the linear score.

• This is identical to linear regression up to this point.

4. SIGMOID FUNCTION

• Converts the linear score (z) into a probability between 0 and 1:

  σ(z) = 1 / (1 + e⁻ᶻ)

• Properties:

  • When z → +∞, output → 1 (class 1 likely).
  • When z → −∞, output → 0 (class 0 likely).
  • When z = 0, output = 0.5 (classifier is uncertain).

Interpretation:

• Output of sigmoid = P(y = 1 | x), probability that the input belongs to class 1.

5. EFFECT OF PARAMETERS

• Bias (b): shifts the sigmoid curve left or right, changing the model’s baseline.
• Weights (w): change the steepness of the curve (sensitivity).
  • Large w = model reacts strongly to that feature.
  • Small w = that feature matters less.

6. DECISION BOUNDARY

• The decision boundary separates the two classes.
• At boundary, model output = 0.5 → z = 0 → wᵀx + b = 0.
• This defines a line (2D), plane (3D), or hyperplane (nD).
• Samples above the boundary → class 1.
• Samples below the boundary → class 0.

7. CLASSIFICATION THRESHOLD

• Default threshold = 0.5.
  • p ≥ 0.5 → predict 1
  • p < 0.5 → predict 0
• You can change this threshold.
  • Higher threshold (e.g., 0.7): more conservative (fewer positives).
  • Lower threshold (e.g., 0.3): more lenient (more positives).
• Used to balance false positives vs false negatives.
  • Example: spam filters lower threshold to avoid missing real emails.

8. LOSS FUNCTION: BINARY CROSS-ENTROPY (LOG LOSS)

• Measures how wrong predictions are:

  L = −[y log(p̂) + (1 − y) log(1 − p̂)]

  • y: true label (0 or 1)
  • p̂: predicted probability from sigmoid

• If prediction matches true label → loss small.

• If prediction far from true label → loss large.

• This loss punishes confident wrong predictions heavily.

9. OPTIMIZATION

• Goal: find weights (w) and bias (b) that minimize the loss.
• Optimization is done using gradient descent:

  1. Compute gradients (partial derivatives) of loss w.r.t. each parameter.

  2. Update parameters step-by-step:

     w ← w − η * ∂L/∂w

     b ← b − η * ∂L/∂b

     (η = learning rate).

  3. Repeat until convergence (loss stops changing).

Gradient meaning:

• The derivative of loss with respect to z equals the error (p̂ − y).
• So, model updates weights proportional to input × error.
• This minimizes error over time.

10. CONVEXITY

• Logistic regression loss is convex, so it has a single global minimum.
• Gradient descent will always converge to the optimal solution if learning rate and iterations are chosen properly.

11. REGULARIZATION

• Prevents overfitting by penalizing large weights.
• Adds a penalty term to the loss function.

Types:

1. L2 (Ridge) – adds sum of squared weights.
   • Encourages smoothness, keeps all weights small but nonzero.
2. L1 (Lasso) – adds sum of absolute weights.
   • Encourages sparsity, many weights become exactly zero (feature selection).
3. Elastic Net – combines L1 and L2.

• Regularization reduces model variance but slightly increases bias.
• Helps balance underfitting vs overfitting.

12. EVALUATION METRICS (BINARY CLASSIFICATION)

You cannot use regression metrics (like MSE). Classification needs its own metrics.

Confusion Matrix:

• 2×2 table summarizing predictions.

Main metrics:

• Accuracy = (TP + TN) / (TP + TN + FP + FN)
• Precision = TP / (TP + FP)
• Recall (Sensitivity) = TP / (TP + FN)
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall)
• Balanced Accuracy = (Recall + Specificity) / 2
• Specificity = TN / (TN + FP)

13. ROC CURVE AND AUC

• ROC (Receiver Operating Characteristic) shows trade-off between true positive rate and false positive rate as threshold changes.
• AUC (Area Under Curve) measures overall model quality:
• Useful for imbalanced data.

Good metrics for unbalanced data: