Introduction to Machine Learning

by Albert Major

Module 12: Clustering Algorithms (Unsupervised Learning)

Introduction to Unsupervised Learning

Unlike supervised learning, which maps input features (X) to known labels (Y) to predict future values, unsupervised learning deals with unlabeled data.

• Goal: Discover hidden structures, patterns, or groupings within the data rather than making predictions.
• Intuition: "Letting the data speak for itself" rather than teaching the algorithm what is correct.
• Main Categories:
  • Dimensionality Reduction: Reducing feature space (e.g., PCA, NMF).
  • Clustering: Partitioning data into groups (e.g., K-Means, Hierarchical).
  • Density Estimation: Finding dense regions (e.g., DBSCAN, GMM).
  • Manifold Learning: Discovering low-dimensional structures for visualization (e.g., t-SNE).

Clustering Fundamentals

Clustering partitions a dataset of N samples into K clusters (C₁ to Cₖ).

• Hard Clustering: Meets two strict requirements:
  1. Union Assumption: The union of all clusters equals the total dataset.
  2. Disjointness Assumption: The intersection of any two clusters is empty (no overlaps).
• Soft Clustering: Does not strictly meet these assumptions (e.g., allows noise removal or overlapping probabilities).
• Objective: Maximize Compactness (intra-cluster similarity) and maximize Separation (inter-cluster distance).

Distance Metrics

The choice of distance metric defines "similarity" and significantly impacts results.

• Euclidean Distance: Straight-line distance. Works best for compact, spherical clusters. Standardizes data scale.
• Manhattan Distance: Grid-based distance (sum of absolute differences). Useful when movement is restricted to grid lines (blocks).
• Cosine Similarity: Measures the angle between vectors. Scale-invariant; commonly used for text data or high-dimensional sparse data.