# 1. What Is Principal Component Analysis (PCA) Principal Component Analysis (PCA) is a dimensionality-reduction technique that reduces the number of features in a dataset while retaining as much of the important information — the variance — as possible. It works by creating new, uncorrelated features called **principal components**, which are linear combinations of the original features. The first principal component captures the most variance in the data, the second captures the next-most (while remaining uncorrelated with the first), and so on — which means the later, lower-variance components can typically be discarded with minimal information loss. ![[Pasted image 20260706155555.png|380]] # 2. Why Use PCA PCA is useful for three main reasons. First, it helps **reduce overfitting**: high-dimensional data can cause machine learning models to overfit, and lowering dimensionality directly addresses that risk. Second, it **improves visualization**: high-dimensional data is impossible to visualize directly, but projecting it down into 2 or 3 principal components makes it possible to plot and inspect visually. Third, it enables **faster computation**: machine learning models generally train and run faster when there are fewer features to process. # 3. The PCA Procedure 1. **Standardize the data.** Subtract the mean from each feature and scale it to unit variance, so that no single feature dominates the analysis purely because of its scale. 2. **Calculate the covariance matrix.** This matrix captures how every pair of features in the data relates to (co-varies with) one another. 3. **Find the eigenvectors and eigenvalues** of the covariance matrix. The eigenvectors represent the directions of maximum variance in the data, while the eigenvalues represent how much variance each corresponding eigenvector explains (see the Data Science and ML Knowledge note for a full worked example of computing eigenvalues and eigenvectors). 4. **Choose the components.** Sort the eigenvectors by decreasing eigenvalue, and select the top $k$ that together explain a desired amount of total variance — a typical target is choosing enough components to capture 80–90% of the variance. 5. **Transform the data.** Project the original data onto the selected eigenvectors to produce the new, lower-dimensional representation that will actually be used downstream.