# 1. Types of Machine Learning Machine learning approaches are generally grouped into four categories, distinguished by whether the training data includes labels and by what kind of feedback the model learns from. **Supervised learning** works with labeled data, meaning every training example comes with a known correct answer. The dataset is split into training and test sets so the model's performance can be checked on data it has never seen. Supervised learning is used for prediction and classification tasks, and common algorithms include K-nearest neighbors (KNN), regression, support vector machines (SVM), and decision trees. **Unsupervised learning** works with unlabeled data, so there is no "correct answer" to train against and no train/test split in the traditional sense. Instead, the goal is to discover structure in the data itself, through techniques like clustering, dimensionality reduction, K-means, and anomaly detection using models such as a Gaussian mixture. **Semi-supervised learning** is used when labeled data is expensive or time-consuming to obtain. A model is first trained on the small amount of labeled data that is available, and that model is then used to make predictions on (or otherwise leverage the structure of) the much larger pool of unlabeled data. **Reinforcement learning** trains an agent to make a sequence of decisions by rewarding or penalizing its actions within an environment, so it learns a strategy (policy) that maximizes cumulative reward over time. # 2. Bias, Variance, and the Bias-Variance Tradeoff **Bias** is the error introduced by an overly simple model that fails to capture the true patterns in the data. High bias leads to **underfitting**, which shows up as poor performance on both the training set and the test set. Linear regression and logistic regression are examples of high-bias models, while more flexible models like decision trees, K-nearest neighbors, and support vector machines have lower bias. To reduce bias, you can add more features, engineer better features, switch to a more flexible model, or reduce the strength of regularization. **Variance** is the error introduced by a model that is overly complex and overly sensitive to the specific training data it saw, which leads to **overfitting**. High-variance models fit the training data very well but generalize poorly to new data — examples include deep decision trees, high-degree polynomial regression, deep neural networks, and complex ensemble models. To reduce variance, you can collect more data, increase regularization (L1 Lasso or L2 Ridge), simplify the model, perform feature selection, or use cross-validation to get a more honest estimate of generalization performance. The **bias-variance tradeoff** refers to the fact that reducing one of these two error sources often increases the other, so model-building is really a search for the sweet spot between them. Three tools are commonly used to navigate this tradeoff: - **Regularization** (L1, L2, or Elastic Net) penalizes model complexity directly, pulling coefficients toward zero to control variance. - **Hyperparameter tuning via cross-validation** evaluates many combinations of parameters across multiple data splits, helping identify the combination that produces the optimal decision-boundary complexity — one that minimizes both variance and bias together. - **Ensemble methods** (such as bagging and random forests) combine multiple models to reduce variance without necessarily increasing bias. # 3. Evaluating Classification Models ## 3.1 Confusion matrix A confusion matrix is a 2x2 table used to evaluate a classification model's performance by comparing actual outcomes against predicted outcomes. | | actual positive | actual negative | | ---------------- | ---------------- | ---------------- | | predict positive | TP (true positive) | FP (false positive) | | predict negative | FN (false negative) | TN (true negative) | From this table, several standard metrics can be derived: **Accuracy** is the overall proportion of correct predictions out of all predictions made. It can be misleading on an **imbalanced** dataset, where simply predicting the majority class every time can produce a high accuracy score without the model having learned anything useful. $accuracy = \frac{TN+TP}{TN+TP+FN+FP}$ **Error rate** is the proportion of incorrect predictions, and is simply $1-accuracy$. **Recall** (also called sensitivity, or the true positive rate) measures how many of the actual positives were correctly identified by the model. $recall = \frac{TP}{TP+FN} = \frac{TP}{P}$ **Precision** measures how many of the predicted positives were actually correct. $precision=\frac{TP}{TP+FP}=\frac{TP}{Predict_{Positive}}$ **F-score** balances precision and recall using their **harmonic mean**, where a score of 1 is perfect and 0 is the worst possible score. It is especially useful when the classes are imbalanced and you need a single metric that reflects a balance between precision and recall. $F=\frac{2\times precision\times recall}{precision+recall}$ **Choosing the right metric** depends on which type of error is more costly in your specific context. When false positives are costly — for example, flagging a legitimate email as spam, or recommending an unnecessary medical treatment — you should optimize for **precision**, since you need your positive predictions to be exactly accurate. When false negatives are costly — for example, missing a case of fraud, cancer, or a security threat — you should optimize for **recall**, since you need to catch as many true positives as possible even at the cost of some false alarms. ## 3.2 ROC and AUC The **ROC curve** (Receiver Operating Characteristic curve) plots the true positive rate (recall) on the y-axis against the false positive rate on the x-axis, evaluated across every possible probability threshold used to convert a model's continuous predicted probability into a binary label (the default threshold is usually 0.5). Each point on the curve represents a different threshold, and the curve as a whole illustrates the tradeoff between sensitivity and specificity as that threshold changes. **AUC** (Area Under the Curve) condenses the entire ROC curve into a single number between 0 and 1. It can be interpreted as the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance — in other words, it summarizes how well the model separates the two classes across all thresholds, not just at one specific cutoff. | roc-auc | interpretation | | ------- | --------------- | | 0.5 | random guessing | | 0.6-0.7 | weak | | 0.7-0.8 | good | | 0.8-0.9 | strong | | 0.9+ | excellent | ROC-AUC is preferred over plain accuracy specifically because it remains informative on **imbalanced datasets**, such as cancer detection or fraud detection. To see why accuracy alone can be dangerously misleading, consider 100 observations made up of 99 "good" cases and 1 "bad" case: a model that predicts every single observation as "good" achieves 99% accuracy while being completely useless — it has a 100% true positive rate for the majority class, but also a 100% false positive rate for the minority class it was actually supposed to catch. A well-performing model's ROC curve will hug the upper-left corner of the plot, since a reasonable model should always achieve a lower false positive rate than true positive rate. The more area a model's ROC curve covers, the better it performs at distinguishing the two classes. ROC-AUC is particularly useful for comparing multiple models against each other: you can run cross-validation for one model, plot its curve, then repeat the process for a second model and directly compare which one performs better. It is also a useful diagnostic tool for choosing hyperparameters and for deciding whether to prioritize true positives or true negatives, depending on the specific case study. Beyond binary classification, ROC-AUC concepts extend to multi-class classification as well, though the extension requires some care. Because of all of this, ROC-AUC is a widely used tool for evaluating and comparing classification algorithms, especially in fields like medical decision-making where the cost of different error types varies dramatically. ## 3.3 AIC and BIC (general model selection) **AIC** (Akaike Information Criterion) and **BIC** (Bayesian Information Criterion) are model-selection metrics that balance goodness of fit against model complexity, in order to avoid overfitting by penalizing models that use too many parameters. $AIC=2k-2\ln(L)$, where $k$ is the number of parameters and $L$ is the likelihood of the model. A lower AIC (or BIC) indicates a better-balanced model. # 4. Resampling and Validation Methods Before you can trust any of the evaluation metrics above, you need a validation strategy that gives you an honest estimate of how the model will perform on new data. Several resampling methods are commonly used for this purpose, each suited to a different kind of data. **Bootstrap** resampling repeatedly samples observations *with replacement* from the original dataset. Because sampling is done with replacement, the same observation may appear multiple times in a single bootstrap sample, while other observations may not appear at all. Bootstrap is commonly applied to compute confidence intervals, estimate standard errors, assess model stability, quantify uncertainty in A/B test metrics, and analyze small datasets whose underlying distribution is unknown. Its main advantages are that it makes no distributional assumptions, works for complex statistics that don't have a simple closed-form formula, and is easy to implement. Its downsides are that it is computationally expensive, it does not create genuinely new information (it only resamples what you already have), and it can behave unpredictably on very small datasets. **K-fold cross-validation** evaluates model performance by repeatedly splitting the data into training and validation sets, such that each observation serves as the validation sample exactly once. Performance is then averaged across all folds to produce stable estimates of accuracy, precision, recall, F1, or ROC-AUC. Five-fold cross-validation is the most common choice; ten-fold can lower bias further but requires more computation. When the target classes are imbalanced, **stratified k-fold** should be used so that each fold preserves the overall class distribution. Within this framework, it's worth distinguishing three different data splits: the **training set** is used to fit model parameters, the **validation set** (a portion of the training data) is used for parameter selection and to detect overfitting during cross-validation, and the **test set** is held out entirely to evaluate the final model's predictive power and generalization. K-fold cross-validation makes efficient use of limited data and produces a more stable performance estimate than a single train/test split, and it helps detect overfitting — but it is more computationally expensive than a single split and assumes the observations are independent of one another. **Time series cross-validation** uses a rolling time window to split training and test data, respecting the chronological order of observations. It is used for tasks like stock prediction, demand forecasting, revenue forecasting, marketing mix modeling (MMM), and user retention forecasting. Its key advantage is that it preserves temporal ordering, which prevents data leakage from the future into the past and mimics how the model will actually be deployed in production. Its downside is that early folds have less training data available, which tends to produce higher variance in the estimated performance compared to standard cross-validation. # 5. Regularization Regularization is a technique used to **reduce overfitting** and improve a model's ability to **generalize to unseen data**. It works by adding a **penalty term** to the model's loss function, which discourages the model from assigning excessively large coefficients to any single feature. This helps control model complexity and reduces variance. Regularization is especially helpful when a model has many features, when features are highly correlated with each other (multicollinearity), when the dataset is small, when overfitting is already a problem, or when working with high-dimensional data in general. The general form of a regularized loss function is $loss + \lambda \times penalty$, where a small $\lambda$ produces weak regularization and a large $\lambda$ produces strong regularization. **Lasso (L1) regularization** adds the absolute value of the coefficients to the loss function. This shrinks coefficients toward zero and can force some of them to exactly zero, effectively performing automatic feature selection. Lasso is the right choice when interpretability matters and you need explicit feature selection, and it is commonly used in genomics, text features, and advertising features. **Ridge (L2) regularization** adds the squared magnitude of the coefficients to the loss function. This shrinks coefficients toward zero but keeps every variable in the model. Ridge is the right choice when most features are genuinely useful, when features are correlated, when interpretability is less important than raw predictive accuracy, and it is commonly used in marketing attribution, demand forecasting, and customer lifetime value (LTV) modeling. **Elastic Net** combines both L1 and L2 penalties: $Loss + \lambda_1\sum{|\beta_j|}+\lambda_2\sum{\beta_j^2}$ In practice, regularized models are easy to fit with libraries like scikit-learn: `from sklearn.linear_model import LogisticRegression; model = LogisticRegression(penalty='l2', C=1)`. **When to use which:** choose **L1** when you want to reduce the number of features in your model and improve interpretability through automatic feature selection. Choose **L2** as a good general-purpose default for preventing large weights and reducing overfitting when explicit feature selection isn't required. Choose **Elastic Net** when you want the combined benefits of both approaches. # 6. Feature Selection and Dimensionality Reduction Feature selection aims to identify the most informative variables while removing irrelevant, redundant, or noisy ones. Doing this well reduces overfitting, improves model interpretability, reduces training time, improves generalization, and helps manage multicollinearity. **Filter methods** evaluate each feature independently of any specific machine learning model, which makes them fast and computationally efficient. Common filter techniques include **correlation**, **ANOVA** (which tests whether the mean of a numerical feature differs significantly across target groups), **Chi-square** (which measures the association between categorical features and a categorical target), **mutual information** (which measures how much information a feature provides about the target, captures nonlinear relationships, and works for both classification and regression), and **tree-based feature importance**. Filter methods are fast, model-agnostic, and good for an initial screening pass, but they might miss features that only become important in combination with others. **Wrapper methods** repeatedly train models on different subsets of features and select whichever subset produces the best model performance. **Forward selection** starts with no features, adds one feature at a time, and stops once additional features provide little further improvement. **Backward selection** starts with all features and removes them one at a time until performance drops noticeably; this becomes expensive as the number of candidate features grows large. **Recursive feature elimination** starts with all features and repeatedly removes the least important one. Wrapper methods account for feature interactions and can produce better-performing models than filter methods, but they are computationally expensive and carry a risk of overfitting to the specific model used during selection. **Embedded methods** perform feature selection as part of model training itself, as in L1 Lasso regression or decision tree and random forest models, which inherently produce feature importance scores. Embedded methods are efficient because selection is integrated directly into the modeling process, and the regularization involved can also help reduce multicollinearity — but the resulting feature selection is specific to whichever model type was used. **Dimension reduction** techniques like **PCA** (which uses matrix factorization to reduce the overall dimensionality of the feature data) and **autoencoders** (neural-network-based models that are trained to reconstruct their own input, where the compressed hidden layer becomes a dense, lower-dimensional representation of that input) take a different approach: instead of selecting a subset of existing features, they construct new, compressed features. PCA and similar approaches (like ICA) are easy to implement and can help prevent overfitting, but they sacrifice the interpretability of the original features, since the model now trains on the output of a dimensionality-reduction step rather than the raw variables themselves. **Choosing the right technique** depends on a few practical factors. Dataset size matters: filter methods are generally faster and more suitable for large datasets, while wrapper methods can become too computationally expensive for very high-dimensional problems. Model type matters too: embedded methods are convenient when your primary model choice is already something like a decision tree or a penalized regression model. Your goal matters: if you primarily want to understand which features matter most, filter or embedded methods are a good starting point; if maximizing predictive performance is the priority, wrapper methods may yield better results. And overfitting risk matters: wrapper methods require careful cross-validation to avoid overfitting to the selection process itself. A few additional practical tips: incorporate domain knowledge to guide which features are worth considering in the first place, identify and remove features that are essentially duplicates of one another (highly correlated pairs), and treat feature selection as an iterative process — trying different methods and evaluating their impact rather than committing to one approach up front. # 7. Distance Metrics ## 7.1 Euclidean distance Euclidean distance is the straight-line distance between two points in Euclidean space — the most intuitive notion of "distance" from everyday geometry, generalized to any number of dimensions. It is widely used in machine learning algorithms such as KNN, K-means, hierarchical clustering, and recommendation systems. $ \begin{aligned} A(1,2),\ B(4,6)\\ d=\sqrt{(4-1)^2+(6-2)^2}=5\\ d(X,Y)=\sqrt{\sum{(x_i-y_i)^2}} \end{aligned} $ Euclidean distance is sensitive to feature scale and outliers: if one dimension has a much larger numeric range than the others, it will dominate the distance calculation regardless of its actual importance. Because of this, features should be standardized (for example with `StandardScaler`) before computing Euclidean distances. Its advantages are that it's easy to understand, computationally efficient, works well with continuous numerical features, and is widely supported across ML algorithms. Its drawbacks are that it's sensitive to feature scale and outliers, becomes less effective in high-dimensional spaces (where distances between points tend to become less meaningful — the "curse of dimensionality" again), and requires numerical features. ## 7.2 Manhattan distance Manhattan distance is another common way to measure distance between data points, frequently used alongside Euclidean distance in algorithms like K-nearest neighbors, clustering, and recommender systems (e.g. collaborative filtering). Instead of squaring the differences between coordinates, it sums the **absolute value** of those differences, which makes it more robust to outliers than Euclidean distance. $M(x,y)=|x_1-x_2|+|y_1-y_2|$ # 8. Time Series Fundamentals ## 8.1 Stationarity A time series is considered **stationary** when its statistical properties don't change over time. Specifically, three conditions must hold: the mean of the series cannot drift over time, the variance of the series cannot change over time, and the covariance between the $i_{th}$ term and the $(i+m)_{th}$ term cannot depend on time (only on the lag $m$ between them). Many classical time series models, including ARIMA, assume stationarity, which is why differencing or other transformations are often applied before modeling. ## 8.2 ARIMA **ARIMA** stands for AutoRegressive Integrated Moving Average, and it's built from three components, each controlled by one of the three parameters in ARIMA(p, d, q): - **Autoregression (AR)** captures the relationship between an observation and a certain number of its own lagged observations. The parameter $p$ represents the order of the autoregressive term — for example, ARIMA(1, 0, 0) is a model with one autoregressive term. - **Differencing (I)** makes a time series stationary by removing trends or seasonality, calculated as the difference between consecutive observations. The parameter $d$ represents how many times differencing is applied — for example, ARIMA(0, 1, 0) applies differencing once. - **Moving Average (MA)** captures the dependency between an observation and the residual errors from a moving-average model applied to lagged observations, assuming the current value depends linearly on past error terms. The parameter $q$ represents the order of the moving-average term — for example, ARIMA(0, 0, 1) is a model with one moving-average term. The parameters $p$, $d$, and $q$ are typically chosen using model-selection criteria like AIC or BIC (see §3.3), comparing candidate parameter combinations and selecting whichever balances fit and complexity best. **Assumptions of the ARIMA model:** - **Stationarity**: the time series' statistical properties (mean, variance) must remain constant over time. - **Linearity**: future values are assumed to be a linear combination of past values and past error terms. - **No autocorrelation in residuals**: the differences between predicted and actual values (the residuals) should not be correlated with each other. - **Normally distributed residuals**: the residuals are assumed to follow a normal distribution centered at zero, which is necessary for valid statistical inference, parameter estimation, and hypothesis testing. # 9. Linear Algebra Foundations ## 9.1 Eigenvalues and eigenvectors Eigenvalues and eigenvectors are fundamental concepts in linear algebra that describe how a matrix transforms data, and they help identify the most important directions of variation within that data. For a matrix $A$, a vector $v$ is an eigenvector (and $\lambda$ its corresponding eigenvalue) if: $A*v=\lambda*v$ In other words, the eigenvector's *direction* is left unchanged by the transformation $A$ — only its length changes, and the eigenvalue tells you by how much it is stretched or shrunk. **Worked example.** Given the matrix: $ A = \begin{bmatrix} 4,2 \\ 1,3 \end{bmatrix} $ Subtracting $\lambda$ from the diagonal: $ A - \lambda*I = \begin{bmatrix} 4-\lambda,2\\ 1,3-\lambda \end{bmatrix} $ Setting the determinant equal to zero and solving: $ |A-\lambda*I|=0 \rightarrow (4-\lambda)(3-\lambda)=0 \rightarrow \lambda_1=5,\ \lambda_2=2 $ Taking the eigenvalue $\lambda=5$ and solving $(A-5I)v=0$: $ \begin{bmatrix} -1,2\\ 1,-2 \end{bmatrix}*\begin{bmatrix} x\\ y \end{bmatrix}=0 $ gives the eigenvector: $ v = \begin{bmatrix} 2\\ 1 \end{bmatrix} $ **Applications** include PCA (where each eigenvalue divided by the sum of all eigenvalues, $\frac{\lambda}{100}$, gives the proportion of variance explained by that component), spectral clustering (which uses the eigenvectors of a similarity matrix to find cluster structure), and recommendation systems (which can rank items by eigenvalue). The advantages of eigenvalue-based methods are data compression, noise reduction, faster training, and the ability to reveal underlying structure in the data. The drawbacks are that the results can be hard to interpret, they rely on a linear-relationship assumption (as in PCA), they are sensitive to feature scaling (features should be standardized first), and some information is inevitably lost in the compression. ## 9.2 Principal Component Analysis (PCA) PCA reduces data from a high-dimensional space to a lower-dimensional one while maximizing the variance captured along each new dimension. The typical procedure is: first, standard-scale all features so that no single feature dominates due to its scale; second, compute the correlation or covariance matrix across features; and third, compute the eigenvalues and eigenvectors of that matrix — the eigenvectors become the new "principal component" axes, and the eigenvalues indicate how much variance each axis explains. # 10. Classification Algorithms ## 10.1 Naive Bayes Naive Bayes predicts the probability that an observation belongs to a particular group — for example spam vs. not-spam, fraud vs. not-fraud, or a medical diagnosis vs. no diagnosis, or a sentiment category. The "naive" part of the name comes from its core simplifying assumption: that all features are conditionally independent of one another given the class. This assumption is rarely exactly true in practice, but it makes the model extremely fast to train and evaluate, and it works surprisingly well even on high-dimensional data like text. Naive Bayes is a direct application of Bayes' theorem: - $P(A|B)$ = probability of class $A$ given the observed evidence $B$ - $P(B|A)$ = probability of observing evidence $B$ given that the class is $A$ - $P(A)$ = the prior probability of class $A$ - $P(B)$ = the overall probability of observing evidence $B$ ## 10.2 Support Vector Machines (SVM) SVM is a supervised learning algorithm used primarily for classification. It works by finding the best boundary to separate classes of data — specifically, the optimal decision boundary (or hyperplane) that separates the different classes while maximizing the distance between that boundary and the nearest data points from each class. The **margin** is the distance between the decision boundary and the closest observations from each class. SVM specifically chooses the boundary that **maximizes this margin**, which leads to better generalization on unseen data. The **support vectors** are the observations closest to the decision boundary — they are the most influential data points in the entire dataset because they are what actually determine the position of the optimal hyperplane; every other point could move around without changing the boundary, but the support vectors cannot. In the simplest case, when classes are linearly separable, SVM finds the hyperplane that maximizes the margin between them, represented as $w^Tx+b=0$, where $w$ is the weight vector and $b$ is the intercept. SVM's advantages are that it performs well on small and medium-sized datasets, is robust against overfitting, handles high-dimensional data well, and is flexible thanks to kernels that let it model nonlinear relationships. Its drawbacks are that it can be computationally expensive, it is sensitive to feature scaling (features should be standardized), and it is less interpretable than simpler models like logistic regression. **Kernels** allow SVM to construct nonlinear decision boundaries by implicitly mapping the data into a higher-dimensional space where a linear separator can be found. | kernel | boundary shape | typical use | | --------------------------- | --------------- | -------------------------------------------- | | linear | straight line | text classification, high-dimensional data | | polynomial | curved | modeling feature interactions | | RBF (radial basis function) | highly flexible | general-purpose nonlinear classification | | sigmoid | neural-network-like | specialized cases resembling a neural net | **SVM vs. logistic regression:** | feature | logistic regression | SVM | | ------------------ | --------------------- | -------------------------------------- | | goal | predict probability | maximize margin | | output | probability | class label / distance from boundary | | interpretability | high | low | | small data | good | excellent | | large data | fast | slow | | nonlinear relationships | limited | strong, via kernels | # 11. Ensemble Learning Ensemble methods combine multiple models to improve predictive performance, reduce overfitting, and increase model stability — the core idea being that a group of "weak learners," each individually mediocre, can be combined into a single "strong learner" that outperforms any of them alone. **Base models and diversity.** Ensembling begins with training several different base models, sometimes called weak learners — these can be different algorithm types (e.g. a decision tree, a linear regression, and a neural network) or the same algorithm trained on different subsets of the data or with different hyperparameters. The key to successful ensembling is ensuring **diversity** among the base models, meaning they should tend to make different kinds of errors so that their individual strengths and weaknesses complement one another rather than overlap. **Combining predictions.** Once you have predictions from multiple base models, you need a way to merge them into one final prediction: - **Averaging** simply takes the mean of the predictions from each model, and works well for regression problems. - **Voting** has each model "vote" for a class label, with the final prediction assigned to whichever class receives the most votes; this works for classification problems. - **Weighted averaging or voting** assigns each model a weight based on its validation performance, then computes a weighted combination rather than treating every model equally. - **Stacking** trains a separate meta-model whose job is to learn how to best combine the predictions of the base models. **Why ensembling works.** It exploits the bias-variance tradeoff directly: different models have different biases (their tendency to systematically underfit or overfit) and different variances (their sensitivity to changes in the training data), and combining multiple models can reduce overall variance and, in some cases, reduce bias too — producing a more stable and generalizable model. This mirrors the "wisdom of the crowd" idea from outside machine learning: the collective judgment of a diverse group is often better than any single individual's judgment. ## 11.1 Decision tree A decision tree is a supervised learning algorithm used for both classification and regression. It works by recursively splitting the data into smaller groups based on feature values, choosing each split to maximize separation between target classes (for classification) or minimize prediction error (for regression). Decision trees are easy to interpret, handle nonlinear relationships naturally, and require no feature scaling — but a single tree is prone to overfitting, has high variance, and can be unstable, meaning small changes in the training data can produce a very different tree. ## 11.2 Random forest Random forest is a **bagging** (bootstrap aggregation) technique built from a group of decision trees. It works by drawing multiple bootstrap samples from the training data, training one decision tree on each sample, randomly selecting a subset of features to consider at each split, and then aggregating the predictions of all the trees — the trees are trained independently of one another and can be trained in parallel. Each tree "votes" on the outcome (for classification, the forest takes the majority vote; for regression, it takes the average). Because the individual trees are trained on different bootstrap samples and different feature subsets, they end up relatively uncorrelated with each other, which is what allows their errors to average out and reduces the overall error rate. This makes random forest more robust and less prone to overfitting than a single large, complex decision tree, while also being fairly easy to run and providing useful feature-importance rankings. Its main drawbacks are reduced interpretability compared to a single tree, and a larger overall model size. ## 11.3 AdaBoost (Adaptive Boosting) AdaBoost is a **boosting** algorithm that combines many weak learners into a single strong learner, but — unlike random forest's parallel, independent trees — it trains its learners **sequentially**, with each new learner focused on the mistakes of the ones before it. It works by training a weak tree, identifying which observations it misclassified, increasing the weight assigned to those incorrectly classified observations, and then training the next tree with more focus on those harder cases; this process repeats sequentially. AdaBoost is a relatively simple boosting algorithm that is effective for classification and less prone to overfitting than very deep trees, but it is sensitive to noisy data and outliers, since misclassified noisy points keep getting more weight in every subsequent round. ## 11.4 Gradient boosting Gradient boosting improves on AdaBoost's idea by fitting each new tree to the **residual errors** of the previous models, rather than simply reweighting misclassified observations. Gradient Boosted Trees (GBTs) are a supervised learning technique that combines multiple weak learners, usually decision trees, into a strong final model by sequentially adding new trees that focus on correcting the mistakes the previous trees made. The "gradient" in the name refers to the fact that the algorithm uses gradient descent to minimize a chosen loss function — a measure of how far off the model's predictions currently are — which determines the direction each new tree should push the model in order to improve. **How it works, step by step:** 1. Initialize a weak model $F_0$ with a constant value, typically the average of the target values in the data. 2. For each boosting iteration $m = 1$ to $M$, calculate the **pseudo-residuals** — the negative gradient of the loss function for each observation $i$, $r_i=-\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}$ — which represent how much the current model's prediction needs to be adjusted for that data point. 3. Fit a new decision tree $H_m$ to predict these pseudo-residuals $r_i$ using the original input data. 4. Update the overall model: $F_{m}(x)=F_{m-1}(x)+v\times h_m(x)$, where $v$ is the learning rate (a shrinkage parameter) that controls how strongly the new tree's predictions are incorporated, which helps prevent overfitting. 5. Repeat until the final model $F_m(x)$ is reached. **Core hyperparameters:** - **n_estimators**: the number of boosting rounds (trees) in the ensemble. Larger values generally improve performance but increase the risk of overfitting. - **learning_rate**: shrinks the contribution of each individual tree. Smaller learning rates slow down learning and require more trees, but often improve generalization and help prevent overfitting. - **max_depth**: the maximum depth of each individual tree, limiting its complexity and helping prevent overfitting; shallower trees are weaker learners. - **subsample**: the proportion of samples randomly selected to train each tree, which introduces randomness and helps prevent overfitting, similar in spirit to random forest's bootstrap sampling. - **min_samples_split**: the minimum number of samples required to split a node; larger values prevent overly complex trees. - **min_samples_leaf**: the minimum number of samples required in a terminal leaf node, which helps control tree complexity and avoid overfitting to noisy data. - **colsample_bytree / colsample_bynode / colsample_bylevel**: control the proportion of features randomly sampled for each tree, node, or level, introducing further randomness and helping prevent overfitting. - **gamma**: the minimum loss reduction required to justify a node split; a larger gamma produces a more conservative model. - **max_features**: limits the number of features considered at each split, similar to random forest. ## 11.5 XGBoost (Extreme Gradient Boosting) XGBoost is an optimized implementation of gradient boosting that adds regularization, parallel processing, tree pruning, native handling of missing values, and built-in cross-validation support. It typically delivers excellent predictive performance, fast training, and strong generalization, and it handles nonlinear feature interactions well — but it comes with more hyperparameters to tune, is harder to interpret than simpler models, and can overfit if those hyperparameters aren't tuned carefully. ## 11.6 Rotation forest Rotation forest is a less common ensemble technique. It works by randomly dividing the features into groups, applying **PCA** to each group separately, transforming the features using the resulting principal components, training decision trees on this transformed data, and finally aggregating the predictions across trees. This increases diversity among the trees and can improve accuracy, but it is computationally expensive and much less common in industry practice than random forest or gradient boosting. ## 11.7 Bagging vs. boosting, revisited To summarize the two dominant ensembling philosophies: **bagging** uses bootstrap resampling to train many learners in parallel and then averages their results — this helps with overfitting and high variance because the resampled datasets overlap substantially but aren't identical, though it doesn't help when the base model is underfitting. Random forest is the classic bagging example and is relatively fast to run. **Boosting** is an iterative technique that adjusts the weight of each observation based on how the previous round classified it — observations that were predicted incorrectly get increased weight in the next round, as in gradient boosting. Boosting can overfit, and tends to increase both computation time and model complexity compared to bagging, but it often outperforms bagging methods like random forest specifically on **imbalanced data**, at the cost of taking longer to run. # 12. Clustering: Choosing the Number of Clusters Choosing the appropriate number of clusters, $K$, is one of the most important decisions in any clustering analysis. The goal is to find a $K$ that balances cluster quality against model simplicity. Several methods are commonly used: **The elbow method** looks at the within-cluster sum of squares, $WCSS = \sum\sum{(x_i-\mu_k)^2}$, where $\mu_k$ is the centroid of cluster $k$. As the number of clusters increases, each cluster becomes smaller and WCSS naturally decreases, because observations end up closer to their (now more numerous) centroids. The technique is to plot WCSS against $K$ and look for the "elbow" point where additional clusters stop producing meaningful reductions in WCSS. **Hierarchical clustering** builds clusters by repeatedly merging or splitting observations based on distance (typically Euclidean distance): each observation starts as its own cluster, the two closest clusters are merged, and this repeats until all observations belong to a single cluster. The result is visualized as a **dendrogram**, and the optimal number of clusters can be read off by looking for the largest vertical gap in the dendrogram — the number of branches remaining at that gap corresponds to the suggested number of clusters. **Silhouette score** measures how well each observation fits within its assigned cluster compared to neighboring clusters, on a scale from -1 to 1. A score near 1 indicates well-separated clusters, a score near 0 indicates overlapping clusters, and a negative score suggests the observation may be misclassified into the wrong cluster. In practice, you run clustering across a range of candidate $K$ values and select whichever produces the highest average silhouette score. **The gap statistic** compares a clustering solution's performance against what you would expect from randomly generated (unclustered) data of the same size and shape, and selects the $K$ at which the real clustering structure is significantly better than what random chance would produce. # 13. Recommendation Systems Recommendation systems suggest products, movies, music, ads, articles, or other content to users based on their preferences and past behavior. The two most common approaches are collaborative filtering and content-based filtering. **Collaborative filtering** recommends items based on the behavior of similar users or similar items, under the assumption that users who behaved similarly in the past will continue to behave similarly in the future, or that items similar to ones a user already likes are good recommendations. A common technique is **ALS** (Alternating Least Squares, a matrix factorization method), which decomposes a `user x item` interaction matrix into latent factors and uses them to predict missing ratings. Collaborative filtering requires no item metadata and can discover hidden patterns that aren't obvious from item descriptions, and it's usually quite accurate — but it suffers from the **cold start problem** (a brand-new user with no history can't be recommended anything meaningfully) and from **sparsity** (most users only interact with a tiny fraction of the total item catalog). **Content-based filtering** recommends items similar to ones a specific user has previously liked, essentially looking "vertically" within a single user's own history rather than across users. It works by building a feature representation (catalog data) for each item, representing those features as vectors, and computing similarity between items — for example using Euclidean distance. Content-based filtering doesn't need data from other users, handles brand-new items well (since it only needs the new item's features, not interaction history), and is relatively explainable — but it risks **overspecialization** (recommending only items very similar to what's already been liked) and requires substantial feature engineering, since it depends on having good item metadata. **Personalized models** more broadly draw on collaborative filtering, **uplift modeling** (which scores the incremental effect of an action or offer on a user, then ranks users by that incremental score rather than by raw likelihood), and general ranking models. # 14. Choosing the Right Algorithm | situation | recommended models | | ------------------ | --------------------------------- | | small data | logistic regression, SVM | | large data | random forest, neural networks | | high interpretability needed | decision trees, linear regression | | high accuracy needed | gradient boosting, deep learning | | noisy data | K-NN, ensemble methods | # 15. Pooling Strategies in Grouped Data When data is organized into groups (such as stores, users, or tickers), there are three general strategies for how much information to share across groups when fitting a model. **Complete pooling** ignores group membership entirely and fits one model to all the data as if it came from a single population — simple, but it can produce misleading estimates when groups genuinely differ (as in Simpson's paradox). **No pooling** fits a completely separate model to each group independently — this respects real group differences, but performs poorly for groups with little data, since each group's estimate relies only on its own (possibly very small) sample. **Partial pooling** (also called hierarchical or multilevel pooling) sits between these two extremes: each group gets its own estimate, but those estimates are pulled ("shrunk") toward the overall population average by an amount that depends on how much data that group has — groups with little data get pulled more strongly toward the shared average, while groups with lots of data are allowed to deviate further based on their own evidence. This is the approach used by hierarchical Bayesian models (see the Bayesian Knowledge note for a detailed worked example) and is generally the most statistically sound option when groups are meaningfully different but some groups have limited data. # 16. End-to-End Modeling Workflow A typical modeling project follows a consistent sequence of steps: - understand the **business problem** the model is meant to solve. - **explore** the available data to understand its structure, quality, and limitations. - **prepare the data** for modeling by handling outliers, missing values, duplicates, and other data-quality issues, and by applying any necessary transformations. - **engineer features** that will help the model capture the relevant signal. - **train** the model and tune its hyperparameters iteratively. - **validate** the model's results using a new, held-out set of data it hasn't seen during training or tuning. - **implement (deploy)** the model into production and track its performance over time. # 17. Data Engineering for ML ## 17.1 ETL vs. ELT **ETL** (Extract, Transform, Load) transforms data before loading it into the warehouse, while **ELT** (Extract, Load, Transform) loads the raw data first and performs transformations afterward, inside the warehouse itself. Modern cloud platforms like BigQuery and Snowflake typically favor ELT, because storage is inexpensive and warehouse compute scales efficiently enough to handle transformations on demand. In practice at Unity, most analytics pipelines followed an ELT pattern using Airflow, Spark, and BigQuery, which made it possible to retain raw data and build flexible downstream models for experimentation, attribution, and reporting. For integrating external or unstructured data, an ETL-style process is sometimes used instead — transforming the data with Python within the pipeline first, then loading it into the database — in order to make downstream modeling more efficient. ## 17.2 Integrating user identity across systems When user data lives across multiple systems (product analytics, CRM, marketing tools), a mapping table is used to connect the different IDs together: `internal_user_id, email_hash, salesforce_lead_id, salesforce_contact_id, account_id, device_id, anonymous_id, created_at, source_system, confidence_score`. The typical workflow is: 1. Collect raw IDs from internal product data, Salesforce, marketing tools, and web analytics. 2. Standardize the identifying fields — for example, lowercasing emails, trimming whitespace, and normalizing domains. 3. Create **deterministic matches** first, based on strong signals like a shared email address, a shared CRM contact ID, or a shared account ID. 4. Add **probabilistic matching** where needed, based on weaker signals like a shared company domain, similar names, or similar device/session behavior. 5. Build a master identity table that serves as the single source of truth. 6. Assign one canonical user ID to be used consistently for analytics. 7. Track a match-confidence score for every linkage, and avoid overwriting high-confidence matches with lower-confidence ones. 8. Validate the result with data-quality checks, including checks for duplicate IDs, many-to-one mappings, and missing CRM links. ## 17.3 Feature store A feature store is a central repository of machine learning features, refreshed on a batch schedule (daily or hourly) for features like LTV, retention, or historical spend, and via streaming for real-time features like the current session, recent clicks, or the most recent purchase. Common platforms for building a feature store include Databricks and Vertex AI. # 18. Model Operations and Monitoring ## 18.1 Model monitoring framework A production model monitoring framework generally tracks four categories of signals: - **Drift**: whether the distribution of input data is changing over time, typically measured with the Population Stability Index (PSI) or the Kolmogorov-Smirnov (KS) test, which measures the maximum distance between two distributions. - **Performance**: standard predictive metrics like AUC, precision, and recall, or MAP for ranking models. - **Bias**: whether the model's fairness holds consistently across different subgroups. - **Stability**: operational signals like missing values, whether all expected features are available, and whether the distribution of prediction scores stays consistent day to day — feeding into alerting and automatic retraining triggers. ## 18.2 Diagnosing model failure in production When a model that performed well in development suddenly fails in production, a structured troubleshooting process helps isolate the cause. **Isolate the issue** by checking three broad categories: data changes (differences in the distribution, format, or quality of production data compared to training data — this is the most common culprit), code discrepancies (making sure the production code exactly matches the development version, and checking for preprocessing errors, model-loading mistakes, or configuration issues), and system issues (infrastructure problems, network errors, or resource constraints that might be affecting the model indirectly). **Gather information** by comparing performance metrics (accuracy, precision, recall, F1) between the development and production environments, analyzing error logs for clues about the root cause, and examining specific instances where the model fails in production alongside their input data. **Perform root cause analysis** by considering: **concept drift** (whether the underlying relationships the model learned during training have changed in the real world — often driven by the data changes mentioned above); **overfitting** (if the model performed exceptionally well in development but poorly in production, it may have memorized training data rather than learning generalizable patterns); **training-serving skew** (verifying that the data preprocessing pipeline used in production is identical to the one used during training, since inconsistent preprocessing can produce very different inputs to the model); and **hidden biases** (assessing whether the training data was sufficiently representative, since a biased training set can produce a model that fails specifically on certain real-world segments). **Take action** based on what the root cause analysis reveals: if concept drift or data-quality issues are the cause, collect new data and retrain, potentially with more diverse examples; if code discrepancies are found, fix them directly; if overfitting is suspected, apply techniques like regularization (L1/L2), dropout, or early stopping; and if infrastructure shortcomings (processing power, memory) are the cause, address those directly. **Building a more robust production environment** going forward involves several practices: implementing robust monitoring with real-time alerts for performance degradation, so issues are caught early; setting up continuous learning pipelines that automatically retrain or recalibrate models as new production data arrives; running thorough pre-deployment validation using data that mimics the expected production environment; and using gradual deployment strategies like canary deployment (rolling out a new model to a small subset of traffic first) or shadow deployment (running the new model alongside the old one in production without using its outputs yet) to catch problems before they affect all users. # 19. Why Data Preprocessing Should Be Done Inside a Pipeline Data preprocessing (e.g., missing value imputation, encoding, scaling, and feature transformation) should **always be performed inside a machine learning pipeline after splitting the data into training and testing sets**. Fitting preprocessing on the entire dataset before splitting introduces **data leakage**, because statistics such as the mean and standard deviation are computed using information from the test set. This leads to overly optimistic evaluation results that do not reflect real-world performance. When using **cross-validation**, the preprocessing pipeline is fitted separately within each training fold, so each fold may learn slightly different scaling parameters. This is expected and desirable, as it simulates training the model on different samples of unseen data. After selecting the best model and hyperparameters, the entire pipeline is retrained on the full training dataset (or all available labeled data before deployment) and then used to transform and predict future observations consistently.