# 1. Descriptive Statistics Fundamentals ## 1.1 Types of statistics Statistics is generally divided into two branches. **Descriptive statistics** describes and summarizes a dataset as it is, without trying to generalize beyond it. **Inferential statistics** uses a sample of data to make inferences about a larger population that wasn't fully observed. ## 1.2 Representing categorical data numerically Categorical variables need to be converted into numbers before most models can use them, and the right encoding depends on whether the categories have a natural order. **Nominal data** (unordered categories, like city or color) is typically represented using dummy variables. **Ordinal data** (ordered categories, like a satisfaction rating) can instead be coded onto a numeric scale that preserves the ordering. ## 1.3 Measures of center and skew The three standard measures of central tendency are the **mean**, the **median**, and the **mode** (the most frequently occurring value). Comparing the mean and median reveals the shape of a distribution's tail: a **left skew** has a long tail on the left side and the median is greater than the mean, while a **right skew** has a long tail on the right side and the median is less than the mean. ![[Pasted image 20260531165541.png|201]]![[Pasted image 20260531165556.png|257]] A **boxplot** visualizes this shape directly: the box spans the interquartile range from Q1 to Q3, the whiskers extend to 1.5×IQR (where $IQR = Q3 - Q1$) beyond the box edges, and any points beyond the whiskers are considered outliers. **Quantiles** can be computed directly with `np.quantile(data[col], [0.25, 0.5, 0.75])`, where the 0.5 quantile is the median. ## 1.4 Measures of spread **Variance** measures the average squared deviation from the mean: $ variance = \frac{\sum(x_i - \mu)^2}{n - 1} $ In Python, `np.var(data, ddof=1)` computes the sample variance; without `ddof=1`, the function instead computes the population variance, which divides by $n$ rather than $n-1$. **Standard deviation** is simply the square root of variance: $ standard\ deviation = \sqrt{variance} $ Standard deviation is generally preferred when *reporting* results, because it's on the same scale and unit as the original data, which makes it much easier to interpret — for example, "revenue has a standard deviation of $20" directly tells you that revenue typically varies by about $20 from the mean. Variance, meanwhile, is preferred when *calculating* other statistics, because it has convenient mathematical properties and shows up directly in linear regression, Cohen's d, and ANOVA. **Mean Absolute Deviation (MAD)** offers a more robust alternative to variance and standard deviation: instead of squaring each deviation from the mean, it takes the absolute value, which keeps large outliers from being amplified the way squaring does. $ MAD = \frac{1}{n}\sum|x_i-\bar{x}| $ ## 1.5 Point estimates A point estimate assigns a single specific value to an unknown population parameter, and there are three standard approaches for producing one: the **method of moments**, **maximum likelihood estimation (MLE)**, and **least squares estimation**. The "moments" referenced by the method of moments correspond to increasingly detailed descriptions of a distribution's shape: the 1st moment is the mean, the 2nd moment is the variance, the 3rd moment is skewness, and the 4th moment is kurtosis. # 2. Sampling and Probability Distributions ## 2.1 Sampling data `df.sample(5, replace=False)` (or `replace=True`) draws a random sample from a DataFrame; with `replace=True`, the same row can appear more than once in the sample. Setting `np.random.seed(10)` makes repeated runs of the same sampling code return identical results, whereas without a seed, every run produces a different random sample. A seed matters most in contexts like train/test splitting and bootstrap resampling, where reproducibility lets anyone re-running the code get the exact same split, and the exact same confidence interval, that you originally reported. ## 2.2 Sampling methods Sampling methods fall into two broad categories. **Probability-based methods** give every unit in the population a known chance of being selected, and include simple random sampling, **stratified sampling** (randomly selecting from each subgroup separately), **cluster sampling** (randomly selecting a few whole subgroups in their entirety), and **systematic sampling** (selecting every $x_{th}$ row at a fixed interval). **Non-probability methods** select units based on predefined, non-random criteria — they're generally easier and cheaper to collect, but the resulting sample's relationship to the broader population is much harder to characterize statistically. ## 2.3 The Law of Large Numbers The Law of Large Numbers states that as sample size increases, the sample mean converges toward the true expected value of the population. This is the foundational justification for why larger samples produce more trustworthy estimates. ## 2.4 Common probability distributions **Uniform distribution** — every value within a range $(a,b)$ is equally likely to occur. - $X \sim Uniform(a,b)$: $E[X] = \frac{a+b}{2}$ - Generate data with `uniform.rvs()`: ```python from scipy.stats import uniform data = uniform.rvs( loc=0, # min scale=10, # max size=1000 ) ``` ![[Pasted image 20260531173041.png|187]] - **CDF** (cumulative distribution function) gives $P(X \le x)$ via `uniform.cdf()`, and `1 - uniform.cdf()` gives $P(X > x)$: ```python print(f'probability of X<=3: {uniform.cdf(5, loc=0, scale=10)}') ``` - **PPF** (percent point function, the inverse of the CDF) gives the value corresponding to a given percentile, via `uniform.ppf()`: ```python print(f'what is the 25% below cutoff value: {uniform.ppf(0.25, loc=0, scale=10)}') ``` - **PDF** (probability density function) gives the density at a single point, via `uniform.pdf()`: ```python print(f'probability of X=3: {uniform.pdf(3, loc=0, scale=10)}') ``` **Normal (Gaussian) distribution** — the classic symmetric, bell-shaped curve. - Symmetric around the mean, with mean = median = mode. - Its shape is fully determined by the mean $\mu$ and standard deviation $\sigma$. - The total area under the curve is 1, and the tails approach but never reach 0. - Roughly 68% of values fall within 1 standard deviation of the mean, 95% within 2 standard deviations, and 99.7% within 3 standard deviations. ```python data = norm.rvs( loc=20, # mean scale=15, # std size=5000 ) ``` ![[Pasted image 20260531173901.png|213]] - CDF: `norm.cdf(3, loc=20, scale=15)` - PPF: `norm.ppf(0.25, loc=20, scale=15)` - PDF: useful in hypothesis testing, where a p-value $\le \alpha$ leads to rejecting the null hypothesis: `norm.pdf(3, loc=20, scale=15)` **Exponential distribution** — models the waiting time until the next event occurs, such as the time until the next customer arrives, the time until a lightbulb fails, the probability a user converts within 3 days, or the 90th-percentile conversion time. - It has the **memoryless property**: if you've already waited 10 minutes, the probability of waiting another 5 minutes is exactly the same as if you had just started waiting from scratch. - $X \sim Exp(\lambda)$: $E[X] = \frac{1}{\lambda}$, $Var(X) = \frac{1}{\lambda^2}$. The distribution is right-skewed. ```python data = expon.rvs( scale=5, # 1/lambda size=5000 ) ``` ![[Pasted image 20260531175353.png|213]]![[Pasted image 20260602101315.png|275]] - CDF: `expon.cdf(3, scale=5)` - PPF: `expon.ppf(0.25, scale=5)` - PDF: `expon.pdf(3, scale=5)` **Binomial distribution** — models the number of successes across a fixed number of independent trials. - $X \sim Binomial(n,p)$: $E[X]=np$, $Var(X)=np(1-p)$ ```python data = binom.rvs( n=10, # number of independent trials, e.g. 10 coin flips p=0.5, # probability of success size=5000 # repeat the experiment 5000 times ) ``` ![[Pasted image 20260531180526.png|236]]![[Pasted image 20260602101556.png|295]] - CDF: `binom.cdf(3, n=10, p=0.5)` gives the probability of 3 or fewer successes. - PPF: `binom.ppf(0.25, n=10, p=0.5)` - Since this is a discrete distribution, its density function is a **PMF** (probability mass function): `binom.pmf(3, n=10, p=0.5)` gives the probability of exactly 3 successes. **Bernoulli distribution** — a single trial with only two possible outcomes. - $X \sim Bernoulli(p)$: $E[X]=p$, $Var(X)=p(1-p)$. A Bernoulli distribution is simply $Binomial(1,p)$, restricted to the two outcomes 0 and 1. ```python data = bernoulli.rvs( p=0.3, size=5000 ) ``` ![[Pasted image 20260531183411.png|197]] - CDF: `bernoulli.cdf(3, p=0.3)` (note this trivially returns 1, since 3 is above the maximum possible value of 1) - PPF: `bernoulli.ppf(0.25, p=0.3)` - PMF: `bernoulli.pmf(1, p=0.3)` is simply the probability of success, $p$. **Poisson distribution** — models the number of times an event occurs within a fixed period of time or space, such as the number of orders per day. The distinction from the exponential distribution is that Poisson models *how many* events occur (e.g. 5 customers per hour), while exponential models *how long* until the next event (e.g. wait time until the next customer). - $X \sim Poisson(\lambda)$: $E[X]=\lambda$, $Var(X)=\lambda$, and $\lambda$ is also the peak of the distribution. A quick way to sanity-check whether data plausibly follows a Poisson distribution is to compare its mean and variance — under a true Poisson process, they should be approximately equal. `smf.glm()` is used to fit a Poisson regression in Python. ```python data = poisson.rvs( mu=5, size=5000 ) ``` ![[Pasted image 20260531213938.png|229]]![[Pasted image 20260602101803.png|287]] - CDF: `poisson.cdf(3, mu=5)` - PPF: `poisson.ppf(0.25, mu=5)` - PMF: `poisson.pmf(k=3, mu=5)` **Student's t-distribution** — models the distribution of a sample mean when the true population standard deviation is unknown. Not knowing the true standard deviation introduces extra uncertainty, which makes the t-distribution **wider** than a comparable normal distribution. - Its shape is governed by degrees of freedom — for a one-sample t-test, $df = n-1$ — and as $df$ increases (i.e. as the sample grows), the t-distribution converges toward the normal distribution. ```python data = t.rvs( df=10, size=5000 ) ``` ![[Pasted image 20260531220125.png|235]] - CDF: `t.cdf(3, df=10)` - PPF: `t.ppf(0.25, df=10)` - PDF: `t.pdf(3, df=10)` - **Confidence intervals** are a natural application: given a sample mean and standard deviation, plug them into the t-distribution to compute a range that's likely to contain the true mean. ```python # confidence interval for mean=100, std=20, n=10 std = 20 mean = 100 n = 10 se = std / np.sqrt(n) crit = t.ppf(0.975, df=n-1) # two-sided range lower = mean - crit * se upper = mean + crit * se print(f'95% confidence interval: [{lower}, {upper}]') ``` **Log-normal distribution** — a variable is log-normal if $\log(X)$ follows a normal distribution. Quantities like revenue per user, income, order value, time spent on site, and customer lifetime value (LTV) are commonly log-normal: they're always positive, strongly right-skewed, and most of their mass sits close to zero. ```python data = lognorm.rvs( s=0.5, # sigma scale=np.exp(2), # exp(mu) size=5000 ) ``` ![[Pasted image 20260531221135.png|243]] `np.log1p()` is the standard transform used to convert raw data of this shape back toward normality, and it's preferred over a plain log because it gracefully handles zero values. - CDF: `lognorm.cdf(3, s=0.5, scale=np.exp(2))` - PPF: `lognorm.ppf(0.25, s=0.5, scale=np.exp(2))` - PDF: `lognorm.pdf(3, s=0.5, scale=np.exp(2))` ## 2.5 Central Limit Theorem The Central Limit Theorem (CLT) states that the sampling distribution of a statistic (like a sample mean) becomes closer to normal as the number of trials increases, regardless of the shape of the original underlying data — provided three requirements hold: the observations are independent, the sample size is large enough, and the variance is finite. For **zero-inflated and right-skewed** data, such as revenue, the CLT can fail to kick in at practical sample sizes because the variance is too large relative to the sample size. In this situation, more robust alternatives include: a log-transformed t-test; a **zero-inflated model** (a two-stage approach that first fits a logistic regression on whether the value is zero or nonzero, then fits an OLS model on the positive spending amount); or the **Mann-Whitney U test**, a rank-based, non-parametric test that doesn't rely on the CLT's normality approximation. # 3. Relationships Between Variables ## 3.1 Correlation and covariance **Covariance** measures whether two variables tend to move together, defined as: $ Cov(X,Y) = E[(X-E[X])(Y-E[Y])] = E[XY] - E[X]E[Y] $ **Correlation** standardizes covariance onto a fixed scale, which makes it directly interpretable as the strength and direction of a *linear* relationship between two variables: $ \rho = \frac{Cov(X,Y)}{\sigma_x\sigma_y} $ Equivalently, computed directly from sample data: $ \rho = \frac{1}{n-1}\sum\frac{(x_i-\bar{x})(y_i-\bar{y})}{\sigma_x\sigma_y} $ In practice, use `sns.scatterplot()` or `sns.lmplot()` (which additionally overlays a trend line) to visualize the relationship, and `df.corr()` to compute the correlation coefficient directly. **A key caveat: correlation only captures linear relationships.** It should never be relied on blindly — always visualize the data with a scatter plot first, since two variables can have a strong nonlinear relationship (and thus real predictive value) while showing a correlation near zero. When a relationship looks nonlinear, transforming the data can often reveal (and linearize) it: - **Log transform** (`np.log(x)` or `np.log1p(x)`): appropriate for exponential growth or right-skewed data, such as revenue or income. - **Square root transform** (`np.sqrt(x)`): appropriate for count data or moderately skewed variables. - **Reciprocal transform** (`1/x`): appropriate for hyperbolic relationships or diminishing-returns patterns. These transforms can also be combined — for example, testing `1/x` against `log(y)` — to find whichever combination produces the most linear relationship. Transformation matters because several standard statistical methods, including correlation and linear regression, explicitly assume a linear relationship between variables. **Correlation is not causation.** Establishing actual causation requires either a randomized experiment (an A/B test) or a causal inference method suited to observational data, such as difference-in-differences or propensity score matching (see §6). ## 3.2 Confounding A **confounder** is a variable that influences both the dependent variable and the independent variable of interest, which can make an observed association look causal when it isn't. To identify a potential confounder, ask what other variables might plausibly affect both the treatment and the outcome, and use `pd.crosstab()` to check for imbalance and test the difference statistically. Several methods exist to control for confounding: - **Randomized experiments** balance both observed *and* unobserved confounders automatically, which is what makes them the gold standard. - **Regression adjustment** adds the confounding variables directly into the fitted model. - **Stratification** analyzes the relationship separately within subgroups, to check whether the effect holds consistently across them. - **Propensity score matching** matches similar units on observed characteristics before comparing outcomes, though it cannot account for unobserved confounders (see the dedicated [[Propensity Scoring Matching]] note). - **Inverse propensity weighting** reweights the sample by the inverse of each unit's estimated treatment probability. - **Difference-in-differences** compares the change in outcomes over time between a treated and untreated group. - **Instrumental variables** are used specifically when unobserved confounders are suspected (see §6.4). ## 3.3 Linear regression assumptions A linear regression model relies on five core assumptions: a genuinely **linear relationship** between predictors and the outcome; **independence of errors**, meaning observations don't influence one another; **homoscedasticity and normality** of residuals, meaning the residual variance should be roughly constant across the range of predicted values and approximately normally distributed; **minimal multicollinearity** between predictor variables; and the **absence of significant outliers** that could disproportionately influence the fitted line. ## 3.4 Variance Inflation Factor (VIF) VIF quantifies how much a regression coefficient's variance is inflated due to correlation with the other predictors, by comparing the variance of the full model to the variance of a model with only a single predictor: $ VIF = \frac{1}{1-R_j^2} $ where $R_j^2$ is the R² obtained from regressing predictor $j$ on all of the other predictors. A high VIF indicates that predictor $j$ is highly collinear with the rest of the feature set. # 4. Simpson's Paradox Simpson's Paradox occurs when a result looks different when examined in aggregate versus when examined within segments, driven by an imbalance in segment sizes. The overall rate can shift dramatically after a change in traffic mix, even if the rate within every individual segment stays the same. For example: - Desktop users: 80/100 = 80% conversion before, 9/10 = 90% conversion after. - Mobile users: 10/100 = 10% conversion before, 180/900 = 20% conversion after. - Totals: 90/200 = 45% conversion before, but 189/910 = 20.8% conversion after — a large apparent *drop*, even though *both* segments individually *improved*, because the traffic mix shifted heavily toward the lower-converting mobile segment. # 5. Experimental Design ## 5.1 Controlled experiments and randomization In a **controlled experiment**, participants are assigned by the researcher to either a treatment or control group, and the two groups need to be genuinely comparable for any observed difference to be attributed to the treatment — if the groups aren't comparable, the result is biased. Several types of randomization are used to strengthen this comparability: a standard **randomized controlled trial**; a **placebo-controlled trial**, where the control group receives a treatment that resembles the real one but has no active effect, so participants don't know which group they're in, reducing behavioral changes driven purely by expectation; and a **double-blind trial**, where neither participants nor researchers know the group assignments, reducing both response bias and interpretation bias. ## 5.2 Observational studies When randomization isn't feasible — whether for ethical, practical, or cost reasons — an **observational study** can be conducted instead, though at the cost of weaker causal claims (see §6). ## 5.3 Longitudinal vs. cross-sectional studies **Longitudinal studies** track the same individuals or entities over time, which captures trends and changes and enables before/after analysis — useful for methods like difference-in-differences — but is comparatively time-consuming and expensive. **Cross-sectional studies** collect data at a single point in time, such as surveying users about their satisfaction today; this is much faster and well-suited to purely descriptive analysis, but can't capture change over time. ## 5.4 Factorial design and statistical interaction A **factorial design** tests multiple treatments simultaneously — for example, study time and teaching method together — while a **statistical interaction** occurs when the impact of one treatment changes depending on the level of another treatment. (See the dedicated [[Multivariate Experiment]] note for a full worked example.) # 6. Causal Inference When a randomized experiment isn't possible, the analysis moves into **causal inference** from observational data, where the goal is to estimate what would have happened in the counterfactual scenario where treatment hadn't occurred. ## 6.1 Common causal inference methods - Propensity score matching — [[Propensity Scoring Matching]] - Inverse propensity weighting - Difference-in-differences and synthetic control — [[Difference in Difference]] - Regression discontinuity design — [[Regression Discontinuity Design]] - Instrumental variables — [[Instrumental Variables]] - Uplift modeling — [[Uplift Modeling]] - Causal forest - Bayesian structural time series ## 6.2 Instrumental variables Instrumental variables (IV) are used to estimate a causal effect when **unobserved confounders** influence both treatment assignment and the outcome but aren't captured in the data. A valid instrument must satisfy three conditions: it affects the likelihood of receiving treatment; it does not directly affect the outcome except through its effect on treatment; and it is independent of the unobserved confounders. IV is typically implemented using **two-stage least squares**: 1. Identify a plausible instrument and gather the corresponding data. 2. **Stage 1:** predict treatment using the instrument and other covariates: $Treatment = \beta_0 + \beta_1 Instrument + \beta_2 X + \epsilon$. 3. **Stage 2:** estimate the outcome using the *predicted* treatment from stage 1, rather than the actual observed treatment: $Outcome = \alpha_0 + \alpha_1 \widehat{Treatment} + \alpha_2 X + \eta$. Because the variation in the predicted treatment comes only from the instrument — not from whatever unobserved factors were driving the original treatment assignment — the resulting estimate is less biased by hidden confounders. See the dedicated [[Instrumental Variables]] note for more detail. ## 6.3 Synthetic control Synthetic control is a causal inference method used when there is a **single treated unit** and no single control group that serves as a good natural comparison. Rather than picking one control region (say, Canada) as a standard difference-in-differences analysis would, synthetic control constructs a **weighted combination of multiple control units** — for example, 40% Canada + 30% UK + 30% Australia — to build a more realistic counterfactual for the treated unit. ## 6.4 Doubly robust learners Doubly robust estimation combines an outcome model with a propensity (treatment) model, and only *one* of the two needs to be correctly specified for the resulting causal estimate to be unbiased — which makes it a more robust choice than relying on regression adjustment or propensity matching alone, and a natural fit for estimating incremental impact from observational marketing data. The outcome model estimates $E[Y \mid treatment, X]$, while the treatment result comes from a propensity model estimating $P(T=1 \mid X)$. If the outcome model is wrong but the propensity model is correct, the causal estimate remains unbiased; if the propensity model is wrong but the outcome model is correct, the estimate also remains unbiased. Only when *both* models are misspecified does the estimate become biased. This approach still only handles *observed* confounders, however — it offers no protection against unobserved ones. ## 6.5 Uplift modeling Uplift modeling predicts the **incremental impact of a treatment or marketing campaign** on an individual, rather than simply predicting who is likely to purchase overall. The goal is to identify users whose behavior actually *changes* because of the treatment. For example, User A might have an 85% purchase probability without treatment and 87% with it — a +2% lift — while User B might increase from 5% to 40% — a +35% lift. Even though User A is more likely to purchase overall, User B generates far more *incremental* value from the campaign. The uplift model outputs an incremental-lift score for each user, which can then be used to rank and target the users most likely to respond positively to treatment, maximizing campaign ROI while reducing wasted marketing spend. See the dedicated [[Uplift Modeling]] note for more detail. ## 6.6 Bayesian structural time series Bayesian structural time series models combine forecasting with causal impact estimation, producing a full probability distribution over the estimated effect rather than a single point estimate — useful for questions like "what would this metric have looked like without the intervention?" # 7. A/B Testing: Pitfalls and Special Cases ## 7.1 Common pitfalls A number of issues can quietly undermine an A/B test's validity: novelty effects, the peeking problem (checking results repeatedly before the test concludes, which inflates the false-positive rate), selection bias, sample ratio mismatch, network effects, misalignment between the measured metric and the true business goal, seasonality, and insufficient statistical power. ## 7.2 Network effects In a social network like Facebook, one user's actions are likely to influence the people around them, which violates the independence assumption most tests rely on and reduces statistical power, since cluster randomization effectively shrinks the sample size. A few strategies help control for this: creating **clusters** of closely connected people and assigning entire clusters to treatment or control together for better isolation — mathematically, this separation can be done via social-graph partitioning methods like normalized cuts, which group users based on their social interactions; running a **geo-based experiment**, which randomizes at the level of geographic regions instead of individual users; or running a **switchback experiment**, which assumes no spillover effect and instead randomizes which condition an entire unit (like a city or market) receives over successive time periods. ## 7.3 Novelty and primacy effects An A/B test can show an exaggerated initial effect due to a **novelty effect** — a new feature attracts attention (sometimes amplified by social media), inquisitive users rush to try it out, and metrics like time spent and engagement temporarily spike before settling back down to a new, only slightly higher, baseline. The opposite pattern, a **primacy effect**, occurs when users are anchored to what's familiar and show an aversion to change in general, temporarily depressing metrics for a new feature that will actually perform well once users adjust. To detect either effect, analyze the test results for **new users only**: if there's a statistically significant effect for new users but not for existing users, a novelty (or primacy) effect is likely at play, since new users have no prior baseline to be excited by or resistant to. Running the A/B test on new users specifically can also help guard against novelty effects directly, though this introduces its own complication if new users don't resemble the existing user base. Another approach is a **holdout test**: after gradually rolling out a feature, a small percentage of users are deliberately kept on the old experience for an extended period. Because a standard A/B test only runs for a short window while a holdout can run much longer, it helps quantify the long-term lift from a feature and makes it easier to distinguish a genuine effect from a temporary novelty effect. # 8. Probability vs. Likelihood **Probability** describes how likely an event is to occur; the p-value in hypothesis testing is a probability, specifically $P(\text{observed result} \mid \text{null hypothesis is true})$. **Likelihood** instead measures how plausible the *observed data* is, given a particular model or hypothesis — for example, if a hypothesis states that people's average height is 6 feet, the likelihood asks how plausible it would be to observe a sample of people who are actually 5 feet tall under that hypothesis. Probability and likelihood use the same underlying math but ask the question in opposite directions: probability fixes the model and asks about the data, while likelihood fixes the data and asks about the model. # 9. Prediction Interval vs. Confidence Interval A **confidence interval** describes the range in which a population parameter estimate is expected to fall, at a given significance level. A **prediction interval** instead describes the range for an individual future prediction: a point estimate plus or minus a margin of error, which is generally wider than a confidence interval because it accounts for both the uncertainty in the estimated parameter and the additional variability of a single new observation. # 10. Data Quality Issues in Practice ## 10.1 Types of bias Several recurring types of bias can distort an analysis: **selection bias** (probability-based or non-probability-based sampling that isn't representative of the population), **under-coverage bias** (some part of the population is systematically excluded from the sampling frame), and **survivorship bias** — the logical error of focusing only on the entities that "survived" some process while overlooking those that didn't, precisely because the ones that failed are no longer visible. For example, looking only at businesses that survived a recession and noting they're performing poorly can be misleading, because the businesses that failed and were removed from the dataset entirely likely performed *even worse* — the survivors are actually the relatively strong performers, not the weak ones. ## 10.2 Handling imbalanced data When one class vastly outnumbers another, several strategies help: **under-sampling** the majority class, **oversampling** the minority class, using models that are inherently less sensitive to imbalance (like tree-based models or Bayesian approaches), and — perhaps most importantly — focusing on the actual business cost of false positives versus false negatives rather than optimizing a metric like accuracy that imbalance can distort. ## 10.3 Handling outliers Outliers can be addressed by replacing them with the mean or median, standardizing the feature's scale, applying a log transformation (or a Box-Cox transformation more generally), dropping the affected observations, or capping extreme values at a percentile threshold like 99% (winsorization). It's also worth remembering that some model families — random forest and gradient boosting in particular — are inherently less sensitive to outliers than linear models. ## 10.4 Handling missing values Start by checking whether there's a *pattern* to the missingness (missing completely at random vs. missing for a systematic reason), since that shapes which fix is appropriate. Common approaches include replacing missing values with the mean or median, dropping the variable entirely if 80%+ of its values are missing, and using the mode to fill missing categorical data. # 11. Applied Modeling: LTV and Business Metrics ## 11.1 LTV predictive modeling Customer Lifetime Value (LTV) models predict a customer's future value using features like acquisition channel, behavior, retention, and engagement, commonly modeled with XGBoost or survival analysis techniques. A useful decomposition is: $ LTV = retention \times conversion \times expected\ revenue $ ## 11.2 LTV-weighted acquisition Rather than optimizing for traffic volume or raw registrations, a more sophisticated acquisition strategy optimizes the ratio of predicted LTV to Customer Acquisition Cost (CAC). Consider a channel bringing in 1,000 users at $5 LTV each versus a channel bringing in 500 users at $50 LTV each — optimizing purely for volume would favor the first channel, while optimizing for LTV/CAC would favor the second. In practice, this means predicting downstream LTV (often with XGBoost), calculating LTV/CAC for each channel, and using that ratio — rather than registrations or conversions alone — as the actual optimization target. This often shifts budget toward smaller but higher-quality channels that generate stronger long-term ROI and retention. A LTV/CAC ratio above **3:1** is generally considered healthy, and channel budget caps are typically set at the point where marginal CAC rises enough to push the ratio below a chosen threshold — commonly 1.5–2x for growth-stage companies. The general rule is to increase spend on channels with a high LTV/CAC ratio until diminishing returns set in. This approach has real limitations worth keeping in mind: LTV estimates carry real uncertainty, especially for new products without much historical data; CAC attribution depends heavily on the attribution model used (last-click and multi-touch attribution can give meaningfully different answers for the same spend); the ratio itself ignores payback period, so a 5:1 ratio realized over 3 years may actually be worse for a cash-constrained company than a 3:1 ratio realized over 6 months; and the ratio doesn't account for interactions between channels — cutting brand spend, for instance, may quietly raise CAC on performance channels that were benefiting from the brand awareness. ## 11.3 Bias detection in models Beyond overall accuracy, it's important to check a model's performance separately across demographic or business-relevant subgroups — comparing precision, recall, and false positive rate (FPR) across groups — to check for fairness and prevent the model from systematically discriminating against any particular segment.