# 1. Define the Research Question As a data scientist, an experiment needs to be framed in precise, quantified language from the start — for example, "p-value analysis indicates X had an effect on Y with a 10% risk of Type I error," rather than a vague claim of "it worked." Defining the research question well means clarifying the business goal and the question being asked, identifying potential proxy metrics and which of them is sensitive enough to reflect long-term success, thinking through the full user journey the experiment touches, and considering how the user segment chosen for the experiment might introduce confounders — for example, users who open the app on their own are naturally higher-intent than the general population, which can bias results if not accounted for. # 2. Create Hypotheses Every experiment needs a formal null and alternative hypothesis, which can be framed as two-sided, left-sided, or right-sided depending on the question. Choosing the right primary metric matters enormously: it should directly reflect the business objective and be sensitive enough to actually capture the expected treatment effect. Alongside the primary metric, **guardrail metrics** should be defined to ensure the experiment doesn't negatively affect other important behaviors, such as retention, latency, or customer satisfaction. Good experiment metrics share four properties: they're measurable, stable, aligned with business value, and resistant to manipulation or short-term gaming. Most business experiments use **two-sided hypotheses**, because the goal is usually to detect *any* meaningful effect, whether positive or negative — a new feature could improve a KPI, have no effect, or unexpectedly make it worse, and all three outcomes matter for the resulting business decision. # 3. Choose Treatment and Control via Randomization Randomization is the foundation of a valid experiment, but it introduces two practical problems worth anticipating. **The uneven-groups issue.** Random assignment can sometimes produce treatment and control groups with different numbers of subjects purely by chance. To reduce this imbalance, researchers can enforce equal-sized groups directly during assignment. **The covariate issue.** Randomization can also accidentally create groups that differ substantially on an important covariate, introducing confounding. For example, when testing whether a pill affects reaction time, pure random assignment might happen to put more young, physically active participants into the treatment group — and their reaction times would differ from the control group's regardless of the pill itself. The standard fix is **stratified randomization**: subjects are first divided into homogeneous subgroups based on the important covariate (like physical activity level), and randomization to treatment/control happens *within* each subgroup separately. This reduces confounding and improves comparability between the final groups. # 4. Select the Randomization Unit Experiments can randomize at the individual level (user ID, order ID), at an aggregated group level (store, region, team), or across a time window (a switchback experiment, see §16.2). In some cases — such as Prime vs. non-Prime members, or feature-adopters vs. non-adopters — group membership is dynamic and can't be forced onto a user, which makes user-ID an unsuitable randomization unit for that particular question. Whatever unit is chosen, it needs to align with the primary metric such that each observation is independent and identically distributed. For example, if the question concerns user behavior but only a user's *first session* after entering the experiment is measured, that choice is deliberate: many standard tests (like the t-test) assume independence between observations, and tracking only the first session per user is what preserves that independence, rather than tracking every session and inadvertently correlating repeated observations from the same person. # 5. Choose the Key Metrics Primary and guardrail metrics should be chosen *before* the experiment runs, not after seeing results. Microsoft's **STEDII** framework offers a useful checklist for evaluating candidate metrics: - **Sensitivity** — does the metric actually move when the product changes? A metric that rarely moves can't tell you anything from an experiment. "Proportion of monthly active users" moves too slowly to be useful; "sessions per user per week" is far more sensitive. - **Trustworthiness** — is the underlying data clean, and free of selection bias? Does the metric measure what it claims to? At Dropbox, for instance, does a "file sync" metric inadvertently include automated backups, artificially inflating the number? - **Efficiency** — can the metric be computed fast enough to actually inform a decision? A 90-day retention metric can't realistically be evaluated inside a 2-week experiment — a faster-moving proxy metric is needed instead. - **Debuggability** — can the metric be decomposed to understand *why* it moved? If conversion rate drops, can it be broken down into impression → click → upgrade steps to localize the cause? - **Interpretability** — can a product manager tell at a glance whether the metric moving is good or bad? Its directionality needs to be unambiguous. - **Inclusivity** — does the metric treat all user segments fairly, or could it hide harm being done to a specific subgroup? # 6. Power Analysis: Sample Size and Effect Size **Statistical power** is the probability of correctly rejecting a false null hypothesis — in other words, of detecting a real effect when one actually exists. It's the complement of the Type II error rate: power evaluates whether an effect is genuinely coming from the alternative hypothesis, while the significance level evaluates the risk of falsely detecting an effect that's actually coming from the null. A higher power is generally better: 0.7 is considered moderate detection ability, 0.8 is the common standard for A/B tests, and 0.9 is used when a rerun needs high confidence that a real effect exists. For example, if an experiment returns a p-value of 0.06 against an alpha of 0.05 with a power of 0.8, the p-value being this close to significance suggests rerunning the test with power increased from 0.8 to 0.9 (alpha is normally left untouched, since it reflects the acceptable risk of falsely rejecting the null, not a tuning knob). Even once statistical significance is established, it's still worth checking the minimum detectable effect (MDE) to assess *practical* significance — a 1% lift might be meaningful for a large company but immaterial for a startup, while a 10% lift might be the bar a smaller company actually needs to clear. Two error rates anchor this whole framework: - $\alpha$ = Type I error rate = false positive rate = the probability of concluding the treatment works when it actually doesn't (commonly set to 0.05). - $\beta$ = Type II error rate = false negative rate = the probability of missing a real treatment effect (commonly set to 0.2). | alpha | beta | case | | ----- | ---- | -------------------------------------------------------------------------------------- | | high | high | bad design: many false positives and many false negatives | | high | low | easy to detect effects, but many false positives — appropriate for low-risk UI tests | | low | low | ideal, but requires a large sample size | | low | high | a strict test that may miss real effects — appropriate for medical, financial, or safety-sensitive experiments | **Minimum Detectable Effect (MDE)** is the smallest effect size the experiment is designed to reliably detect, usually chosen based on the business case and the impact a smaller effect would actually have. Setting the MDE requires weighing engineering cost, risk, and how long the team can afford to run the experiment — a smaller MDE requires a much larger sample size, because a smaller true effect is harder to distinguish from ordinary random variation. During experiment design, the MDE (converted using Cohen's d, see §13) is what gets plugged into the power function to calculate the required sample size. **Effect size** measures the actual magnitude of the difference between groups. After the experiment runs, group means or proportions are estimated, and **Cohen's d** converts that into a standardized effect size, which is then compared against the pre-registered MDE to assess practical impact. | Metric | Typical MDE | | -------------------- | -------------- | | Website conversion | 1%–5% lift | | Ads CTR | 1%–5% lift | | Revenue per user | 2%–5% lift | | Retention | 0.5%–3% lift | | Low user-size contexts | 5%–10% | To make the tradeoff concrete: suppose an online store generates $100M per year at a 10% baseline conversion rate. If the MDE is set at a 1% lift (conversion rising to 11%), that translates to roughly $10M in additional annual revenue. That sounds significant, but it ignores the cost of building and maintaining the new feature — the real question is return on investment (ROI): if the cost of the change is less than the $10M return, the 1% lift is practically significant for the business; if not, it isn't, regardless of statistical significance. Statistical power itself can be computed directly with `power_analysis.solve_power(effect_size=1, nobs=30, alpha=0.05)`. ![[Pasted image 20260528231916.png|255]]![[Pasted image 20260611080309.png|272]] **Sample size** and power move together: increasing sample size generally increases power, making smaller effects easier to detect reliably. Required sample size can be estimated directly with `TTestIndPower().solve_power(effect_size=d, alpha=0.05, power=0.99, ratio=1)`, where `d` is the MDE converted into Cohen's d. The underlying formula is: $ n = \frac{(Z_{1-\frac{\alpha}{2}}+Z_{1-\beta})^2\sigma^2}{\delta^2} \approx \frac{16\sigma^2}{\delta^2} $ where $\delta$ is the absolute difference, $(\theta_2-\theta_1)^2$. For example, given an MDE of 20% relative lift over a 15% baseline CTR: $ \delta^2 = (CTR_2 - CTR_1)^2 = (CTR_1(1+0.2)-CTR_1)^2 = 0.0009 $ And $\sigma$ is the variance term, computed differently depending on the type of comparison: - One-sample proportion: $\sigma^2 = p(1-p)$ - One-sample mean: $S^2 = \frac{\sum{(x_i-\bar{x})^2}}{k-1}$ - Two-sample proportion: $\sigma_{pooled}^2 = p_2(1-p_2)+p_1(1-p_1)$ - Two-sample mean: $S_{pooled}^2 = 2 \times S_1^2$ It's often useful to visualize how statistical power changes across different combinations of sample size and effect size, to build intuition for the tradeoffs before committing to a design. # 7. Randomize Participants Once randomization is designed, check group balance directly by comparing covariates between groups to confirm they're similar before treatment begins. An **AA test** — comparing both groups' *pre-treatment* historical behavior, where no difference should exist since both are effectively "control" at that point — is a useful sanity check on the randomization mechanism itself. To avoid selection bias more broadly: proper randomization, consistent exposure rules, balanced traffic allocation, and avoiding contamination between groups all help. It's also worth monitoring instrumentation quality, data integrity, novelty effects (§16), network effects (§16), and the logic that triggers experiment exposure, all in service of accurate measurement. Pre-registering hypotheses and avoiding early stopping further reduce analytical bias — together, these requirements are sometimes summarized as the **Stable Unit Treatment Value Assumption (SUTVA)**. It's also worth watching for **survivorship bias** in longer-running experiments, where treatment users might churn or delete their account before the experiment concludes, silently skewing the remaining sample. # 8. Run the Experiment The experiment should run for its full planned duration, based on the required sample size, traffic allocation, and expected weekly fluctuation, to ensure sufficient statistical power and a reliable result. **Multiple simultaneous experiments.** Running several experiments on the same users at once can introduce interference effects and inflate the false-positive rate through multiple hypothesis testing: $P(\text{at least one false rejection of } H_0) = 1-P(\text{no false rejection}) = 1-\text{binomial\_pdf}(k=0, n=T)$. In these cases, a correction like the **Bonferroni adjustment** can help control the overall Type I error rate. **Avoid peeking.** Continuously checking results before the planned stopping point inflates the false-positive rate and can lead to premature conclusions drawn from an insufficient sample. ![[Pasted image 20260528232140.png|394]] **Sequential testing** offers a principled alternative to peeking: results are evaluated continuously as data arrives, rather than waiting for the full planned sample size, with significance thresholds adjusted dynamically as data accumulates. This allows earlier stopping and faster decisions. Common methods include the Sequential Probability Ratio Test (SPRT), group sequential testing, alpha spending, and Bayesian sequential testing (which monitors a posterior probability directly). Sequential testing is especially valuable when traffic is expensive, results need to be usable quickly, and rapid iteration matters. # 9. Data Exploration, Transformation, and Validation **Factorial design.** A factorial design studies multiple independent variables within the same experiment by testing all possible combinations of factor levels — this is what a multivariate experiment is (see the dedicated [[Multivariate Experiment]] note). It lets you evaluate both each factor's main effect and the interaction effects between factors; `sns.heatmap()` can help visualize patterns and relationships among the variables involved. **Covariate adjustment.** Covariates are variables that may influence the outcome; including them in the statistical model helps account for additional variability, improves estimation accuracy, and reduces potential confounding. This is typically done with an OLS or logistic regression model, visualized with `sns.lmplot()`: ```python import statsmodels.formula.api as smf model = smf.logit( 'converted ~ C(test_group) + total_ads + C(most_ads_day) + C(most_ads_hour)', data=df ).fit() print(model.summary()) ``` ![[Screenshot 2026-05-28 at 10.38.28 PM.png|414]] To interpret the model, `np.exp(coef) - 1` gives the percentage higher or lower chance that users in the treatment group convert relative to control. Note that this only expresses a *relative* lift or drop, not the absolute lift — for that, a confidence interval on the estimate is needed. **Synthesizing insights from a complex experiment.** Visualization is essential for summarizing treatment effects and interaction patterns: `sns.catplot()` produces grouped bar charts and other categorical comparisons across treatment groups, while `sns.relplot()` generates scatterplots useful for exploring trends and correlations between variables. **Confounding variables.** A confounder influences both the independent and dependent variable, and can bias the estimated treatment effect if left unaddressed (see the Statistical Knowledge note for more detail on identifying and controlling confounders). **Box-Cox transformation.** `stats.boxcox()` identifies an optimal transformation parameter to make data more normally distributed and stabilize its variance. ![[Pasted image 20260528173153.png|342]] **Heteroscedasticity.** This occurs when the variability of one variable changes across the range of another, producing non-constant variance in the residuals — a violation of a standard regression assumption. **Residual analysis.** After fitting a regression between the outcome and treatment variable, `sns.residplot()` visualizes the residuals directly, helping detect heteroscedasticity or model misspecification. **Multicollinearity.** When multiple features are involved, check for multicollinearity using `variance_inflation_factor` before interpreting any individual coefficient's statistical impact. | VIF | Meaning | | ----- | -------------------------- | | <5 | good | | 5–10 | moderate concern | | >10 | high multicollinearity | **External factors.** Holidays, marketing campaigns, product releases, and broader economic conditions can all confound an experiment's results if they coincide with the test window. **Sample ratio mismatch (SRM).** A chi-square goodness-of-fit test checks whether the actual traffic split across variants matches the intended allocation; a significant mismatch can indicate the randomization mechanism itself is broken. **Novelty, primacy, and network effects.** Network effects can be mitigated by randomizing at the cluster or organizational level rather than the individual level. Novelty effects (a temporary spike in usage right after a new feature launches, which decays back down over time) can be isolated by looking only at new users, who have no "before" state to be excited about. Primacy effects — the opposite pattern, where users initially resist a new feature because they're attached to the old one, only for usage to jump once they adjust — can similarly be studied by focusing on new users. # 10. Check Distributional Assumptions **Normal distribution.** Parametric tests generally require the sampling distribution of the *mean* to be approximately normal, not necessarily the raw data itself — once the sample size is large enough, the Central Limit Theorem ensures the sample mean tends toward normality regardless of the underlying data's shape. To check normality directly, density plots and Q-Q plots reveal skewness and overall shape, and formal tests like Shapiro-Wilk (better suited to smaller datasets) or Anderson-Darling can assess it more rigorously. These checks are typically run during exploratory data analysis, and again after any data transformation. With very large datasets, Shapiro-Wilk becomes extremely sensitive and may flag even tiny, practically irrelevant deviations from normality — in that regime, a visual check is often more informative than the formal test. ![[Pasted image 20260528114711.png|281]]![[Pasted image 20260528115233.png|284]] **Random sampling.** Observations should be randomly sampled so the resulting sample genuinely represents the target population. **Independence of observations.** Individual observations need to be independent of one another; violations distort variance estimates and increase the risk of both Type I and Type II errors. **Sufficient sample size.** An adequate sample size improves estimation stability and reduces uncertainty — small samples produce wider confidence intervals and lower statistical power. Rough guidelines: $n \ge 30$ per group for many t-tests, $np \ge 10$ for proportion-based tests, and an expected count $\ge 5$ per cell for chi-square tests. **Bootstrap validation.** Bootstrap resampling serves as a useful diagnostic or sanity check: if the resulting bootstrap sampling distribution looks highly irregular or non-normal when normality was expected, that's a sign the model's assumptions may not actually be well satisfied. # 11. Select the Appropriate Statistical Test The right test depends on the data type, the number of groups being compared, and the assumptions the data can actually support. **One-sample t-test.** Compares a sample mean against a known value, with degrees of freedom $dof = n-1$. **Two-sample t-test.** Compares the means of two independent groups, assuming approximate normality and (in the classic version) equal variances, run with `scipy.stats.ttest_ind`. T-tests are most appropriate for smaller samples ($n<30$) where the population standard deviation is unknown, since the t-distribution's heavier tails account for the extra uncertainty that comes with a small sample. - **Degrees of freedom**: shapes the t-distribution; for a two-sample test, $dof = n_1 + n_2 - 2$. As $dof$ increases, the t-distribution converges to the standard normal distribution. - **P-values** can be computed with `t.cdf()`, or `1 - t.cdf()` for a right-tailed test. - **Equal-variance (Student's) t-test**: pools the two groups' variances, $S_p^2=\frac{(n_1 - 1)S_1^2+(n_2 - 1)S_2^2}{n_1+n_2-2}$, giving $SE = \sqrt{S_p^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}$ and $t_{stat} = \frac{\mu_1-\mu_2}{SE}$, with $dof = n_1+n_2-2$. - **Welch's t-test (unequal variance)**: does not pool variances, instead using $SE = \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$ with $dof = \frac{\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)^2}{\frac{\left(\frac{s_1^2}{n_1}\right)^2}{n_1-1}+\frac{\left(\frac{s_2^2}{n_2}\right)^2}{n_2-1}}$ — this produces a lower degrees of freedom, larger critical values, a wider confidence interval, and more conservative inference overall. Run with `ttest_ind(a, b, equal_var=False)`. **Paired t-test.** Compares two related measurements from the same subjects or matched pairs (e.g. weight before and after a diet program), rather than two unrelated groups. Degrees of freedom is $n-1$, and it's run with `pingouin.ttest(paired=True)`. Using an *unpaired* test on paired data ignores the natural pairing structure and inflates variability, which raises the false-negative rate — the unpaired test's p-value often ends up larger than the paired test's, potentially hiding a real effect. **One-way ANOVA.** Compares mean differences across 2 or more groups, assuming independence, approximate normality, and homogeneity of variance across groups. - **Post-hoc tests.** When ANOVA (or its non-parametric analogue, Kruskal-Wallis) finds a significant overall difference, post-hoc tests identify which specific pairs of groups actually differ, using something like `pairwise_test()`. **Tukey's HSD** compares all pairwise group differences while controlling the overall Type I error rate, and is commonly used for broad comparisons across many groups. The **Bonferroni correction** is a more conservative adjustment to the p-value or significance threshold, generally preferred for a smaller number of focused comparisons. - Degrees of freedom: $dof_{between}=k-1$, $dof_{within}=N-k$. **Z-test.** Standardizes how far an observation sits from the mean, in units of standard deviation, and compares a sample statistic to a hypothesized population value relative to the standard error: $ \begin{aligned} standardized\ value &= \frac{value - mean}{standard\ deviation}\\ z &= \frac{sample\ stat - hypothesized\ value}{standard\ error} \end{aligned} $ The **standard normal distribution** is simply the normal distribution with mean 0 and standard deviation 1. Thanks to the **Central Limit Theorem**, the sampling distribution of the mean tends toward normal as sample size grows, even when the raw data isn't normally distributed — which is what makes the Z-test broadly applicable in large-sample settings. **Bootstrap distributions** offer an alternative way to approximate a statistic's sampling distribution, by repeatedly resampling the observed data; the result can be visualized with `sns.displot()` and summarized with `np.std()`. Z-tests generally require $n \ge 30$ to lean on CLT and bootstrap approximations; bootstrapping itself can be done either via random resampling or by shuffling the treatment/control label and comparing against the observed result. P-values can be computed from z-scores with `norm.cdf()` (or `1-norm.cdf()` for a right-tailed test), and confidence intervals can be estimated from the quantiles of the sampling or bootstrap distribution via `np.quantile()`. **Choosing between t and z**: use Z when the population variance is known *and* $n \ge 30$; otherwise use t. **Proportion Z-test.** Z-tests are generally preferred for proportions, since proportions (which follow a binomial distribution) are well approximated by a normal distribution at large sample sizes, per the Central Limit Theorem: $ z = \frac{(\hat{p} - p_0)}{\sqrt{\frac{p_0(1-p_0)}{n}}} $ For small sample sizes, **Fisher's exact test** or the **exact binomial test** should be used instead. When constructing a confidence interval (rather than testing against the null), an **unpooled** standard error is used, since you're no longer assuming the null hypothesis: $ SE_{unpooled} = \sqrt{\frac{p_t(1-p_t)}{n_t}+\frac{p_c(1-p_c)}{n_c}} $ For a **two-sample proportion test**, the standard error instead uses a pooled proportion: $ SE(p_1 - p_2) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n_1}+\frac{\hat{p}(1-\hat{p})}{n_2}}, \quad \hat{p} = \frac{n_1\hat{p_1}+n_2\hat{p_2}}{n_1+n_2} $ **Chi-square test of association (independence).** Evaluates whether a relationship exists between two categorical variables, without requiring a normality assumption — for example, "does gender relate to product preference?" or "does device type relate to conversion?" A contingency table (via `pd.crosstab()`) is built first, and **statistical independence** means the proportion of success in the response variable is the same across every category of the explanatory variable. Degrees of freedom is `(# response categories - 1) × (# explanatory categories - 1)`. Chi-square tests are almost always right-tailed, since the statistic measures squared deviations from expected values and is therefore always non-negative. ```python from scipy.stats import chi2_contingency table = pd.crosstab(df['test_group'], df['most_ads_day']) chi2_contingency(table) ``` **Chi-square goodness of fit.** Evaluates whether observed frequencies for a *single* categorical variable match an expected distribution — for example, testing whether a die is fair. Both the goodness-of-fit and independence tests use the same underlying formula: $ \chi^2 = \sum_{i=1}^{n}\frac{(O-E)^2}{E} $ where $O$ is the observed frequency and $E$ is the expected frequency (for the independence test, $E = \frac{\text{row total} \times \text{column total}}{\text{grand total}}$). **F-test.** Used to compare variances directly, or to compare multiple group means as in ANOVA: $F_{stat} = \frac{s_1^2}{s_2^2}$, where a larger F suggests the variances differ; for ANOVA specifically, $F_{stat} = \frac{\text{between-group variance}}{\text{within-group variance}}$, and exceeding the critical value leads to rejecting the null hypothesis. **Fisher's exact test.** Used for small samples with a binary outcome, typically framed as a 2×2 contingency table (treatment/control × converted/not converted). **Sequential test.** Allows continuous monitoring while still controlling the Type I error rate — for example, using a strict $\alpha$ (like 0.001) early on to avoid false positives, computing the SPRT's likelihood ratio $\frac{P(data \mid H_1)}{P(data \mid H_0)}$ continuously, or using a Bayesian approach that monitors $P(treatment > control)$ and stops once it exceeds 0.95. **Bayesian test.** Frequentist testing tends to fit large-scale product experiments, mature experimentation platforms, a clear fixed sample size, and simple launch decisions — the standard framework in most mature orgs. Bayesian testing tends to fit situations needing continuous monitoring, smaller sample sizes, sequential decision-making, business-friendly probability outputs, or adaptive experiments (common in ads and recommendation systems). See the dedicated [[Bayesian Testing]] note for more detail. **Nonparametric tests.** Used when the data doesn't satisfy the assumptions parametric methods require, such as normality, and tend to be more reliable on small samples — they rely on ranks or ordering rather than a specific probability distribution, and a violin plot is a useful way to display the underlying distribution. The **Wilcoxon signed-rank test** is the nonparametric alternative to the paired t-test, comparing two related measurements by ranking the absolute differences between them rather than comparing raw values (`scipy.stats.rankdata()` followed by `pingouin.wilcoxon()`). The **Mann-Whitney U test** is the nonparametric alternative for comparing two *independent* groups by ranking observations (`pingouin.mwu()`). The **Kruskal-Wallis test** is the nonparametric alternative to one-way ANOVA, comparing distributions across 2+ independent groups (`pingouin.kruskal()`). # 12. Analyze Results and Check Statistical Significance The **p-value** represents the probability of observing results this extreme (or more extreme), assuming the null hypothesis is true. The significance threshold $\alpha$ determines what counts as statistically significant and is directly tied to the Type I error rate: $\alpha=0.01$ is used when the cost of a Type I error is high, $\alpha=0.05$ is the common default, and $\alpha=0.1$ is used when there's greater tolerance for a false positive. The **confidence interval** gives the range of plausible values for the true treatment effect, providing an uncertainty estimate, an effect-size interpretation, and a read on practical significance all at once — it can be computed from bootstrap draws with `np.quantile(bootstrap_means, [0.025, 0.975])`, and it directly expresses the actual expected lift or drop at a $1-\alpha$ confidence level. **The multiple-testing problem.** When several tests run simultaneously, corrections like the **Benjamini-Hochberg** procedure or the **Bonferroni** correction are needed to control the rate of false discoveries. Once statistically significant variants are identified this way, they should still be evaluated on effect size before declaring a winner. For a large multi-arm test, this can be complemented with a **hierarchical Bayesian** model, which ranks variants by their probability of outperforming the baseline and stabilizes noisy estimates through partial pooling. ```python # Benjamini-Hochberg procedure, 50 tests, alpha=0.05 p_values = [0.001, 0.004, 0.01, ...] # ranked ascending rank = [1, 2, 3, ...] n = 50 thresholds = rank / n * alpha # use these per-rank thresholds instead of a flat 0.05 to decide which variants to keep ``` # 13. Interpret Practical Significance Statistical significance alone doesn't tell you whether an effect is worth acting on — that requires weighing the effect's practical meaning against the cost of implementation. **Cohen's d** is the standard tool here: a standardized effect size comparing the difference in group means relative to the pooled standard deviation. When treatment is added, this represents the minimum meaningful size the effect needs to reach — otherwise, what's observed may just be natural variation. $ Cohen's\ d = \frac{\mu_2-\mu_1}{SE} = \frac{\bar{x_1} - \bar{x_2}}{\sqrt{\frac{(n_1-1)var_1+(n_2-1)var_2}{n_1+n_2-2}}} $ A larger Cohen's d indicates a stronger treatment effect relative to the data's underlying variability: 0.2 is a small effect, 0.5 is medium, and 0.8 is large. Two other unstandardized effect-size measures are also useful: the **relative difference**, $\frac{\theta_2-\theta_1}{\theta_1}$, which is often used to express practical significance directly; and the **absolute difference**, $\theta_2-\theta_1$, which is the natural effect-size measure reported by Z-tests and t-tests. Finally, check whether the confidence interval extends beyond the pre-registered minimum effect size (e.g. 1%–5%) — this is what actually determines whether an effect clears the bar for practical significance, independent of its statistical significance. # 14. Check Risks and Side Effects Before recommending a launch, review the guardrail metrics defined back in §2 and look explicitly for any unintended consequences the primary metric alone might have missed. # 15. Make a Recommendation A launch recommendation should combine statistical significance, practical impact, risks and tradeoffs, the overall quality and validity of the experiment, side effects and guardrail metric performance, and consistency of the effect across segments. | Result | Recommendation | | -------------------------------------------------------------------- | ------------------------------------- | | Strong positive effect, with CI exceeding the expected effect size | Launch and scale | | Weak p-value, but CI spreads beyond the expected effect size | Rerun the experiment with higher power | | Weak p-value, and CI does not exceed the expected effect size | Do not launch | | Negative side effects observed | Do not launch, and redesign | After launch, because the experiment itself ran on a limited percentage of traffic over a short window, a **holdout set** — a small percentage (5%–10%) of the population deliberately kept off the change for an extended period — helps confirm the result actually generalizes to the full population over the long term. It's worth internalizing one subtlety here: a **wide confidence interval that crosses zero** means the experiment was underpowered — it simply can't distinguish a real effect from noise. That is *not* the same thing as "no effect." The right response is to check the power calculation retrospectively; if the experiment was underpowered, the result is uninformative rather than genuinely negative. # 16. Special Situations ## 16.1 What if the experiment duration has to be shortened? If there isn't enough time to reach the originally planned sample size, the goal becomes increasing statistical power *without* collecting more users. A few approaches help: 1. **CUPED (Controlled Experiments Using Pre-Experiment Data).** The idea is to use each user's historical behavior *before* the experiment — prior conversion rate, revenue, or engagement — as a covariate. Because past behavior is often highly correlated with future behavior, adjusting for it removes some of the natural user-to-user variability, which reduces the variance of the outcome metric. Lower variance means tighter confidence intervals and higher statistical power, which in practice can reduce the required sample size by 30–50% or more. See the dedicated [[CUPED]] note for more detail. 2. **Stratified randomization**, balancing users across treatment and control based on important characteristics (tenure, subscription plan, geography), and choosing a **more sensitive metric** that reacts faster to the treatment effect, both help extract more power from the same sample. 3. If a decision is genuinely needed early, a properly designed **sequential testing** framework or a **Bayesian** approach (see §11) can also be used. What should *not* be done is repeatedly checking p-values and stopping the moment significance appears — this inflates the Type I error rate and can produce incorrect conclusions unless a valid sequential-testing methodology is explicitly in place. ## 16.2 Switchback experiments Consider testing a new algorithm that determines which users get a prompt, and when. A standard A/B test fails here because the "treatment" is a **dynamic allocation policy**, not a static feature — you can't cleanly assign a user to "always get the prompt" vs. "never get it" without the two groups becoming systematically different over time (treated users who already converted are no longer eligible; control users keep accumulating). The answer is a **switchback experiment**: the policy is switched on and off across successive time periods (e.g. one week on, one week off), and aggregate outcomes are compared *across periods* rather than across users. A switchback experiment is really an A/B test where the randomization unit is a time window, not a user. This design carries its own key assumption: **no carryover effect between periods**, meaning a user's behavior during an "off" week isn't still being influenced by what happened during the preceding "on" week. If carryover does exist, longer washout periods are needed between switches. The natural unit for this kind of test is something like `city × time window`, analyzed with a t-test/z-test at that aggregated level (not at the individual-user level), or with a regression like `revenue ~ C(treatment) + C(city) + C(hour)`. ## 16.3 Simpson's Paradox in A/B testing Simpson's Paradox occurs when the aggregate result reverses after the data is segmented, due to a hidden confounding structure across subgroups (see the Statistical Knowledge note for a full numeric example). To avoid drawing a misleading conclusion, analyze the treatment effect across important dimensions — device type, geography, new vs. existing users, acquisition channel — and use segment-level consistency checks alongside stratified randomization to confirm the overall effect genuinely represents what's happening within each subgroup. ## 16.4 When results flip back after rollout When an experiment shows improvement but the effect reverses after a full rollout, the cause generally falls into one of three buckets: - **Statistical error**: a Type I error, too small a sample size, a violated test assumption (independence, identical distribution, proper randomization), or ending the experiment early. - **Computational error**: a bug in the randomization logic, a system or data-pipeline instrumentation bug, or an unnoticed Simpson's Paradox in the original analysis. - **External factors**: a holiday effect, a seasonal effect (which can be checked by running an ARIMA forecast of the no-treatment trend and comparing it statistically against the treatment trend), a novelty effect that faded after rollout, or a shift in competitive dynamics.