## What problem does CUPED solve Many experiment metrics have **high variance** because users naturally behave very differently. | user | pre X | experiment y | | ---- | ----- | ------------ | | a | 100 | 120 | | b | 5 | 10 | | c | 200 | 240 | Without any adjustment, these large user-to-user differences make it harder to detect treatment effects. CUPED uses **pre-experiment behavior** to explain part of this natural variation, reducing noise while leaving the treatment effect unchanged. ## Intuition Instead of analyzing the raw experiment metric $Y$, CUPED analyzes an adjusted metric $Y_{CUPED}=Y-\theta(X-\bar{X})$ Where: $Y$: experiment metric $X$: pre-experiment metric $\bar{X}$: mean of the pre-experiment metric $\theta$: adjustment coefficient Users who normally spend much more (or much less) than average are adjusted toward the population mean, removing predictable variation. ## Choose adjustment coefficient The optimal coefficient minimizes the variance of the adjusted metric: $\theta=\frac{COV(Y,X)}{Var(X)}$ This is equivalent to the slope from a simple linear regression of $Y$ on $X$. ## Variance reduction After applying CUPED, $Var(Y_{CUPED})=Var(Y)(1-\rho^2)$ where $\rho = Corr(X,Y)$ $\rho$: is the correlation between the pre-experiment and experiment metrics. - If $\rho$=0, the pre-experiment metric provides no information, so CUPED offers **no improvement**. - If $\rho$=0.8, $1-0.8^2=0.36$, meaning only **36% of the original variance remains**, corresponding to a **64% variance reduction**. The stronger the correlation between pre-period and experiment behavior, the greater the variance reduction. ## Why Variance Reduction Matters Lower variance leads to more powerful experiments: - Detect statistically significant effects **more quickly**. - Detect **smaller** treatment effects. - Require **fewer** users to achieve the same statistical power. - Produce more **stable and precise** estimates. - Shorten experiment **duration**. In practice, CUPED allows experiments to reach significance faster without changing the experiment itself. ## Example Revenue during experiment: User A normally spends $100 User B normally spends $5 User C normally spends $500 instead of analyzing Y, analyze $Y_{cuped}=Y - \theta(X-\hat{X})$ $Y$: experiment metric $X$: pre-experiment metric $\hat{X}$: mean of pre-experiment metric $\theta$: adjustment coefficient Choose $\theta$ to minimize variance: the optimal value is $\theta = \frac{Cov(Y,X)}{var(X)}$ Variance becomes: $Var(Y_{cuped}=Var(Y)(1-\rho^2)$ where $\rho=Corr(X,Y)$ This shows if there is no correlation between pre and experiment outcome, there there is no improvement, if $Corr(X,Y)=0.8$ then $1-0.8^2=0.36$ there are 64% variance reduction | user | pre X | experiment y | cuped y | | ---- | ----- | ------------ | ------- | | a | 100 | 120 | 121.97 | | b | 5 | 10 | 124.07 | | c | 200 | 240 | 123.97 | $\hat{X}=101.67$, -> $X-\hat{X}$ and calculate $\theta$ ## Key Takeaways - CUPED is a **variance reduction technique** for A/B testing. - It leverages **pre-experiment data** to remove predictable user-level variation. - The adjustment does **not** introduce bias or change the expected treatment effect. - The effectiveness of CUPED depends on the correlation between pre- and during-experiment metrics. - High correlation can dramatically reduce variance, enabling **smaller sample sizes**, **shorter experiments**, and **greater sensitivity to treatment effects**.