# 1. What Is a Multivariate Experiment A multivariate experiment tests **multiple factors simultaneously** and measures two distinct kinds of effects: **main effects**, which capture the impact of each factor independently (the effect of factor A, and separately the effect of factor B), and **interaction effects**, which capture whether the impact of one factor depends on the level of another factor (the effect of A combined with B, written A×B). Unlike a standard A/B test, which evaluates a single change in isolation, multivariate testing lets you understand how multiple features work together. For example, if factor 1 is button color and factor 2 is headline text, testing both together produces $2 \times 2 = 4$ total variants — this is called a **factorial design**. # 2. Why Use Multivariate Testing A standard A/B test can only answer a narrow question like "does a green button improve conversion?" A multivariate test can instead answer richer questions like "which combination of button color and headline maximizes conversion?" and "does the best-performing headline depend on which button color it's paired with?" This second kind of question is exactly what an **interaction effect** captures: it exists when one factor's effectiveness genuinely depends on the level of another factor. In the absence of an interaction effect, you can safely assume each factor works consistently regardless of what the other factors are set to — which is a much stronger (and often incorrect) assumption than it first appears. # 3. Running a Multivariate Experiment **Continuous outcome.** For a continuous metric like revenue, fit a linear regression model that includes both main effects and their interaction term, for example using `statsmodels.formula.api as smf` to fit `revenue ~ C(color) + C(headline) + C(color):C(headline)`. Alternatively, `statsmodels.api as sm` can run an ANOVA test via `sm.stats.anova_lm()` to compare two or more groups directly. **Binary outcome.** For a binary metric like conversion (converted vs. not), fit a logistic regression model instead, using `smf.logit` with the same formula structure. # 4. Visualizing and Interpreting Results **Interaction plots** are the standard way to visualize whether an interaction effect is present, for example: ``` sns.pointplot(data=df, x='color', y='conversion_rate', hue='headline') ``` If the lines for each level of `headline` are roughly **parallel** across the levels of `color`, there is no meaningful interaction effect — each factor is acting independently. If the lines **cross or diverge** noticeably, that's a sign of a strong interaction, meaning the best choice of one factor genuinely depends on the level of the other. # 5. Sample Size, Assumptions, and Validity Checks **Sample size.** A major drawback of multivariate testing is that it requires a larger overall sample size than a simple A/B test, because traffic gets split across more variant combinations, which reduces the statistical power available to detect an effect within any single group. **Randomization and selection bias.** Check that groups are balanced using `pd.crosstab()`, and assess group heterogeneity — comparing the mean of relevant covariates across groups — to determine whether a propensity score matching (PSM) adjustment is needed to correct for imbalance. **SUTVA (Stable Unit Treatment Value Assumption).** This assumption requires that one user's assigned treatment does not affect another user's outcome — violations of this (for example, through social sharing or network effects) can bias the experiment's results. **Sample ratio mismatch.** Use a chi-square test to check whether the actual allocation of users across variants matches the intended allocation ratio; a significant mismatch usually points to a bug in the randomization or logging pipeline rather than a real effect. **OLS model assumptions**, relevant when the outcome is continuous: - **Independence**: observations should not be correlated with one another. - **Homogeneity of variance**: the outcome's variance should be similar across groups, which can be checked using `from scipy.stats import levene`. - **Residual normality** (checked on the model's residuals, not on the raw outcome variable): extract `resid = model.resid`, visualize with `sns.histplot(resid)`, and formally test normality with `stats.shapiro(resid.sample(5000))`. - **No severe multicollinearity**: check whether the experimental factors are highly correlated with each other, which would make their individual effects hard to separate. **Logistic regression assumptions**, relevant when the outcome is binary: - **Independence**: observations should not be correlated with one another. - **Sufficient positive-class sample size**: there need to be enough observations where the binary outcome equals 1 for the model to estimate stable coefficients. - **No perfect separation**: this occurs when a specific combination of factor levels perfectly predicts the outcome — for example, if every user who saw the "green button + headline A" combination converted at 100% while every other combination converted at 0%. Perfect separation breaks logistic regression's estimation and needs to be checked for directly. # 6. Full Factorial vs. Fractional Factorial Designs A **full factorial** design tests every possible combination of factor levels, which lets you estimate all main effects and all interactions with the highest possible accuracy — but it requires a correspondingly large sample size, since the number of variants grows multiplicatively with the number of factors and levels. A **fractional factorial** design tests only a carefully chosen subset of the possible combinations. This lowers the traffic requirement and lets the experiment run faster, at the cost of not being able to estimate some of the interaction effects. Fractional factorial designs are commonly used in large-scale product experimentation, where testing every combination simply isn't practical. # 7. Worked Example: Re-Engaging Churned Users The goal of this example is to determine the most effective way to re-engage users who have churned from Uber Eats. | Hypothesis | Treatment | | ---------- | --------------------------------------------------------------------- | | H0 | No treatment | | H1 | Send a reminder email: *"Order on Uber Eats."* | | H2 | Send a promotional email: *"Get 15% off your next Uber Eats order."* | **Why include H1?** A natural question is why not simply compare H0 (no email) against H2 (discount email) directly. The reason is that including H1 lets you isolate the effect of sending an email at all from the effect of the discount specifically. Without H1, if H2 outperforms H0, all you know is that the overall campaign works — you can't tell whether the improvement comes from simply reminding users that the app exists, or from the financial incentive of the discount. Adding H1 separates these two effects cleanly: the **H0 vs. H1** comparison measures the impact of sending a reminder on its own, the **H1 vs. H2** comparison measures the incremental value the discount adds on top of a reminder, and the **H0 vs. H2** comparison measures the total impact of the full campaign. Together, these comparisons help determine whether a discount is actually necessary, or whether a simple reminder email is sufficient on its own. **Choosing the primary metric.** A good experiment needs a metric that genuinely aligns with the business objective, and the two natural candidates here have different tradeoffs. *Option 1: First purchase rate* — the percentage of churned users who place at least one order within 7 days of the treatment. This is easy to measure and clearly indicates successful reactivation, but it risks overestimating long-term success, since some users may return only to redeem the discount once and then churn again immediately after. *Option 2: Repeat purchase rate (preferred)* — the percentage of churned users who place a second order shortly after returning (for example, within one week of their first post-treatment order). This is the stronger metric because the real objective isn't to generate a single discounted purchase — it's to reactivate users as ongoing customers. A user who returns because of the email or discount and then makes a second purchase without needing an additional incentive is much more likely to have genuinely become an active customer again, which is why repeat purchase rate is a better indicator of sustainable re-engagement than first purchase rate alone. **Interpreting the experiment results.** Three broad outcomes are possible, and each implies a different business decision: *Scenario 1 — no significant difference.* Neither H1 nor H2 significantly outperforms H0. This means the campaign, as designed, is ineffective, and it's worth testing alternatives: different messaging, larger or different incentives, alternative communication channels (push notifications, SMS, in-app messages), or different user segments. *Scenario 2 — H1 and H2 both outperform H0, but H1 ≈ H2.* This means sending an email successfully re-engages users on its own, and the discount adds little or no additional benefit. The business implication is to use H1, since it achieves similar performance to H2 while avoiding the cost of the discount and preserving profit margins. *Scenario 3 — H2 outperforms both H0 and H1.* This means the discount generates meaningful incremental lift beyond what a reminder email achieves on its own. The business implication is to use H2 for churned users, provided the additional revenue it generates exceeds the cost of the discount itself. **Follow-up experiment: optimizing H2.** Suppose H2 is selected as the winning treatment, and the next objective becomes optimizing the campaign itself. Several sub-factors are worth testing: the discount percentage, the email's message tone, and the email's subject line. Rather than running a separate A/B test for each of these components one at a time, a **factorial experiment** evaluates all of them simultaneously: | Factor | Levels | | ------------ | ----------------------- | | Discount | 10%, 15%, 50% | | Message | Friendly, Urgent | | Subject line | Short, Personalized | This produces $3 \times 2 \times 2 = 12$ total combinations, with the previously validated H2 version serving as the baseline and users randomly assigned across all 12 variants. **Analysis strategy.** Rather than comparing every pair of treatment combinations against each other, each variation is compared against the baseline: for each variation, estimate its lift relative to the baseline, calculate a confidence interval and p-value for that lift, and identify which improvements are statistically significant. Because testing 12 variants against a baseline involves many simultaneous comparisons, which inflates the chance of a false positive somewhere in the set, a multiple-testing correction should be applied — options include the **Bonferroni correction** (conservative, simple to apply), **Tukey's Honest Significant Difference (Tukey HSD)** for pairwise comparisons specifically, or the **False Discovery Rate** procedure (e.g. Benjamini-Hochberg) when evaluating a larger number of variants at once. # 8. Appendix - [Project](https://app.hex.tech/019eb406-8c5e-755b-986f-e0693fc2a83c/hex/Speaker-Sales-Experiment-033XhpHRJM4KGysykPbRyj/draft/logic)