## 1. What it's for ANOVA (Analysis of Variance) tests whether **3+ group means are equal** on continuous data not binary data such as conversion. - H0: all group means are equal - H1: at least one group mean differs - ANOVA tells you that groups differ, not which — that's what post-hoc tests are for (step 4) ## 2. Explore the data - plot each group (box-plot / histogram) — look for shape, spread, obvious outliers - get sample size (n), sample mean, sample variance (std) per group - outliers and small sample size can violate the assumption ![[Screenshot 2026-07-15 at 11.12.23 AM.png|374]] ## 3. Check assumptions 1. **independence** — observations (and groups) aren't related to each other, this is related to how the data was collected 2. **normality within each group**, plus a QQ plot per group for a visual sanity check 1. if sample < 5000 → `Shapiro-Wilk test` 2. else → `normaltest` (D'Agostino-Pearson) — both from `scipy.stats` 3. Shapiro loses power at large sample size (flags trivial deviations as significant), hence we use D'Agostino-Pearson normal test 3. **equal variance across groups (homogeneity of variance)** using `levene` (`scipy.stats.levene`) — Levene's is preferred over Bartlett's since it's robust to non-normality 4. **sample size match**: use chi-square goodness of fit to test actual sample size % vs expected even % ## 4. Choose the appropriate test | normal | equal variance | test | | ------ | -------------- | ---------------------------------------------------- | | yes | yes | one-way ANOVA using `f_oneway` | | yes | no | Welch ANOVA using `pingouin.welch_anova` | | no | — | Kruskal-Wallis using `kruskal` or `permutation` test | - one-way ANOVA / Welch ANOVA: parametric, compare means - Welch ANOVA doesn't assume equal variance (it's the ANOVA analog of Welch's t-test) — prefer it over standard ANOVA by default when in doubt, since equal variance is a strong assumption that's often violated in practice - Kruskal-Wallis: non-parametric, compares mean *ranks* instead — use whenever normality fails, regardless of variance - Kruskal–Wallis tests distributional/rank differences rather than directly testing equality of means. If your research question is explicitly about **group means**, permutation ANOVA may be preferable ## 5. Post-hoc test Only run if the overall test in step 4 is **significant** (p < 0.05) — otherwise there's nothing to localize. | overall test | post-hoc test | | --------------------------- | --------------------------------------------- | | one-way ANOVA | `pairwise_tukeyhsd` (statsmodels) — Tukey HSD | | Welch ANOVA | `pingouin.pairwise_gameshowell` — Game's howell Test | | Kruskal-Wallis | `scikit_posthocs.posthoc_dunn` — Dunn's Test | | Permutation ANOVA | `permutation_test` — Pairwise permutation Test | - both control the family-wise error rate across all pairwise comparisons (that's why you can't just run repeated t-tests instead) - Dunn's test needs an explicit p-value adjustment method — Bonferroni is a common, conservative default ## 6. Effect size and power Significance says *whether* groups differ; effect size says *how much*. - **eta-squared (η²)** — proportion of total variance explained by group membership ($\frac{SS_{between}}{SS_{total}}$). Simple, but slightly biased upward, especially with small samples - **omega-squared (ω²)** — bias-corrected version of η²; preferred when comparing effect sizes across studies/sample sizes - **Cohen's f** — derived from η², used as the effect-size input for power analysis (small ≈ 0.10, medium ≈ 0.25, large ≈ 0.40), showing group difference - **power** — probability of correctly detecting a true effect, given the effect size, n, k groups, and $\alpha$; low power on a non-significant result means "inconclusive," not "no effect", underpowered study - **MDE** — small MDE showing the study can only reliable detect effects >= MDE, if there is very small effect, then the study sample size is not powerful - **effect** — small effect means the observed effect is smaller than the study is designed to detect ![[Screenshot 2026-07-15 at 11.13.20 AM.png]] ## 7. Two-way ANOVA Adds a second factor, so you can test both factors' effects plus how they interact. - main effect of factor A, main effect of factor B, and the A×B **interaction** effect - a significant interaction means the effect of one factor depends on the level of the other — interpret main effects cautiously when this happens, and check an interaction plot before reading main effects at face value - same assumption checks apply (normality of residuals, homogeneity of variance — e.g. Levene across all cells), typically implemented via `statsmodels` (`ols` + `anova_lm`) - effect size: partial $\eta^2$ per term, since total variance is now split across more sources ## Quick reference | step | function | | ---------------- | ------------------------------------------------------------------ | | normality | `scipy.stats.shapiro` (n<5000) / `scipy.stats.normaltest` (n≥5000) | | equal variance | `scipy.stats.levene` | | one-way ANOVA | `scipy.stats.f_oneway` | | Welch ANOVA | `pingouin.welch_anova` | | Kruskal-Wallis | `scipy.stats.kruskal` | | post-hoc (ANOVA) | `statsmodels.stats.multicomp.pairwise_tukeyhsd` | | post-hoc (K-W) | `scikit_posthocs.posthoc_dunn` | | two-way ANOVA | `statsmodels.formula.api.ols` + `statsmodels.stats.anova.anova_lm` | ## Appendix [Project framework](https://app.hex.tech/019eb406-8c5e-755b-986f-e0693fc2a83c/hex/ANOVA-Playground-033phqyQCTDouoa7xEeWky/draft/logic)