# 1. What Is Difference-in-Differences Difference-in-Differences (DiD) estimates a treatment effect by comparing the *change* in outcomes over time for a treatment group against the change observed over the same period in a comparable control group. The key assumption underlying DiD is **parallel trends**: absent treatment, both groups would have followed similar trajectories over time. Critically, parallel trends does **not** require the two groups to start at the same *level** — only that their trends would have moved in parallel. This assumption is validated by examining pre-treatment trends (visualized with `sns.lineplot()`) and by running a regression with a treatment-by-post interaction term, where the interaction coefficient represents the causal effect estimate. Two further assumptions matter: there should be **no spillover effect** (the treatment shouldn't indirectly affect the control group), and **no concurrent interventions** (no other campaign or change should be running at the same time that could confound the estimated effect).  # 2. When to Use DiD DiD is a natural fit when working with **aggregated data** and **time-series data**, and especially when **randomization isn't feasible**. A common example is when the "treatment" is really a user's own choice — comparing Prime vs. non-Prime members, or feature-adopters vs. non-adopters — since you can't force a user to subscribe or adopt a feature, and the group membership is inherently dynamic rather than something you can randomly assign. A few illustrative use cases: - **Uber** introduces a ride-pooling feature in select cities to provide more affordable options and reduce carbon footprint. DiD can assess the feature's impact on ridership by comparing the change in ridership in cities that got the feature against cities that only kept the traditional ride options. - **Lyft** launches billboard ads across highways around San Francisco and Chicago to increase driver sign-ups. DiD can assess the change in driver sign-ups in those two cities compared to similar cities without billboards. - **CVS Pharmacy** offers a home diagnostic kit and mails promotional brochures to residents in select zip codes. DiD can measure the brochure's impact by comparing the change in kit sales in zip codes that received the mailing against zip codes that didn't. # 3. Running a DiD Analysis Step by Step ## 3.1 Step 1: Problem scoping Before any analysis begins, clearly define the intervention being studied, which group counts as "treatment" and which as "control," and the specific outcome metric the analysis will evaluate. Getting this scoping right up front avoids ambiguity later when interpreting the treatment-by-post interaction term. ## 3.2 Step 2: State the hypothesis and duration **Consider the total experiment duration.** Options generally fall into three buckets: (A) 1 to 4 weeks for a short-term analysis, (B) 1 to 6 months for a mid-term analysis, and (C) 6+ months for a long-term analysis. When using DiD to measure a causal effect, the typical experiment window is longer than the 1–2 weeks common in standard A/B tests run as randomized controlled trials (RCTs). This is because in a controlled experiment, a behavioral change from treatment is usually immediate, so 1–2 weeks is often enough to see a noticeable difference between groups. DiD, by contrast, often takes several weeks for the treatment to actually shift behavior, which is why a mid-term window of 1–6 months is usually the right choice; 6+ months is typically unnecessary overkill for most business questions. **Consider the pre- and post-intervention periods.** As a general rule of thumb, the pre-intervention period should be equal to or longer than the post-intervention period — this also helps the model capture and account for seasonality. A common choice is 3 months of pre-intervention data against 2 months of post-intervention data. ## 3.3 Step 3: Exploratory analysis A few things are worth checking before modeling begins. Unlike a standard randomized experiment, it's acceptable for DiD's treatment and control group sizes to be imbalanced. It's still worth checking for **spillover risk** between groups, and exploring the data's distribution, variance, mean, median, and standard deviation. It's also worth checking statistical power directly, using `TTestIndPower()`: ```python from statsmodels.stats.power import TTestIndPower power = TTestIndPower() power.solve_power(effect_size=0.2, nobs1=75, ratio=309/75, alpha=0.05) # np.float64(0.34107427712439303) ``` A power of 0.35 here means that if a true effect of this size exists, the study only has about a 34% chance of actually detecting it — this experiment would be underpowered, equivalent to roughly a 66% chance of a Type II error (a false negative). ## 3.4 Step 4: Validate the parallel trends assumption The most common validation method is visualizing the pre-treatment data directly, checking whether the treatment and control groups show any diverging upward or downward trend before treatment began. ![[Pasted image 20260531161146.png|361]] Beyond visual inspection, a few more formal checks help confirm the assumption: fitting a regression such as `smf.ols` where the `treatment × month` interaction term shows no effect during the pre-treatment period (assuming there's enough pre-treatment data to support this); checking the correlation between time and the pre-treatment difference between groups, which should be close to zero: ```python from scipy.stats import pearsonr corr, p = pearsonr(time, pre_empl_diff['employment']) print(corr, p) # -0.06904417087648847 0.5561295102500324 ``` checking whether the time series of the pre-treatment difference is stationary, using `statsmodels.tsa.stattools.adfuller`; running separate regressions for the treatment and control groups' pre-treatment data (e.g. `smf.ols("revenue ~ month", data=control_data).fit()`) and confirming their slope coefficients are similar; and finally, running the DiD estimator itself on the pre-treatment data alone, where it should show no effect. ## 3.5 Step 5: Model diagnostics Before trusting the regression output, check the standard set of regression assumptions: normality of residuals, constant variance of residuals (homoscedasticity), a linear relationship between predictors and the outcome, no or low multicollinearity, no autocorrelation, and no highly influential outliers. ## 3.6 Step 6: Run the DiD regression and estimate the causal effect In a DiD model, a low R² is not necessarily a problem, because the objective here is **causal identification**, not predictive accuracy. What actually matters is whether the treatment-interaction coefficient is statistically significant, economically meaningful, and supported by valid assumptions like parallel trends. The standard DiD regression specification is: $ Y=\alpha+\beta \cdot Treatment+\gamma \cdot Post+\delta \cdot (Treatment \times Post)+\epsilon $ Here, $Treatment$ captures the baseline difference between whoever is or isn't in the treatment group, $Post$ captures any organic lift that happened in the post-period regardless of treatment, and $Treatment \times Post$ — the coefficient $\delta$ — is the actual DiD causal effect estimate. To analyze the model's output, check the p-value, confidence interval, variance, and effect size (the model coefficient itself), and calculate Cohen's d for a standardized effect size. If the p-value isn't below $\alpha$, check the confidence interval, statistical power, and Cohen's d together to determine whether the null result is actually due to low sample size and low power, before concluding whether to accept, reject, or rerun the experiment. ## 3.7 Step 7: Recommendation on launch Once the causal effect is estimated, translate it into a business recommendation the same way you would for any experiment: weigh the statistical significance and practical size of the effect against the cost of implementing and maintaining the change, and be explicit about the confidence interval around the estimate rather than presenting a single point estimate as certain. # 4. Limitations of DiD 1. **Parallel trends assumption.** The core requirement is that, absent treatment, the treated and control groups' average outcomes would have followed the same trend over time. If the trends aren't actually parallel, a different causal inference method — such as synthetic control or Bayesian structural time series — may be more appropriate. 2. **Confounding factors.** Any time-varying factor that could independently affect the treatment effect needs to be identified and controlled for in the model. The pre- and post-intervention comparison can easily be disrupted by unrelated factors like economic fluctuations or a competitor's actions occurring during the same window. # 5. Appendix - [Project](https://colab.research.google.com/drive/1mv0yAwr78_Jy6o0eyapEnQ-w4-RyuQaw#scrollTo=a8dhfkhLrsP3)