# 1. What Is Propensity Score Matching Propensity Score Matching (PSM) is an observational causal-analysis method used when a randomized experiment isn't available and treatment was instead assigned (or self-selected) through some real-world process. The **propensity score** is the estimated probability that a given unit received treatment, based on its observed pre-treatment characteristics — for example, a particular user might have an estimated 85% chance of receiving treatment given their observed attributes. PSM is sometimes described as a synthetic control method, since it constructs a comparable "control" group after the fact rather than relying on the balance that a genuine randomized assignment would have produced. The overall PSM workflow follows four stages: first, estimate each unit's treatment probability from its pre-treatment covariates; second, match treated units to comparable control units based on those estimated probabilities; third, check that the matched groups are actually balanced on the observed covariates; and fourth, compare the outcome of interest (for example, `quantity_sold` or `revenue`) between the matched treated and control groups. # 2. When Is PSM Useful Before reaching for PSM, it's worth working through a short checklist to confirm it's actually the right tool for the problem: 1. **Was treatment randomly assigned?** If yes, PSM isn't needed — the whole point of randomization is that it already balances observed and unobserved characteristics between groups. 2. **Could treatment assignment depend on observable characteristics?** If yes, PSM might help, since it can correct for imbalance driven by variables you actually measured. 3. **Do the treatment and control groups look different?** Check the mean of covariate features by group, or use `pd.crosstab()` for categorical variables, and run a statistical test on the differences. If many variables differ meaningfully, the groups aren't directly comparable, and PSM might help close that gap. 4. **Can treatment assignment be reasonably explained by observed data?** If treatment was assigned based on something measurable, like price or discount level, PSM is a good fit. If treatment was assigned based on something unmeasured, like what a manager "felt like" doing, there's no way to quantify or control for that with PSM — this is **unobserved confounding**, and it calls for a different method entirely, such as difference-in-differences, instrumental variables, or regression discontinuity design. 5. **Is the sample size large enough?** Check that neither the treatment nor control group is too small, since small samples make the matching process unstable. 6. **Is there common support after matching?** If, for example, treated units' propensity scores cluster between 0.8 and 1.0 while control units cluster between 0 and 0.2, there's no overlap between the groups, meaning no comparable controls exist for the treated units — and PSM will fail under these conditions. # 3. Exploring the Data Before Modeling ## 3.1 Explore covariates Use `df.groupby('treatment')[num_cols].agg(['mean','std']).T` for numeric features, and `pd.crosstab()` for categorical features, to check whether the treatment and control groups are balanced and to spot any strong pre-existing differences between them. ## 3.2 Explore confounders Use a quick `logit()` or `ols()` fit relating candidate covariates to both the treatment indicator and the target outcome, to see whether any variable is influencing both — that's the signature of a confounder. Once a confounder is identified, the right response is generally not to remove it from the analysis. Instead, control for it by including it in the post-matching balance check, re-estimating the treatment effect, and considering a regression-adjustment step that includes the confounder directly in the model fit after matching. ## 3.3 Consider seasonality If the outcome or treatment assignment has a seasonal pattern — for example, treatment was more common during a particular time of year, or the outcome naturally fluctuates seasonally — that pattern can masquerade as a treatment effect if it isn't accounted for. It's worth explicitly checking whether treated and control periods overlap in time, and including time-based controls (month, week, or a time trend) in the propensity or outcome model when seasonality is present. ## 3.4 Handle multicollinearity Use `sns.heatmap()` on `corr()` alongside VIF (variance inflation factor) scores to check for multicollinearity among the covariates. Correlation captures pairwise relationships between two variables, while VIF checks whether one variable can be predicted from all the other variables combined — a more complete picture of redundancy. Variables with high VIF can typically be removed without losing much information, since their signal is already captured by other covariates. # 4. Determining Whether PSM Is Actually Required Before committing to the full PSM pipeline, a few diagnostic checks can confirm whether it's genuinely necessary: - **Check treatment/control balance** on the raw covariates, as described in §3.1. - **Compare covariate balance using the Standardized Mean Difference (SMD)**: $SE_{pool} = \sqrt{\frac{var_t+var_c}{2}}$ and $SMD = \left|\frac{mean_t-mean_c}{SE_{pool}}\right|$ computed for each covariate. An absolute SMD greater than 0.1 indicates meaningful imbalance on that variable. - **Check whether treatment status is predictable from the covariates.** If it is, PSM is likely needed. This is tested by splitting the data and evaluating a classification model's AUC in predicting treatment assignment, using only **pre-treatment features** — the outcome variable, any treatment-created variables, and any post-treatment behavior should never be included in this check. ``` AUC ≈ 0.50 → treatment looks random; PSM may not be needed AUC > 0.70 → treatment is predictable; selection bias likely AUC > 0.80 → strong selection bias; PSM/causal adjustment recommended ``` # 5. Checking Distributional Assumptions Logistic regression, which is commonly used to estimate the propensity score, does not require the feature covariates to be normally distributed. Normality only becomes relevant later, as a basic assumption when testing the target/outcome column. That said, it's still worth checking for outliers, skewness, and the overall distribution shape of the feature columns, since large outliers can still degrade a logistic model's performance even without a normality requirement. Once a skewed feature is identified, it can be **log-transformed** with `np.log1p()` or have its extreme values **winsorized** at, say, the 99.9th percentile — outliers generally shouldn't simply be removed. Tree-based models used for propensity estimation are inherently less sensitive to outliers than logistic regression. ``` from scipy.stats import skew for col in num_cols: print(col, skew(df_groupon[col])) # |skew| < 1: good, |skew| > 2: highly skewed ``` # 6. Fitting the Propensity Model The propensity model itself is typically fit with `sklearn.linear_model.LogisticRegression` or `statsmodels.formula.api.logit()`. After fitting, plot the density of the predicted propensity scores separately for the treatment and control groups — this visually confirms whether there's good overlap between the two, a condition referred to as **common support**. ![[Pasted image 20260601122011.png|334]] # 7. Matching Methods ## 7.1 Nearest-neighbor and caliper matching `sklearn.neighbors.NearestNeighbors` can be fit on the control group's propensity scores to find, for each treated unit, the closest comparable users in the control group. Other matching approaches include stratification matching and kernel matching. A **caliper** sets the maximum allowable distance between the propensity scores of a treated unit and its matched control. Without a caliper, the matching algorithm will always find the *closest* available control, even if that control is still quite far away in propensity-score terms — which produces poor matches. Applying a caliper improves covariate balance, prevents these poor matches, and reduces bias overall. A common rule of thumb is to set the caliper at **0.2 × the standard deviation of the logit-transformed propensity score**. | Method | Description | Advantage | | ---------------- | -------------------------------------------------- | ----------------------------------------- | | KNN | closest propensity score | simple, widely used | | caliper matching | nearest neighbor within a maximum distance | better balance | | radius matching | all controls within a specified radius | uses more information | | 1:k matching | multiple controls per treated unit | lower variance | | optimal matching | minimizes total matching distance across all pairs | better global matching | | full matching | matches on multivariate covariate distance | effective with few continuous covariates | | genetic matching | uses an optimization algorithm to improve balance | excellent covariate balance | ## 7.2 Matching ratios Once propensity scores are estimated, several strategies exist for actually pairing treated and control units into a balanced comparison sample: 1. **1-to-1 matching** pairs each treated unit with the single control unit that has the closest propensity score. A given control unit can be reused across multiple treated units during this process. 2. **K-to-1 matching** pairs each treated unit with its $k$ closest control units, and again allows a control unit to be reused. The outcome comparison then averages across the matched controls — for example, with three matched controls, $diff = Y_t - \frac{Y_{c1}+Y_{c2}+Y_{c3}}{3}$. This produces lower variance and more stable estimates than 1-to-1 matching, and makes better use of the available pool of control units. 3. **Stratified (interval) matching** divides the range of propensity scores into intervals, or strata, and compares treated and control units within each stratum. This ensures matches are made within similar score ranges, producing more homogeneous comparison groups than a simple nearest-neighbor approach. # 8. Model Diagnostics and Assumptions Applying PSM correctly requires several assumptions to hold: 1. **Conditional Independence Assumption (CIA):** the most crucial assumption in PSM — that all confounding variables influencing both the treatment and the outcome have been measured and included in the model. In other words, once you condition on the propensity score, treatment assignment behaves as if it were random. 2. **Common support (overlap condition):** for every combination of covariate values, there must be a positive probability of observing both treatment and control units. This ensures every treated individual has a comparable counterpart in the control group, and vice versa. 3. **Stable Unit Treatment Value Assumption (SUTVA):** this requires that one individual's outcome is not affected by another individual's treatment status (no interference between units), and that there is only a single, well-defined version of the treatment (no hidden variation in what "treatment" actually means). 4. **Balancing hypothesis:** after matching, the distribution of covariates should be similar across the treatment and control groups, confirming that the matching process successfully balanced the observed covariates. 5. **Normality of the sampling distribution:** relevant when applying a t-test or other parametric test that assumes the sampling distributions of the treatment and control groups are normal. If this assumption fails, consider a log transformation of the outcome or a non-parametric test instead. # 9. Checking Covariate Balance After Matching After matching, recheck the mean of each covariate across the new treatment and control groups to confirm they're now comparable. The **standardized mean difference (SMD)** is the standard tool for this: an absolute SMD below 0.1 indicates good balance. | metric | meaning | direction of "better" | | -------------------------- | ------------------------------------------------------------ | ---------------------- | | matched_treated | number of treated units successfully matched | higher | | treated_match_rate | percentage of treated units retained after matching | higher | | average_match_distance | average propensity-score distance between matched pairs | lower | | max_abs_smd_after | worst remaining covariate imbalance after matching | lower | | mean_abs_smd_after | average covariate imbalance after matching | lower | | ATT | estimated average treatment effect on the treated | depends on the study, not itself a quality metric | # 10. Outcome Analysis Once the matched sample is confirmed to be balanced, the actual treatment effect is estimated by running a statistical test on the outcome of interest — a t-test, z-test, or Mann-Whitney U test, depending on the outcome's distribution — and bootstrap resampling can be used to construct a confidence interval around the estimated effect. ![[Pasted image 20260601122641.png|354]] # 11. Limitations of Propensity Score Matching PSM has several important limitations worth keeping in mind before relying on its results: 1. **Unmeasured confounders.** PSM can only correct for confounders that were actually observed and included in the model — any unmeasured or unknown confounder can still bias the estimated treatment effect. 2. **Quality of the matching.** The effectiveness of PSM depends heavily on how well the propensity model and matching procedure are specified. A poorly specified model won't adequately control for confounding, no matter how carefully the matching itself is executed. 3. **Reduction in sample size.** Matching typically discards unmatched units, which reduces the effective sample size and, with it, the statistical power of the analysis. 4. **Assumption of homogeneous treatment effects.** PSM implicitly assumes the treatment effect is similar across individuals, which may not hold in practice — some subgroups may respond very differently to treatment than others. 5. **Reliance on observational data.** Like any method built on observational data, PSM cannot fully replicate the conditions of a randomized controlled trial, so any resulting causal claims are inherently weaker than those from a true experiment. # 12. Handling Many Treated Units but Few Control Units When treated units substantially outnumber available control units, a few strategies help make the most of a limited control pool: - **Matching with replacement** allows the same control unit to be reused across multiple treated units, which gives every treated unit the best possible match and produces lower bias — but some controls end up carrying disproportionately large weight, which reduces the effective sample size and increases variance. **Matching without replacement**, by contrast, uses each control only once, which avoids that weighting problem but means later treated units may be stuck with progressively worse matches once the best controls have already been claimed. - **Inverse probability weighting** reweights units by the inverse of their estimated propensity, rather than discarding unmatched units outright. - **Full matching** and **optimal matching** (see §7.1) make more efficient use of the available controls by jointly optimizing the match structure across all units rather than matching greedily one at a time. - **Overlap weighting** weights each unit by its probability of being in the opposite group, naturally emphasizing the region of common support. In extreme cases, PSM may end up discarding a large share of treated units simply because there aren't enough comparable controls to match them to. ![[Screenshot 2026-07-14 at 11.46.11 AM.png]] # 13. Appendix - [Project](https://app.hex.tech/019eb406-8c5e-755b-986f-e0693fc2a83c/hex/PSM-groupon-usage-033WGEnKiWYp8Q5izY9WYq/draft/logic) - [Project Framework](https://app.hex.tech/019eb406-8c5e-755b-986f-e0693fc2a83c/hex/Untitled-project-033p8tyCzk0QCIHswiTaY0/draft/logic)