# 1. A Practical Feature Selection Workflow Feature selection is best approached as a sequence of increasingly sophisticated filters, moving from cheap and obvious cuts to more expensive, model-aware methods. ## 1.1 Remove obviously bad features first Before doing any real analysis, strip out features that clearly can't help: constant or near-constant features, identified with `VarianceThreshold`, and features with a very high proportion of missing values, checked with `x.isna().mean()`. These features either carry no signal or too little reliable data to be trustworthy. ## 1.2 Remove multicollinearity Next, check for redundancy among the remaining features using a correlation matrix (`corr()`) and the variance inflation factor (VIF). Highly collinear features don't add independent information and can destabilize model coefficients, so they're worth addressing early, before more sophisticated selection methods are applied. ## 1.3 Filter methods: univariate feature selection Filter methods evaluate each feature's relationship to the target independently, without involving a specific model, which makes them fast and useful for an initial screening pass. The right technique depends on the type of target and feature: for a **continuous target**, `sklearn.feature_selection.f_regression` combined with `SelectKBest` ranks features by their individual relationship to the outcome; for a **classification target**, `f_classif` does the same for categorical outcomes; **mutual information** (`mutual_info_regression`) captures nonlinear relationships that a purely linear filter method would miss, and works for both classification and regression; and for **two categorical variables** — a categorical feature against a categorical target — the `chi2` test is the appropriate choice. ## 1.4 Embedded methods Embedded methods perform feature selection as a byproduct of training the model itself, using techniques like Lasso regression (which can shrink irrelevant coefficients to exactly zero) or tree-based models (which naturally produce feature importance scores as part of fitting). ## 1.5 Wrapper methods: recursive feature elimination Wrapper methods iteratively test different feature subsets against actual model performance, which is more expensive than filter methods but accounts for interactions between features. `sklearn.feature_selection.RFECV` repeatedly removes the least useful feature until performance stops improving; for classification tasks, pair it with `StratifiedKFold`, and for general cross-validation, `KFold`. It's worth comparing the resulting models at several feature-count checkpoints — for example, 60 features vs. 40 vs. 25 vs. 15 — to see where performance actually plateaus. **Sequential feature selection**, available via `SequentialFeatureSelector`, offers a similar iterative approach in both forward (starting from no features and adding) and backward (starting from all features and removing) directions. ## 1.6 SHAP for interpretability When using gradient-boosted models like XGBoost, CatBoost, or LightGBM, the `shap` library is generally the best tool for interpretability: it provides global feature importance, local (per-prediction) explanations, and insight into interactions between features — going well beyond what a simple importance ranking can show. ## 1.7 Consider feature transformations Finally, it's worth considering whether combining existing features produces a more informative signal than either does alone — for example, calculating a rate from two related features (like revenue per session rather than revenue and sessions separately) can surface a relationship that neither raw feature captures on its own.