Datasets often contain highly skewed variables and extreme outliers. Before applying machine learning algorithms such as **KMeans**, appropriate preprocessing is essential because KMeans relies on Euclidean distance and is sensitive to variable scales and extreme values.
# Step 1. Measure Feature Skewness
First, calculate the skewness of each numerical feature.
```python
skew = (
df[num_cols]
.skew()
.sort_values(key=np.abs, ascending=False)
)
display(skew)
```
### Skewness Interpretation
| Skewness | Interpretation | Recommended Action |
| -------: | ----------------------- | ---------------------------------------- |
| < 0.5 | Approximately symmetric | No transformation |
| 0.5 – 1 | Moderately skewed | Optional transformation |
| 1 – 2 | Highly skewed | Apply transformation |
| > 2 | Extremely skewed | Transform and consider outlier treatment |
# Step 2. Measure Outlier Percentage
Next, estimate the percentage of outliers using the **IQR rule**.
```python
outlier_pct = {}
for col in num_cols:
q1 = df[col].quantile(0.25)
q3 = df[col].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
pct = (
((df[col] < lower) | (df[col] > upper))
.mean()
)
outlier_pct[col] = pct
(
pd.Series(outlier_pct)
.sort_values(ascending=False)
)
```
The resulting percentages help determine whether extreme values require additional preprocessing.
# Decision Framework
Based on both skewness and outlier percentage, choose an appropriate preprocessing strategy.
| Data Characteristic | Recommended Treatment |
| ------------------------------------------- | ----------------------------- |
| Mild skew | Leave unchanged |
| Right-skewed | `log1p()` transformation |
| Contains negative values | Yeo-Johnson transformation |
| Heavy-tailed distribution | Winsorize (clip) → `log1p()` |
| Many extreme outliers | Quantile clipping → `log1p()` |
| Approximately symmetric with large outliers | `RobustScaler` |
# Recommended Preprocessing Pipeline
For highly skewed financial variables, the preprocessing workflow is:
```text
Raw Data
↓
Handle Missing Values
↓
Clip Extreme Values (1% / 99%)
↓
Log Transformation (log1p)
↓
StandardScaler
↓
ML train
```
# Transformation Strategy by Variable Type
| Variable Type | Recommended Transformation |
| ---------------------------------------------------------- | --------------------------------------------------------- |
| Normally distributed | `StandardScaler` |
| Right-skewed | `log1p()` → `StandardScaler` |
| Extremely skewed | Clip (1st/99th percentile) → `log1p()` → `StandardScaler` |
| Bounded variables (e.g., ratios, percentages, utilization) | `StandardScaler` only |
# Why This Matters
Some ML algorithms uses **Euclidean distance**, making it highly sensitive to feature scale and extreme values.
Applying appropriate preprocessing provides several benefits:
- Reduces the influence of extreme outliers.
- Makes highly skewed financial variables more symmetric.
- Places features on comparable scales.
- Produces more stable and meaningful customer segments.
- Improves clustering quality and interpretability.