# 1. What Is Logistic Regression Logistic regression models the probability of a binary outcome using the sigmoid function, It is called **logistic regression** because it uses the **logistic function** to convert a linear regression-like score into a probability between 0 and 1: Linear function $y = \beta_1 \times x_1 + \beta_2 \times x_2$, where $y \in [-\infty, \infty]$, but probability is $\in [0,1]$ $ P = \frac{1}{1+e^{-z}} $ which is derived from modeling the **log-odds** of the outcome as a linear combination of the features: $ \log\left(\frac{p(X)}{1-p(X)}\right) = \alpha + \beta_1X_1+\dots+\beta_nX_n+\epsilon $ For any given feature, a coefficient $\beta > 0$ means that feature is associated with a higher probability of the outcome. To make a coefficient interpretable in real-world terms, it's converted into an **odds ratio**: $Odds\ ratio = e^{\beta}$, which tells you that a one-unit increase in that feature multiplies the odds of the outcome by a factor of $e^{\beta}$. # 2. Implementing Logistic Regression From Scratch ```python def sigmoid(z): return 1 / (1 + np.exp(-z)) for i in range(number_of_iterations): # Calculate the linear combination z = np.dot(X, w) + b # Calculate sigmoid predictions = sigmoid(z) # Calculate the error error = y - predictions # Calculate gradients gradients_w = np.dot(X.T, error) / m gradient_b = np.sum(error) / m # Update weights and bias using gradient descent w = w + alpha * gradients_w b = b + alpha * gradient_b def predict(X_new, w, b): # Calculate the linear combination of features and weights (z = X_new * w + b) z = np.dot(X_new, w) + b # Calculate the predicted probabilities using the sigmoid function predictions = sigmoid(z) # Apply threshold for classification (e.g., > 0.5 is class 1, else class 0) return np.where(predictions >= 0.5, 1, 0) ``` Walking through what each piece does: the **linear combination** ($z$) combines the dataset's features (`X`) with the model's weights (`w`) and bias (`b`). The **sigmoid function** then transforms that linear combination into a probability, which is the model's actual prediction. The **error** is simply the gap between the true labels (`y`) and those predicted probabilities, telling the model how far off it currently is. The **gradients** describe the direction and magnitude by which the weights and bias need to shift to reduce that error. Finally, the **update step** adjusts the weights and bias in the direction that reduces the error, scaled by the learning rate (`alpha`). # 3. Assumptions Logistic regression relies on several assumptions worth checking before trusting its output: a genuinely **binary target** variable; **independent observations**; **no severe multicollinearity** among predictors, checked with `from statsmodels.stats.outliers_influence import variance_inflation`; a **linear relationship between numeric features and the log-odds** of the outcome — if this doesn't hold, a log transform, binning, or splines can help, or a tree-based model may be a better fit entirely; **no extreme influential outliers**; and **sufficient sample size and class balance** — if the classes are imbalanced, `class_weight='balanced'` can help; if the overall dataset is small, regularization and cross-validation (`from sklearn.model_selection import cross_val_score`) become important, along with extra caution when interpreting p-values, to guard against both overfitting and underfitting. # 4. Pros and Cons **Pros:** logistic regression is highly interpretable, produces odds ratios that are easy to communicate to non-technical stakeholders, clearly explains each feature's effect on the outcome, and works well even on small datasets. **Cons:** it assumes a linear relationship in log-odds, it cannot automatically learn interactions between features (interaction terms like `C(A)*C(B)` need to be added explicitly), and it's sensitive to multicollinearity and outliers. # 5. Building a Logistic Regression Model Step by Step ## 5.1 Step 1: Exploratory data analysis Start by checking the balance of the target variable. For numeric features, check for group differences, distribution shape, outliers, and whether standardization is needed. For categorical features, check for group differences and decide how to encode them, depending on whether the categories have a natural order (see §7). ## 5.2 Step 2: Train/test split Use `from sklearn.model_selection import train_test_split`, and consider passing `stratify` on the target column, since the outcome is often imbalanced and a plain random split risks producing a skewed train or test set. ## 5.3 Step 3: Fit the model and generate predictions Fit the model on the training data (using `statsmodels` for an inference-focused workflow, or `sklearn` for a prediction-focused one — see §7 for the tradeoffs) and generate predicted probabilities on the held-out test set. ## 5.4 Step 4: Validate the model Evaluate the model using accuracy, precision, recall, F1, and the ROC-AUC score, and visualize both the ROC curve and the precision-recall curve. | ROC-AUC | Meaning | | ------- | ------------------ | | 0.5 | random guessing | | 0.6-0.7 | weak | | 0.7-0.8 | good | | 0.8+ | strong | If the ROC-AUC score is good (0.7+) but accuracy is low, that's a sign the classification *threshold* (0.5 by default) needs adjusting — the model is ranking customers correctly relative to one another, but the default cutoff isn't translating that ranking into good hard classifications. ## 5.5 Step 5: Interpret the results Convert the fitted coefficients into odds ratios (§1) to express each feature's effect in business-interpretable terms. # 6. Alternative Models for Binary Outcomes Several other model families can also predict a binary outcome, each with different tradeoffs: **decision trees and random forest** handle nonlinear relationships and feature interactions well with less preprocessing; **gradient boosting and XGBoost** typically deliver higher predictive performance on nonlinear patterns and complex feature interactions; and **naive Bayes**, **KNN**, **SVM**, and **neural networks** round out the broader toolkit, each suited to different data shapes and scales. # 7. Encoding Categorical Variables: Statsmodels vs. Scikit-learn The right approach to encoding a categorical variable depends on the modeling framework and the goal. When using statistical models in `statsmodels` (like logistic or linear regression) for inference — hypothesis testing, p-values, confidence intervals, or odds ratios — it's usually best to keep the original categorical column and specify it as `C(variable)` in the model formula, letting `statsmodels` automatically create dummy variables and select a reference category. When using machine learning libraries like `sklearn`, random forest, XGBoost, or a production pipeline, categorical variables should typically be explicitly encoded via one-hot encoding (`pd.get_dummies()` or `OneHotEncoder`). For regression-based models, use `drop_first=True` to avoid perfect multicollinearity (the "dummy variable trap"); tree-based models generally don't need a dropped category, since they aren't affected by multicollinearity the same way linear models are. # 8. Regularization When working with a small dataset, many features after one-hot encoding, and a logistic regression model showing a weak ROC-AUC score, regularization is often the fix — with a small dataset, coefficients can become unstable, and shrinking them toward zero usually improves generalization. **L2 (Ridge)** keeps all features but shrinks large coefficients, reducing variance. **L1 (Lasso)** acts as automatic feature selection, pushing some coefficients to exactly zero. **Elastic Net** combines both, and tends to work well when many correlated features are present. ```python model = LogisticRegression(max_iter=1000, class_weight='balanced', C=0.1) ``` Here, a smaller `C` means stronger regularization. To find the best value systematically, use a hyperparameter search: ```python from sklearn.model_selection import GridSearchCV param_grid = {'C': [0.001, 0.01, 0.1, 1, 10]} grid = GridSearchCV(model, param_grid) ``` # 9. When All Models Perform Poorly If logistic regression, tree-based models, and boosting all perform badly no matter what's tried, work through this checklist in order: first, check whether there's actually a signal in the data at all, by comparing the two outcome groups directly — a **likelihood ratio test** comparing a null model against the full model can confirm whether the features carry any real predictive information. Second, check whether the sample size is simply too small. Third, check feature quality — are there variables that plausibly drive the outcome (e.g. cancellation) that simply aren't present in the dataset? Fourth, revisit feature engineering — for example, transforming a raw feature like `avg_monthly_usage` into a more informative ratio like `usage_per_dollar`. Fifth, check the label quality itself — is the 0/1 outcome actually being recorded correctly? # 10. Appendix - [Project](https://app.hex.tech/019eb406-8c5e-755b-986f-e0693fc2a83c/hex/Subscription-Renew-Regression-033YKukBJysVGVyfuZQEVc/draft/logic)