# 1. A Framework for Approaching ML System Design Designing a machine learning system — whether for a real product or for a system design interview — goes much further than picking a model. It requires reasoning through the business goal, the data, the training approach, the serving infrastructure, and the ongoing monitoring loop, in that order. The six-step framework below walks through each of these layers, using a feed-ranking system (deciding what content to show a user) as the running example. # 2. Step 1: Clarify Requirements and Constraints Before touching any part of the design, align on scope, scale, latency, and what success actually means. **Success metrics.** Start by asking what success looks like for the business — click-through rate, revenue, user retention — and whether there's a hard constraint that can't be violated, such as a maximum allowable false-positive rate for a fraud model. It's also worth identifying **guardrail metrics**: metrics that should not get worse even while the primary metric improves (for example, a ranking model that boosts CTR shouldn't be allowed to tank session length). **Scale and latency.** Establish how many requests per second the system needs to serve, and what latency is acceptable for a single prediction. Is this online inference with a strict p99 latency target — for example, under 100ms — or is it a batch job that runs nightly? It helps to actually break a latency budget down by stage: for a 200ms total budget, retrieval might return 500 candidates in 50ms, leaving 150ms for ranking. **Data availability.** Establish what data already exists — historical labels, user interaction logs — versus whether the system is starting cold with no historical signal. **Hosting and compute.** Establish whether the system will run on a cloud platform or an offline/on-prem system, since this affects which serving and infrastructure choices are realistic. A good way to close this step out is with a one-sentence summary that restates the scope back for confirmation — for example: *"So we're building a real-time ranking system, 10K RPS, optimizing for CTR, with historical click logs available going back 18 months. Does that sound right?"* A few terms worth knowing for this step: **DAU** (daily active users), **P99 latency** (the 99th percentile of response time — 99% of requests complete faster than this), and **SLA** (service level agreement — the formal name for this bundle of latency, availability, and performance requirements). # 3. Step 2: Formally Define the ML Problem Once the business requirements are clear, translate them into a concrete ML task with a defined label, loss, and offline evaluation metric. **Task type and ML category.** Decide what kind of ML problem this actually is: binary classification, multi-class classification, ranking, regression, or sequence generation. **Label definition.** Be precise about what the training signal actually represents. "User clicked" is not the same signal as "user clicked and spent more than 30 seconds on the page" — the two labels will train very different models. It's also common to construct a weighted score that combines several engagement signals into one label, for example combining clicks, likes, and shares into a single composite score, with comments weighted more heavily than likes since they signal stronger user intent. **Offline evaluation metric.** Choose a metric appropriate to the task — NDCG for ranking, AUC-ROC for classification (particularly useful when there's class imbalance), or RMSE for regression — and then explicitly connect that offline metric back to the business metric from Step 1. For example: *"We'll optimize for NDCG@10 offline, which should correlate with CTR in production, and we'll also track AUC on a held-out validation set. The business metrics we ultimately care about are downstream CTR and session length."* # 4. Step 3: Data Pipeline and Feature Engineering This step designs the data sources, the feature store, and the flow that turns raw data into training examples. **Data sources.** Name the raw signals feeding the system: user interaction logs, item metadata, user profiles, and contextual signals like time of day, device, or location. **Feature categories.** Features generally fall into three buckets, and how quickly each one changes has real implications for feature store design: **user features** (like account age or historical CTR) change slowly; **item features** (like category or recency) change moderately; and **interaction features** (like user-item affinity or session context) can change in real time. **The pipeline.** Raw data becomes training examples through a chain of infrastructure: event streaming (commonly Kafka), batch feature computation (commonly Spark), storage in a feature store, and a join step that assembles the final training dataset. It's worth explicitly separating the **offline store** used for training (e.g. a cloud data warehouse like BigQuery) from the **online store** used for serving (e.g. Redis or DynamoDB) — see §12 for a full treatment of feature store architecture. **Serving fast-changing interaction features at inference time.** Interaction features change constantly, so the practical solution is a feature store — something like Feast or a Redis-backed store — where user embeddings and similar features are pre-computed and cached on a daily or hourly schedule. At inference time, the system does a fast point lookup rather than a full recomputation. The main risk here is **training-serving skew**: if the offline pipeline computes a feature differently than the online serving path does, the model will see subtly different inputs in production than it saw during training. The fix is to ensure both paths use the exact same feature-computation library. **The cold start problem.** For new users with no history, a reasonable fallback is popularity-based ranking combined with demographic priors, and onboarding signals (things the user indicates during signup) can be used to bootstrap an initial interest vector. This is worth flagging explicitly as a known limitation of the system rather than glossing over it. # 5. Step 4: Model Selection, Training, and Evaluation Strategy This step chooses the model family, the training approach, and how experiments get tracked. **Lead with constraints, not model names.** Rather than opening with "I'd use a gradient boosted tree," open with the tradeoff that's actually driving the decision — for example: *"Given a 100ms latency budget and dense user-item interaction data, the real question is the tradeoff between a simpler model that's fast to serve and a more expressive model that needs extra optimization to hit that latency target."* **Candidate models.** For a ranking problem specifically, three natural candidates are: **logistic regression** as a fast, interpretable, easy-to-debug baseline; a **gradient boosted decision tree (GBDT)** as the practical workhorse for most production ranking systems; and a **two-tower neural network** (see §10) when the system needs to handle cold-start users/items or scale to billions of items. You don't need to commit to a single model up front — proposing to start with a GBDT while running a two-tower model in parallel as a challenger demonstrates realistic production thinking. **Training strategy.** Three things are worth covering explicitly: retraining cadence (daily batch retraining versus continuous online learning), how class imbalance is handled if it's relevant to the task, and how experiments are tracked. Tools like **MLflow** or **Weights & Biases** are standard here for logging hyperparameter sweeps and offline metrics for every run, so new candidates can be compared against the existing baseline (see §11 for more on experiment tracking and deployment validation). # 6. Step 5: Serving and Inference Architecture This step designs the online/offline inference path, the latency budget, and the deployment strategy. **Online vs. offline inference.** The first decision is whether inference happens online or offline. A feed-ranking system generally needs online inference, since predictions are made per-request in real time. A weekly email recommendation, by contrast, can run as an offline batch job. Some systems need both: a batch job precomputes a candidate set ahead of time, and an online ranker re-scores those candidates at request time — worth stating explicitly if that's the design. **The online inference path.** A typical request flow looks like: client request → API gateway → model server (TFServing, Triton, or a custom FastAPI service) → feature fetch from the online feature store (Redis or similar) → inference → response. It helps to name a latency budget at each hop — for example, feature fetch under 10ms and inference under 50ms, leaving headroom for the rest of the request path. **Deployment strategy.** Two standard techniques are worth naming: **shadow deployment**, which runs the new model in parallel with the current one without affecting real users, letting you compare predictions before going live; and **canary rollout**, which routes a small slice of real traffic (typically 1–5%) to the new model first. You don't need to design both in full detail — simply naming them signals that safe deployment has been considered. **Scaling.** Handling high request volume (e.g. 10K RPS) typically involves horizontal scaling of model servers, GPU batching to improve throughput, and caching predictions for popular items where some staleness is acceptable. **Pre-launch experimentation.** Before a model change is fully launched, an A/B test validates its actual business impact rather than relying solely on offline metrics (see §11). # 7. Step 6: Monitor and Iterate The final step defines drift detection, feedback loops, and retraining triggers for the system once it's live. **Data drift.** This asks whether the input feature distributions are shifting over time. The **Population Stability Index (PSI)** is commonly used for categorical features, while **KL divergence** or the **Kolmogorov-Smirnov (KS) test** are used for continuous ones. A common practice is to set up alerts that fire when PSI exceeds 0.2 on any critical feature. **Model drift.** This asks whether prediction quality is degrading over time. It's tracked by comparing the online business metric (CTR, conversion rate) against an offline proxy metric computed on a held-out, recent slice of data. A growing gap between the two is a signal of either training-serving skew or genuine distribution shift in the underlying data. **Business impact monitoring.** Beyond the model's own metrics, it's important to continue tracking downstream key metrics like CTR and session length, as well as the guardrail metrics defined back in Step 1. **Feedback loops and retraining.** New data needs an explicit path back into training. Common triggers include time-based retraining (for example, retraining every 24 hours on a rolling data window), metric-based retraining (triggering a retrain when CTR drops more than some threshold week-over-week), or continuous online learning when freshness is critical to the product. This connects directly back to the data pipeline from Step 3 — for example, a monitoring system might write retraining triggers back to a pipeline orchestrator, which kicks off a new training run (e.g. via Kubeflow) using the updated data slice. In more severe cases, the system might trigger an emergency retrain and roll back to the previous model checkpoint while the new one trains. **When the business metric drops but AUC stays flat.** This is a classic symptom of **Goodhart's Law** — the model is optimizing the proxy metric it was trained on, but that proxy has stopped tracking the actual business goal. The right response is to investigate whether the engagement labels are still representative, check for label distribution shift, and consider incorporating the business metric more directly into the loss function (as a regularization term) or reweighting the training data to better reflect current user behavior. # 8. Ranking Methods: Pointwise, Pairwise, and Listwise Ranking models can be trained with three fundamentally different objectives, each with its own tradeoff between fidelity to the true ranking problem and practical cost to train and serve. Listwise methods like LambdaRank capture the relative ordering of a full result list better than the alternatives, but they are harder to train and serve at scale. Pointwise methods get a system roughly 80% of the way there and are far simpler to iterate on, which makes them a common starting point before revisiting listwise methods once a solid baseline exists. **Pointwise ranking** evaluates items independently, framing ranking as a standard regression or classification problem: the model predicts a score or class label for each item individually, and the final list is produced by sorting on those scores. This approach is simple and scalable, but it ignores the relative relationships between items and any positional context within the overall list. **Pairwise ranking** evaluates items in pairs: for a given query, the model compares two items and predicts which one should rank higher, with the training objective being to minimize the number of "inversions" — cases where a pair is ordered incorrectly. This captures relative relevance better than pointwise ranking, but it scales computationally as O(n²) in the number of items and still ignores the global context of the full list. Popular algorithms include **RankNet** and **LambdaMART**. **Listwise ranking** evaluates the entire list of items at once, optimizing the ordering of the complete set directly rather than treating items individually or in pairs. It typically optimizes information-retrieval metrics directly, such as Normalized Discounted Cumulative Gain (NDCG, see §9) or Mean Average Precision (MAP). While this is theoretically the most faithful representation of the ranking problem, listwise models are also the most complex and computationally expensive to train and serve. Popular algorithms include **ListNet** and **LambdaRank**. # 9. NDCG@K: Measuring Ranking Quality **Normalized Discounted Cumulative Gain at K (NDCG@K)** is a standard metric for evaluating ranked lists, and it's built up from a few simpler pieces: - **Cumulative Gain (CG)** sums the relevance scores of all items in a ranked list, with no regard for their position. - **Discounted Cumulative Gain (DCG)** improves on CG by adding a logarithmic penalty based on rank position, so that highly relevant items ranked near the top of the list contribute more than equally relevant items buried further down: $DCG = \sum_{i=1}^{n} \frac{rel_i}{\log_2(i + 1)}$ - **Ideal DCG (IDCG)** is the maximum possible DCG achievable for that set of items — the DCG you'd get if the items were sorted in perfect descending order of true relevance. - **Normalization** divides the actual DCG by the IDCG to produce a score between 0.0 and 1.0: $NDCG = \frac{DCG}{IDCG}$ Restricting the calculation to the top K items (NDCG@K) focuses the metric on what a user is actually likely to see — the first page of results, say — rather than the full ranked list, which is also why this metric integrates cleanly with learning-to-rank algorithms that are themselves optimizing for top-of-list quality. # 10. Two-Tower Architecture and Cross-Entropy Loss The **two-tower architecture** is a neural network design commonly used for large-scale retrieval and recommendation, especially when the candidate pool is too large (millions or billions of items) to score every item against every user at request time. It consists of two independent neural networks — a **user tower** that encodes user features (profile, history, context) into a fixed-length embedding vector, and an **item tower** that encodes item features (metadata, content, category) into an embedding vector of the same dimensionality. Because the two towers are independent, item embeddings can be precomputed offline for the entire catalog and indexed in an approximate nearest neighbor (ANN) search structure, while the user embedding is computed just once per request — this is what makes two-tower models fast enough to serve retrieval over enormous item catalogs, and it's also why they handle cold-start items reasonably well, since a new item's embedding can be produced directly from its features rather than requiring interaction history. At training time, the similarity between a user embedding and an item embedding — typically their dot product or cosine similarity — is treated as a relevance score. That score is passed through a loss function that pushes embeddings for genuinely matched user-item pairs closer together while pushing unmatched pairs apart. **Cross-entropy loss** is the standard choice here: it measures the difference between the model's predicted probability distribution and the true label distribution. For a simple click/no-click prediction, this takes the form of **binary cross-entropy**. For retrieval-style training, it's common to use **in-batch negative sampling** — treating the other items in the same training batch as negative examples for a given user — combined with a **categorical (softmax) cross-entropy** loss across the positive item and its sampled negatives. The two-tower architecture typically serves as the **retrieval / candidate-generation stage** of a larger recommendation system, narrowing millions of items down to a few hundred candidates that a more expensive ranking model (like a GBDT, see §5) can then re-score in full detail. # 11. Experiment Tracking and Deployment Validation Getting a new model into production safely relies on both offline experiment tracking and a staged online rollout. **Offline experiment tracking.** Tools like MLflow or Weights & Biases are used to track hyperparameter sweeps and log offline metrics for every training run, which makes it straightforward to compare a new candidate model against the existing baseline before it's ever exposed to real traffic. **A/B testing.** The primary validation mechanism before a full rollout is an A/B test: traffic is split so a control group continues to see the current model while a treatment group sees the new one, and metrics like CTR and session length are measured for statistical significance before committing to a full launch. **Canary rollout.** Once an A/B test clears, a canary rollout gradually increases the new model's traffic share — for example, ramping from 5% to 100% over 24 hours — while monitoring closely for regressions. **Shadow deployment.** As an optional earlier step, shadow deployment runs the new model in parallel with the current one, without using its predictions to serve real users, which provides a zero-risk way to validate the new model's outputs before it ever reaches an A/B test. The tradeoff is that shadow deployment roughly doubles serving cost and, because it never actually serves users, doesn't produce real user response data the way an A/B test does. # 12. Feature Store Architecture A **feature store** is a centralized system for defining, computing, storing, and serving machine learning features consistently across both training and production. Its central purpose is to prevent **training-serving skew** — the problem where the same logical feature ends up being calculated differently during model training than it is during real-time inference, silently degrading model performance in production. ## 12.1 Core architecture A feature store is typically composed of four parts. The **offline store** holds historical feature values used for training, backfilling, and point-in-time-correct joins, commonly implemented with technologies like BigQuery, S3, or Hive. The **online store** holds the latest feature values for low-latency inference, commonly implemented with technologies like Redis, DynamoDB, or Cassandra. The **feature registry** tracks feature definitions, ownership, schemas, versions, lineage, and freshness SLAs, making features discoverable and auditable across teams. The **feature pipelines** compute features from batch data or real-time event streams and materialize the results into both the offline and online stores. Critically, the same transformation logic should write to both stores, so there is a single source of truth for how every feature is computed. ## 12.2 Key capabilities **Point-in-time correctness.** Training data must be built using the feature values that existed *at the moment the prediction event occurred*, not values that were only available later. Getting this wrong is a classic source of data leakage, where a model appears to perform very well offline but fails in production because it was inadvertently trained on future information. **Backfilling.** When a new feature is created, its historical values are recomputed across the full training window so the feature can be used immediately in model training, rather than waiting weeks or months for enough fresh history to accumulate. **Low-latency serving.** At inference time, the model retrieves a precomputed feature vector using an entity key such as `user_id` or `item_id`. Online stores are purpose-built for single-digit-millisecond lookups, which is what makes real-time inference feasible at scale. **Monitoring.** A feature store should continuously monitor freshness (is the feature up to date), missing values, distribution drift, schema changes, and pipeline failures — since a silent failure in any of these can degrade model quality without triggering an obvious error. ## 12.3 Common feature patterns | Pattern | Freshness | Best use cases | | ------------------ | ------------------ | ------------------------------------------------- | | Batch features | Hours to days | Demographics, account age, historical aggregates | | Streaming features | Seconds to minutes | Fraud detection, recommendations, recent activity | | On-demand features | Request time | Distance, query similarity, session context | | Shared features | Depends on source | Reuse across models and teams | ## 12.4 Common mistakes A few mistakes show up repeatedly in feature store implementations: computing a feature differently in training than in production, using a feature's current value instead of its true historical point-in-time value, choosing an online database that can't actually meet the required latency, building a full streaming pipeline for a feature that only changes slowly, neglecting the feature registry so features become undiscoverable across teams, and failing to persist streaming features for later historical training — which quietly removes the ability to retrain on real-time features down the line.