How would you handle class imbalance in a fraud detection model?
In fraud detection, the positive class (actual fraud) is typically well under 1% of all transactions. A model that predicts “not fraud” for everything can score 99%+ accuracy while being completely useless, which is exactly why accuracy is the wrong metric here entirely. Start by switching evaluation to precision, recall, and specifically the precision-recall curve (not ROC-AUC, which can look misleadingly good under heavy imbalance), since what actually matters is how many real fraud cases you catch (recall) versus how many false alarms you generate (precision), and the business tradeoff between those two.
Techniques that actually help
Class weighting (penalizing misclassifying the minority class more heavily in the loss function) is usually the first, cheapest lever, most modern ML libraries support a class_weight parameter directly. Resampling techniques like SMOTE (synthetically generating minority-class examples) or undersampling the majority class can help, but should be applied only to the training set, never to the validation or test set, since evaluating on artificially rebalanced data gives a misleading picture of real-world performance. Threshold tuning, moving the decision boundary away from the default 0.5, is often the highest-leverage, lowest-effort change, since the “right” threshold depends entirely on the cost of a false positive versus a false negative for your specific business.
Best practice
Tie the final threshold decision to actual business cost, not a default. If a false positive means annoying a legitimate customer with an extra verification step, and a false negative means losing money to fraud, quantify both costs and pick the threshold that minimizes expected cost, not the one that maximizes an abstract metric like F1.
Edge case interviewers probe for
Ask what happens if fraud patterns shift over time (adversarial drift, since fraudsters actively adapt to evade detection). A static model trained once will degrade, so production fraud systems need ongoing retraining and monitoring of precision/recall in production, not just at training time.
Common mistake
Applying SMOTE or undersampling to the test set along with the training set, which inflates apparent performance since the test set no longer reflects the true, heavily-imbalanced real-world distribution the model will actually face in production.
What the interviewer is checking
Whether you know accuracy is meaningless under severe imbalance, and whether you tie technical choices (threshold, resampling) back to real business costs rather than optimizing an abstract metric in isolation.
Imagine a factory that makes one broken product for every thousand working ones, and you build a quality inspector who just says “this one’s fine” to literally every single item without looking. That inspector would be right 999 times out of 1000, a 99.9% accuracy score, and yet completely useless, since catching the rare broken ones was the entire point of having an inspector.
Class weighting is like telling the inspector “getting a broken one wrong costs you way more than getting a good one wrong,” so they start paying real attention to the rare cases instead of coasting on being right most of the time by default. Adjusting the “how suspicious does something need to look before I flag it” threshold is like deciding how twitchy the inspector should be: too twitchy and they flag perfectly good products constantly (annoying everyone), not twitchy enough and real defects slip through, and the right balance depends entirely on how expensive each type of mistake actually is to the business.
Why interviewers ask this
Class imbalance is nearly universal in real-world ML (fraud, disease detection, defect detection), and this filters for candidates who know accuracy alone is misleading in these settings.
What a strong answer signals
That you tie the evaluation metric and decision threshold to real business costs, not just apply a textbook technique because it’s known to help imbalance in the abstract.
Common follow-ups
- Why is ROC-AUC misleading under severe imbalance?
- How would you monitor for model drift in production fraud detection?
- What’s the difference between SMOTE and simple oversampling?
Advanced variation
“How would you handle a fraud model where labels themselves are delayed by weeks,” which expects discussion of label latency and how it complicates both training and evaluating a fraud model in near-real-time.
A payments team’s first fraud model reported 99.4% accuracy and looked great on paper, but was catching almost no real fraud, since fraud made up only 0.3% of transactions and the model had essentially learned to predict “not fraud” for nearly everything. Switching evaluation to precision-recall curves revealed the true picture, then applying class weighting plus tuning the decision threshold to the point that balanced the cost of a manual review (a false positive) against the average cost of an undetected fraudulent transaction (a false negative) took real fraud recall from under 20% to over 70%, at an acceptable increase in manual review volume.
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import precision_recall_curve
# class_weight="balanced" penalizes minority-class errors more heavily,
# cheaper and often more robust than resampling the training data.
model = RandomForestClassifier(class_weight="balanced", n_estimators=300)
model.fit(X_train, y_train)
# Get probabilities, not just the default 0.5-threshold prediction.
probs = model.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, probs)
# Pick the threshold that matches actual business cost,
# not the default 0.5 or an abstract F1-maximizing point.
cost_per_false_negative = 500 # average fraud loss
cost_per_false_positive = 15 # manual review cost- 1Accuracy is meaningless under severe class imbalance; use precision-recall curves instead.
- 2Class weighting is usually the cheapest, first lever, most libraries support it directly.
- 3Resampling techniques like SMOTE belong only on the training set, never on validation or test data.
- 4Tune the decision threshold to real business costs, not a default 0.5 or an abstract metric.
- 5Fraud patterns drift over time; production models need ongoing monitoring and retraining.