A gradient-boosting churn model has ROC-AUC 0.88. Its calibration curve shows that among customers scored 0.80–0.90, only 55% actually churn.
What is the problem, and why does it matter for the retention budget?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
AUC measures ranking (do churners score above loyal customers?), calibration measures whether a score of 0.85 means 85%. Boosted trees and random forests often rank well but push scores toward the extremes. If you plan a budget assuming 85% of that bucket will churn and only 55% do, every expected-value calculation is off. Fix with CalibratedClassifierCV (isotonic or Platt) on a held-out fold.
Multiplying every score by 0.5 leaves the ranking — and the AUC — identical while destroying calibration. They are independent properties.
Useful for ranking, yes. But the number 0.85 is what enters the threshold formula and the budget — and it is wrong by 30 points.
Scores are higher than reality; that is over-confidence. Under-confidence would be scores of 0.3 for customers who churn 55% of the time.