In a random forest for insurance claims, impurity-based feature importance ranks customer_id_hash (a high-cardinality numeric column) as the #1 feature. Permutation importance on the test set ranks it near zero.
Which importance should be trusted, and why?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Impurity (Gini) importance sums the impurity reduction at every split that uses a feature, and a column with thousands of unique values can always find some split that looks good on training data. Permutation importance shuffles the column on held-out data and measures how much the score drops — an ID that carries no real signal drops the score by nothing. Trust the held-out test.
Impurity importance is computed on training data and is biased toward features with many possible splits. Being built in does not make it right.
A hashed ID cannot generalize to new customers by construction. The forest used it to memorize training rows — classic overfitting, and a leakage warning sign.
Permutation importance and SHAP are sound and widely used. The lesson is to choose the right method, not to abandon the question.