We A/B tested a new recommendation model for 6 weeks.

It showed 3% better engagement. The team was excited.

Then someone checked revenue. The new model was recommending engaging content... that people didn't buy. Revenue dropped 2%.

A/B testing ML models is full of traps that don't exist in traditional software A/B testing.

The model might optimize for the wrong metric. Statistical significance takes longer because ML predictions have higher variance. User behavior changes when they get different recommendations (feedback loops). Novelty effects inflate initial results.

What I do now: define success metrics BEFORE the test (not just the metric the model optimizes). Run tests for at least 4 weeks to wash out novelty effects. Monitor multiple metrics — engagement, revenue, user satisfaction, fairness. Use proper statistical methods (not just "the number is bigger").

And critically: have a rollback plan. If the new model is hurting users, you need to switch back in minutes, not days.

A/B testing is how you bridge the gap between "this model is better on my test set" and "this model is better for the business." Skip it at your own risk.