The worst bug I ever encountered: a model that ran perfectly, passed all tests, and gave…

completely wrong predictions in production.

No error messages. No crashes. Just confident, consistent, wrong outputs.

Took me three days to find it. The preprocessing pipeline applied normalization differently during training and inference. The model had learned to work with one distribution and was receiving another. Classic train-serve skew.

ML bugs are fundamentally different from software bugs. With software, broken code fails visibly. With ML, broken systems run fine — they just produce garbage silently.

My debugging checklist (earned through pain):
Start with the data. Always the data. Check for leakage — is future information sneaking into training? Verify preprocessing is identical in training and inference. Inspect actual predictions, not just aggregate metrics. Use learning curves to diagnose overfitting vs. underfitting. A/B test in production.

Common culprits: future data leaking in, wrong evaluation split, mismatched preprocessing, silent NaN values, corrupted labels.

Debugging is easily 50% of an ML engineer's real job. Courses spend 5% of time on it. That gap is where experience lives.