Predictive modeling and its variants are at the core of an increasing number of technical advances that touch us in every aspect of our life. Today, nobody doubts the ability of machines to learn from historical data and predict with far higher accuracy than any human. But real world applications of machine learning are often a far cry from the well understood academic assurances of how these algorithms should behave. In this talk I will share some practical lessons when models had a surprising secret life and did something very different from what I thought I had asked them to do. As the creators of machine learning solutions it is our responsibility to pay attention to the often subtle symptoms and to let our human intuition be the gate keeper deciding when our models are ready to be released 'into the wild'. Claudia Perlich, Chief Scientist at Dstillery, talks about how data scientists need to use a combination of data science and intuition to deliver accurate insights from data sets.