Consultant, IBM Center for Applied Insights
The IBM CAI blog has moved - please bookmark and follow the new blog at http://www.ibmcai.com
The era of “Big Data” presents a variety of challenges and opportunities for marketers. With the increase in volume, velocity, and granularity of data, marketers can become much more precise in how they interact with both the marketplace and individual customers. But the same time, when you’re dealing with large volumes of data, it’s easy to over-fit your models and mistake “noise” for “signal”, to borrow a concept from Nate Silver’s excellent book, The Signal and the Noise.
This is something that we’ve been dealing with internally at IBM for a while now. In response, we’ve developed a framework internally that we think may help others refine their own approach to generating insights from data.
"Architecting" (or collecting and structuring) data is extremely important. The rest of the process depends on getting access to the right data from a variety of sources and if you haven’t done a good job of dealing with data across your enterprise, it’s like trying to run a 100m race with your shoes untied.
A hypothesis-test-refine approach to data analysis is central to the concept of Marketing Science. Developing and testing hypotheses is one of the main ways you limit your exposure to over-fitting data.
- Within a business setting, insights are only valuable in so far as they’re able to inform decision-making and/or influence action. At the end of the day, driving business outcomes is the goal of Marketing Science. Keeping this in mind helps to keep you focused through the first two steps. And it means that once you’ve uncovered a nugget of insight, the real work may just be getting started as you take that insight back to the business.
Marketing Science is a fascinating topic that we’ll be talking about quite a bit more moving forward. We’ve conducted some market research that I think will be very enlightening and have started collecting some use-cases of how we’re applying these principles in a practical sense. In the meantime, if you have any comments or thoughts on developing insights from data, we’d love to hear from you.