Big Data's True Value
I’m a contributing member of the U.S. Federal Big Data Commission sponsored facilitated by Tech America Foundation. Steve Mills, Senior VP and Group Executive Software & Systems, is one of the primary Commissioners. The purpose of the commission is to provide federal government agencies guidance on approaches for reaping the benefits from the explosion of information that’s resulted from conducting business on the internet and the rise of social media. Much of the discussion within the commission has centered around the value derived by implementing a big data strategy and why is it somehow different than past approaches for managing data. But that’s exactly the point – Big Data is not about managing the data or information so much as it is about how to make all this information useful and ultimately actionable. In other words, what insight can be gleaned from massive amounts information that could not already be derived by using existing systems or methods. Another element of the Big Data paradigm is the need to derive insight from information that may have a short shelf-life and changes in real-time. Information of this type often comes from automated systems that generate data continuously according to the dynamics of the system – this is often referred to as the velocity of data. Of course it’s necessary to establish the infrastructure needed to ingest such large data sets and store the information once it’s obtained. But the fact of the matter is that building out the infrastructure to manage massive amounts of storage or processing real time data is not the end game. The real purpose and value behind the Big Data play is making all this information actionable by gaining deeper insight into what are often the hidden patterns of our business or social interactions. So from my perspective, establishing the infrastructure to manage all this information may be necessary but it’s not where the real value is derived. The more impactful and challenging aspect of Big Data is extracting the value of that information through the application of data science and the analytical tools that uncover normally hidden patterns that offer new business options and drive better decision support. At the end of the day it’s all about gaining deep insight through the application of analytics and how we apply this new understanding to automate our business decisions.