(IBM Senior Product Manager Rachel Bland authors today's post.)
After 16 years working in information analytics as an ETL developer, a metadata modeler and a Business Intelligence product manager I’ve seen pretty much every type of bump in the road that can happen on the journey from data to analytics.
Right now I see a lot of challenges arising through the combination of exploding data volume and increasingly ad hoc use of data by seriously smart people we’re labeling as “data scientists”. We've seen this story play out in SaaS whereby users demand easy and immediate access to applications or they quickly go around IT. Thankfully, there’s a BI app for that! The IBM Business Intelligence Pattern with BLU Acceleration.
Late in 2012 we asked ourselves some questions.
“What are the problems we struggle most to address?” “What are the most prevalent problems?” Speed and agility were the answers.
“How fast is fast enough?” and “Who are we trying to delight?” We came to the answers, “The human brain can’t distinguish much below 1 second”, and “Advanced users as well as the administrators who hate them”.
As we thought about our options we wondered, “Would our clients be deploying massive centralized environments if the cost of administration and management were negligible?” They asked for a quick-reaction 'BI sandbox', whereby you build, use, destroy, and repeat.
We believed taking agility to the next level would require us to offer standalone BI that could be provisioned within an hour and ready for use. Data Scientists told us they needed terabytes of data and they didn’t want to be dependent upon DBAs as they would need to augment that big data with other outside sources. They also told us they wouldn’t wait much longer than 3 seconds for a query to return. The Administrators told us that whatever we offered had to allow for strong data governance and easy systems management.
Back in the lab we took a look at all the technology we had to choose from and decided to take a big leap; IBM still does that! We broke with tradition and rather than focus on a centralized, distributed deployment open to every operating system, middleware product, and data source, we instead focused on a more appliance like approach. We chose:
IBM Business Intelligence with Dynamic Cubes to provide the fastest possible performance, on-disk and in-memory for predictable queries.
IBM DB2 10.5 with BLU acceleration as our in-memory columnar repository for un-predictable queries, to accelerate in-database aggregates and to shorten load times for our cubes.
IBM PureApplication System as a flexible, proven platform.
We reduced DBA dependency by automatically deploying and configuring the content store database and integrating IBM Dynamic Cubes dynamic query analyzer with BLU to enable the automat ic creation and loading of aggregates. We even managed to remove a few hundred user clicks by automatically mapping new aggregates into the cube model. What previously took 200 clicks now takes 20. You might call this simple, but we call this 'pattern'.
In the end we created the IBM Business Intelligence Pattern with BLU Acceleration, a system that requires 5-10 minutes of human intervention, less than an hour to upload and deploy. It handles terabytes of data, it provides single digit second responses for high user concurrency with a mix of interactive and predictable workloads. BI now has an ‘easy button’ and you can find it here.