The IDC HPC Forum was held in
Dearborn, Michigan this week. Customers talked about the issues of dealing with
the volume of data generated from sensors, transactions and humans. The current
systems are failing them to deal with this volume of data and new models were
proposed. NASA JPL talked about the OODT framework
the middleware for meta data. Handling data at mind boggling rates of
700Tb/s generated from various sensors, (e.g. reading data from space
telescopes and radars) requires new architectures for handling and
Hardware solutions are being optimized to provide lower
higher memory and disk, increasing compute density and faster bandwidth
solutions. Think of the energy spent by idle disks - 5w per disk with
millions of disks sitting idle in these data centers - we are talking
about mega watts of power.
In my Big Data Panel, multiple
vendors were asked on many aspects related to the topic. In the next few blogs I
will list my viewpoint and summarize some additional insights from the panelists and speakers.
Many of the deployments of big data systems is happening in the enterprise data centers, which are less experienced in the deployment of low cost clustering technologies, especially related to distributed filesystems. A key characteristic of the problem set these solutions look to solve for, is simply storing massive volumes of data at an affordable price. Without that price point this data would be lost as the value to be derived from the data is less compared to the size. As one of speakers put it "some problems are so hard - its like finding a needle in a needle stack". IBM has been delivering hardware and software solutions designed and optimized for these environments, and with its recent Platform Computing acquisition is in the best position to deliver easy to use, rich functionality to drive the adoption of Big Data in the enterprise.