There is a great deal of data around. Today we call it big data; organizations are storing more of it each day, and it’s becoming expensive to do so. Often data is stored simply because it is easier to do so than to throw it away. Really this is data hoarding rather than data storage; it is simply hanging on to it just in case it might be useful sometime in the future. Just like all that junk in your attic or garage it is unlikely that you ever will find a use for it, and how ever long you are able to put off the fateful day, the day will come when there is no choice but to sort through it all, extract any value that you can, and send the rest to the recycling depot. Data hoarded is a cost; data stored is a (potential) asset.
Extracting value from big data isn’t easy. Most traditional data analytic methods require structured data, and structuring big data is impossible in terms of time and cost. Big data analysis requires alternative approaches that will work on unstructured, structured and semi-structured data. Such approaches include massive parallel processing, stream computing, distributed file systems, data mining grids, and more. The number of big data solutions is increasing but we are far from being there yet.
So how do we discover whether there is any real value in all of that data? And how do we use that value before it is diminished to nothing by the passage of time?
A number of companies are finding the answers to these questions. In the “2013 Big Data Opportunities Survey” it was reported that 43% of organizations surveyed were implementing some kind of big data initiative, and this was fairly evenly distributed across small, medium and large enterprizes. In terms of business type, 61 percent of retail and service organizations, 58 percent of financial and insurance organizations, 29 percent of manufacturing, and 27 percent of government and educational organizations were utilising big data.
The major applications of big data analytics are customer analysis, historical data analysis, web behaviour monitoring, competitor analysis, content management, analysis of social media, and new product testing.
Organizations that have successfully used big data have reaped rewards from doing so. Some examples of these are:
- General Electric used big data analytics for process automation and performance optimizations with a resultant 2012 revenue increase of $45 billion
- Nike+ permits users to monitor their running and walking and this information has been used by Nike to produce products for customers tailored to their fitness levels
- In the US, Installation of smart meters has resulted in a 5% reduction in household energy consumption
There is potential value in big data, but before it can be realized it is necessary to put in place the necessary infrastructure. Doing so is beyond the capability of many IT departments and organizations are turning to cloud services companies such as Mimecast which provide data management solutions based on cloud archiving and data management. It is wasteful to hoard data that has no value, but it is important that you avoid throwing the baby out with the bathwater.