Big Data and the Goldilocks Principle

I was inspired to write this post (I can hear all of you sighing ‘Yet Another on Big Data’) due to another ‘Big’ reason. I listened to a TED (www.ted.com) talk by David Christian titled ‘The History of the World in 18 minutes’ in which he narrates a complete history of the universe, from the Big Bang to the Internet, in a riveting 18 minutes. This is “Big History”: an enlightening, wide-angle look at complexity, life and humanity, set against our slim share of the cosmic timeline. Check out his website – www.bighistoryproject.com , and I promise you that this ‘Big’ has nothing to do with Big Data, as we know it. But what got me interested in his talk is his reference to the ‘Goldilocks moment’ – a moment so precisely right for certain thresholds to be reached to enable higher forms of complexity (life) in the universe.

That got me thinking – Is Big Data the ‘Goldilocks moment’ for organizations with respect to analytics helping them towards achieving better business outcomes?

I think the answer is ‘Yes’ and this stems from the following hypothesis – An organization can utilize analytics for better business outcomes if:

a)      they have more data points to be analyzed (volume)

b)      have the ability to perform sophisticated analysis on large and diverse datasets (variety)

c)       and can do it at a much faster rate than before (velocity)

In that context, I really liked the picture (given below) from one of the IBM articles, which illustrates how Big Data when synthesized  properly along with standard transactional data can help in better business decision making (in this case, it was Fraud Detection)

Source: IBM – Understanding Big Data by Paul Zikopoulos

On the other hand, the exponential increase in processing power of CPUs, the steep fall in memory prices and high bandwidth availability, have enabled the practical use of Big Data techniques. From the human angle, people are creating digital data, viz. social media chatter, video sharing, blogs, mobility etc. at a rapid pace that organizations (with help of Big Data techniques, of course) can potentially solve the ‘Innovators Dilemma’ by providing new products and services that the consumers did not ask for simply because they couldn’t figure out what they actually want.

All in all, I think we are at a precise moment in history (the Goldilocks moment) where organizations can greatly increase their ability to provide better products & services for their consumers using Big Data techniques.

Source: http://blogs.hexaware.com/business-intelligence/big-data-and-the-goldilocks-principle/