Big Data: Big Hype or Big Value?

By: Dennis Draeger

The amount of data organizations are expected to manage for planning, transparency, compliance, etc. is expanding, but the amount of data which could benefit these organizations if analysed effectively is growing exponentially with the aid of social media, RFIDs, machine translators, and other tools. The total amount of digital data is growing exponentially leading to the coining of the term big data which has become a major buzzword in the enterprise and even in the general press, but what is the real value behind the hype?

What is changing?

According to International Data Corporation (IDC)’s fifth annual survey in 2011, 1.8 zettabytes (1.8 trillion gigabytes) of information were created and replicated in 2011 alone which is up from 800 gigabytes in 2009, and the number is expected to more than double every two years surpassing 35 zettabytes by 2020. However, this data includes everything from covert government strategies and medical records to holiday photos and spam. Not all of this data is inherently useful, but all of it can be collected, managed, and analyzed to reveal patterns and trends which could improve decision making at every level for security, system optimization, market research, scientific discovery, etc.

Data analytics has been around for decades, but the current tools are too limited for the massive, fast moving, and disparate information that is increasingly required for analysis, hence the need for big data solutions. And those solutions present new possibilities in data analytics by allowing for greater complexity of data to discover events and populations that may only be seen with the right volume, velocity, and variety of data. With greater segmentation of datasets, big data solutions also allow greater tailoring for an organization’s clients, customers, employees, etc. The biggest challenges are in asking the right questions and ensuring that as much data as possible is included.

Some organizations are already addressing these challenges for the health and energy sectors as well as marketing and cybersecurity. EcoFactor mines thousands of data points about weather, regional building codes, home value, and others to reduce energy consumption in smart homes by 17%. WellPoint is using IBM’s Watson-as-a-Service, the cloud application of IBM’s Jeopardy winning artificial intelligence, to mine millions of pages of medical data to help doctors and nurses improve their decision making while face-to-face with their patients. American retailer, Target, is following the purchasing habits and other information about its customers and coupling that data with behavioural research to improve its target marketing. Zions Bancorporation is using big data solutions to identify phishing attacks, prevent fraud, and stop hackers. The city of Santa Cruz is leveraging data to forecast high crime areas that can be patrolled to reduce offenses, and the US government is also applying big data to homeland security as they search for signals and online activity that could indicate real world security risks.

Why is this important?

…parts of an organization’s data may eventually become more valuable to them released to the wild…

Big data is certainly hyped, but the potential is equally big. Epidemics might be spotted sooner, security threats may be detected earlier, and customer demographics could be made more specific. Regardless of the many opinions about big data and its hype, the data itself will continue to expand exponentially, and more organizations will find themselves hitting the ceiling of previous data management capabilities.

Data has become a commodity, and as it becomes even more valuable, some organizations might loosen their hold on ever more types of data for the sake of mutual benefit. While organizations will still have to maintain compliance for the sake of privacy, intellectual property, and other security concerns, parts of an organization’s data may eventually become more valuable to them released to the wild than securely isolated in their own storehouse. Imagine mining the data of several organizations across multiple sectors added to all the data freely available on the internet, and the potential of big data for organizations of any size can be better understood.

By Dennis Draeger