By Daniel Steeves, Partner at HBVP Prime Advantage, Director at Beyond Solutions
It is easy to drown in big data articles, let alone the numbers that they quote: companies counting customer transactions in the millions (or more) per day, websites with literally billions of photos and videos, not to mention 3000 tweets per second! We are inundated with broad brush information but, like the data being written about, the relevance to you and your business is not so clear… it is your data, not big data, that matters but even that has limited relevance until you know what you want to do with it.
Big data is coming to a cloud near you
Data is of course aggregating at an amazing rate but this is nothing new… it’s clearly been happening for quite some time. What has changed is that, as that data piles up, awareness of the potential value within it is increasing. Combined with advances in technology and the delivery of that technology, storage and access costs are drastically reduced which creates opportunities to derive information, intelligence and insights from that data (in many cases, an asset you already own!).
The Definition
So, a simple definition with some necessary technical undertones: to simplify the vast detaiIs and myriad references on the subject at Wikipedia, I define the point where data becomes “big data” as when you have too much of it to manage, search, manipulate and process using conventional database systems (and / or it no longer suits your database architecture and capabilities).
The four primary issues that contribute to this definition are volume (too much of it), velocity (it moves, changes or aggregates too quickly for you to fully take advantage of it), variety (structured, unstructured, multimedia, internal and external sources) and veracity (how much can you trust the info you are deriving and how are contradictions managed).
Your Definition
Now even if your data doesn’t “qualify” as big data by the above definition, that in itself doesn’t mean that your data can’t add value (or that you won’t have similar issues as you try to manage, search, manipulate and process it).
So, the first suggestion is to move on and talk instead about your data, big or small… the data which you have or to which you could arrange or purchase access.
The second, even more important suggestion is to get a clear idea about what you’d like to (try to) achieve before you start spending all that time, effort and money: there are clever, structured approaches that you should be taking to determine target outcomes, not to mention which will help define budgets and determine timeframes (otherwise how will you know when you’ve succeeded… or not?).
Looking at data this way is different… the data from a purchase at a till was once a record of the transaction only (date, time, stuff purchased, location, payment method) whereas that same customer data can be aggregated and analysed to either target individual consumer preferences (you buy cat food every week: here is a coupon) or to support multi-level forecasts for sales and purchasing decisions.
IT CSI
In a manner similar to how DNA has allowed forensic examiners to reopen criminal investigations, data can let us look back and better understand, with more information and better context, a more precise answer of just how we arrived “where we are”. And if know what happened when, and how, we can see about learning from it to either prevent it from happening again, or to repeat it (depending of course on what IT was).
So, talking big data, or just plain data, is little different from all of the other tech discussions because data itself is not technology: it is a by-product of technology which in some cases can be used to create value.
The Impact, The Future
One thing is clear: analytical and maths education and experience will keep you employed: Big data has increased the demand of information management specialists: the usual suspects, for example, Software AG, Oracle Corporation, IBM, Microsoft, SAP, HP and EMC have spent more than $15 billion acquiring software firms only specializing in data management and analytics. On its own, this piece of the industry worth more than $100 billion and growing at almost 10 percent a year which is about twice as fast as the software business as a whole.
Get in touch at [email protected] if you’d like a chat on big data, your data and your business (and where we now provide a new Sounding Board service).