Big Data requires big storage, big skills

22 Nov 2011

Our expert panel met at IP EXPO 2011 to discuss what’s needed for companies of all sizes to reap the considerable competitive benefits promised by Big Data approaches.

What does the term ‘Big Data’ mean to you? That was the opening question posed to a panel of industry experts in a seminar session held at IP EXPO 2011, Big Data: ‘Scale out’ challenge or ‘massively parallel’ opportunity?

Most panelists agreed that, as a term, Big Data is in many ways too broad to be truly meaningful. “If you ask five people what Big Data is, you’ll get seven definitions,” joked Rob Anderson, chief technology officer in EMEA for Isilon Systems (part of information management company EMC).

Generally, however, he favours the Wikipedia definition: “Datasets that grow so large that they become awkward to work with using on-hand database management tools”.

Independent industry analyst Claus Egge said he would prefer a different term altogether. Perhaps “savage data”, he suggested, would alert end-user companies to the sheer scale of the challenge at hand.

From the start of the debate to its end, however, what WAS clear is that Big Data isn’t only about analytical tools like Hadoop, Cassandra and MongoDB, although these are an important part of the equation, as panelist Alex Housley of Rummble Labs explained.

It’s also about the storage capacity required to accommodate data characterised by its volume, variety, variability and the velocity at which it arrives – and here, companies are increasingly looking to cope with petabytes, said Gurdip Kalley, business development manager at Solid State Solutions (S3).

Bringing analytical and storage management skills together – or to be more precise, even sourcing them in the first place – may be one of the biggest challenge that companies will face. “There are not a lot of data architects out there who fully understand Big Data – and they’re about to get scarcer and much more expensive,” warned Rob Anderson of Isilon. “And people who DO know about the analytics side of Big Data don’t know anything about buying storage, about the need for sequential performance, about how to cope with gigabytes per second of throughput.”

But companies with the skills and technology resources to do Big Data well, he continued, “will have a whopping advantage over their competitors in years to come.”

That was a point underscored by Michael Natusch of Cumulus Analytics, who described network optimisation work conducted by his company on behalf of a leading UK mobile phone network operator with 2 million customers. The network optimisation data, he said, enables the client to get better visibility into “who calls who, who text who, from what locations and at what times of day.” That’s certainly useful, he continued, “but when you use a Big Data approach to combine that information with other information about individual customers – their average spend, how long they’ve been with the provider, how long their contract has to run and so on – then things get really interesting.”

In conclusion, it was agreed that the biggest risk that Big Data poses may be to the companies that don't have the skills and technologies in-house to exploit it – but there is a growing consensus that, in such cases, third-party cloud solutions may fill the gaps.

If these assertions prove correct, smaller companies won’t need an on-team Hadoop genius or an on-site massively scalable storage architecture to tap into Big Data – they’ll just need a cloud provider willing to supply the skills and infrastructure resources on a pay-as-you-go basis. This is a subject we’ll be returning to in a forthcoming edition of the IP EXPO Online bulletin.

View the presentation from IP EXPO here