Big Data edit  


Before we get stuck in, a short digression to talk about Big Data.

There are many different definitions of Big Data, but for arguments sake let’s say this refers to the exploitation of data that would have previously been uneconomical due to the volume of data, the structure or format of the data (e.g. unstructured, semi-structured, or complex file formats such as video and audio) and the types of analytics required (e.g. path, graph or time series analysis)

I’ve a couple of comments to make on this topic.

Firstly, in my (humble) option Big Data is a marketing term (supported by new technologies - primarily Hadoop) that’s been exploited to sell these technologies and to give the industry something to talk about. However, I think this has been a broadly positive thing, in that it’s brought data analytics to the mainstream, spurred uptake of new technologies and encouraged companies to invest in analytics and data processing that they may not have done previously. In any case it’s probably just about run its course now (as demonstrated as it’s fall down the far side of the hype curve), and has definitely resulted in the devaluation of other (perhaps more traditional) analytical capabilities which still have a role to play and in many cases deliver capabilities that Big Data technologies can’t yet match.

Secondly, I think there’s been a lot of misinformation about Big Data and Big Data technologies. It’s not a replacement for existing BI/MI and analytical capabilities, and in fact needs to coexist and integrate with these in order to deliver on its promises. It’s not always cheaper or more performant than existing technologies, and won’t always reduce the timescales and costs for analytics or data exploration. And it’s not a new or innovative technology - I know of companies that were analysing multi-petabyte data stores and doing real time analytics over ten years ago, parallel distributed file systems have been around for a lot longer than that, and there are many established technologies that have data processing capabilities that the new technologies are only just starting to catch up to.

In terms of this site the plan is to look at the wider picture and take an holistic view of data transformation and exploitation. I’ve therefore no plans to talk explicitly about Big Data, but in looking at the wider picture we will absolutely cover everything relating to it (the technologies and the new use cases these enable) alongside coverage of other new technologies, more established technologies and capabilities, and the interesting intersection between them all.