Ok, “Big Data” is going the way of “Information Superhighway” and “Web 2.0”. How do I know? I know this because I’ve seen a number of posts by people both attacking and defending it as a “thing” that could be considered real. There’s a piece in the NYT attacking it, a bunch of people are now talking about “Data Skepticism.” Don’t mistake the meme for reality. Big Data is just a name invented to refer to nothing, and the marketing department has ruined it.
Here’s some short fictional dialogue that illustrates the way in which Big Data evaporates during the budget process at a large corporation:
CIO: When you say Big Data you mean Hadoop, right? I was looking at the budget and I don’t see Big Data anywhere, but I do see this huge line item for Hadoop. Jack wants a promotion, and he tells me everything you say BTW, he said you called me a “greyhair that doesn’t get Big Data”… I get it: Big Data is Hadoop.
Manager: Big Data isn’t just Hadoop, it’s a whole concept. It’s bigger than just a single technology. Big Data is a new approach to data at scale, and I’m sorry about what Jack told you I thought making fun of the boss was essential mid-level management team building. I apologize, I said that during the offsite meeting I thought he’d keep that to himself.
CIO: Forget about it, Jack’s not getting the promotion because I think he’s an idiot. Back to this Big Data thing… ok, so it’s analytics, but we’ve had analytics from quite some time. Is this just another name for Business Intelligence.
Manager: Dear God. Don’t call it that! You’ll scare away all the kids we just hired. We need them. They are the only ones who understand Map/Reduce. Big Data can also cover analytics, yes, but it is a different thing.
CIO: Wait, so the idea that businesses collect data that needs to be processed into reports that then drive the decision making process, that’s a new thing? I thought we were…
Manager: …It isn’t that it is new it’s different. Big Data is a different way of thinking about old problems in a new light.
CIO: Ok, so we’re talking about a analytics at a different scale, right?
Manager: Right, and different tools. Big data is more than just working with data at scale it is about asking different questions. It is about realizing the the value is the data and making sure you are taking advantage of it.
CIO: Paul, if you keep on talking like that, maybe Jack will get the promotion. You realize that you just said nothing to me. You said it very eloquently, but if I wanted bullshit I’d go read eWeek. Back to the budget… so Big Data is stuff like Hadoop and Map/Reduce?
Manager: No, stop it. That’s too limited. Big Data is transformational, it covers not only offline reporting but any systems that need to operate at internet scale: Google is Big Data, Facebook is Big Data, but the government is also starting to adopt Big Data.
CIO: Internet scale… our technology budget is a lot less than Google or Facebook, and we’re not indexing the web. On another note, this Big Data thing sounds like a religion. Are there churches?
Manager: Why aren’t you taking this seriously? There’s a lot of money being spent on Big Data. I’m not the only one saying this stuff.
CIO: There’s a lot of money being spent on religion as well and there are many people that believe the world sits atop an infinite stack of turtles, that doesn’t mean I should take it seriously. I’m asking because you are proposing that we spend a lot of money on it.
Manager: You know what, Big Data is Big Data you’ll know it when you see it. Haven’t you ever worked with a Data Scientist.
CIO: Hold on, a Data Scientist, you mean a Statistician? Years ago, we hired people that knew SAS – that software has been around since the 1970s. Did you put money in the budget for a Data Scientist? And, if so, do I need to install Bunsen burners and fume hoods in that office?
Manager: No. Maybe you should come with me to this conference. A Data Scientist is someone who knows how to work with Big Data.
CIO: You know what? This is fascinating, maybe we should schedule a meeting to talk more about this, but I’ve got to go talk to our analytics guy about the report he just sent me. We’re having a problem with the budget because we spent all this money on machines to support this Hadoop initiative, and it doesn’t look like we’ve found something to use them for yet.
Manager: Right, I wanted to talk to you about that. Our Hadoop initiative isn’t paying off just yet because we need more data to analyze.
CIO: Thanks Paul. A quick tip from upper-management for future reference. When I end a conversation telling you that “we should schedule a meeting to talk more”, that’s just a creative way of saying, “You are full of it, and I’m walking away now.”