A Web Developer from 2001 Wouldn’t Even Recognize this World

I work with people much younger that I, but the reality I’m discussing in this article is really just 12 years ago. It feels like another era entirely. This is especially true if you develop anything that touches the web.

When I started my career it was all about web applications that involved full round-trips to a server. You had a browser (or a WAP phone), your browser makes a request for a web page, waits a few seconds (a few seconds!) and you get a fully assembled HTML page in return. It didn’t matter because the Web was still so full of novelty we were just happy enough to be able to do things like read the news online. Maybe your local newspaper had a website, most likely they didn’t. There was no YouTube. Web pages weren’t really connected together in the way they are now. Back then it wasn’t like loading TheStreet.com required a bunch of asynchronous calls out to social networks to populate Like buttons – there was no social network. It was just HTML and Images, and it took forever. It was fine.

My first two jobs were developing an in-house cross promotional tool for an online gaming company named Kesmai in Charlottesville in 1997, and then I moved to New York to work for TheStreet.com in 1999. Web “applications” at that time were just an inch beyond putting some scripts in cgi-bin. At Kesmai it was Perl-based CGI scripts. Between Kesmai and TheStreet I was working on systems that used a proprietary Netscape server product. And, at TheStreet.com we were using Dynamo behind Apache, so we had JHTML and Droplets and that was my first encounter with a site that had to scale. We had a TV show on Fox and maybe something like 600-700 people could use the site at the same time. (Again, that was huge back then, how times have changed?) Everything was template-based, servlets were around, maybe, but I don’t really remember diving into the Servlet API and JSPs until Struts came along maybe in 2001.

Back then, companies like Forbes.com, which I moved to after TheStreet.com, invested a crazy amount of money in hardware infrastructure. There was still a lot of proprietary software involved in the core of a web site – expensive CMS systems, etc. Open source was around, yes – we ran Apache, but it isn’t like it is now. You likely paid a hefty sum of money for a large portion of your production stack. Around 2001 and 2002, a small group of people were starting to focus on speed, and the way you achieved speed at scale back then? Drop a few million on a couple of big Sun servers. It worked. It seems old-fashioned now, but as a developer I’d work with the operations team (then as now, the operations team didn’t know much about Java), and you’d help them size heaps and figure out how to make the best use of 64 GB of RAM on a E450. You’d run multiple JVMs on the thing, someone might install something like Squid to cache things at the web layer.

Back then, you could touch the servers. They were shipped to your offices. Companies like Sun and SGI invested a lot of money to make servers look all fancy. These things were purple they had blue LEDs (remember high-brightness blue LEDs were, at one point, really new to us all). I remember seeing advertisements for servers in programming magazines. Now if you look back at these, it’s as strange as seeing an advertisement for a station wagon in a National Geographic from 1985. These days, I don’t even know who makes the hardware that runs the sites I work on, and with the exception of the database server, I don’t even care if it is even a physical machine. Back then it was like everybody getting all excited about the E4500 that was in the foosball room.

There was no memcache, there was no AJAX, there was no AngularJS, there was no REST, SOAP was new and you probably didn’t use it yet, there was no Google Analytics, remember, Google was still a tiny startup at the time. I remember having a discussion about Ruby in 2001 with a colleague who was excited by it, but Rails didn’t exist yet. Perl and PHP were around, they’ve been around forever, but you really weren’t confronted with systems that spanned several languages. Javascript was around, but you probably weren’t going to dare use it because it wasn’t like there were any standards between browsers. HTML5, huh? This was back when people still used the blink tag. Need to crunch a huge amount of data: well first of all, huge is relative and you didn’t have things like Hadoop. Just didn’t exist yet. Big Data? Yeah, back then if you had 5-10 GBs you were running a huge database. Huge. XML was still a really good idea. Flash was about to really take off.

If we could travel back in time and snatch a web developer from 2001 and drop them into 2013, they’d flip out. They’d look at your production network and wonder what happened. We’d have to tell them things like, “you’ve missed a bunch of things, this kid at Harvard created a system to keep track of friends in PHP and that changed everything. Google now runs the world. Also, the site ‘Dancing Hamster’ isn’t what it used to be.”

I look at people that started working in 2007 or 2008 and I think about how strange it is that none of this is new – because I’m still living in 1993. I’m still amazed at the functionality of Gopher in 1993.

And you can thank Mark Smith for this YouTube video…

The 7 Rules of Software Development

Nuremberg_chronicles_f_180v_1I hate simplistic blogs like “The 10 Rules of XYZ”, but who am I to buck the trend?  Why the picture?  Every time I read one of these simple lists of edicts it strikes as being very “papal”.  Here I’ve simmered down development into 7 rules:

1. There is No Such Thing as Architecture, and you should avoid “Architects”

Good software developers work in small teams, they discuss everything, and the really good ones understand that “architecture” doesn’t really mean much.  Yes there are choices about what software “stack” you are going to use and how components of the system interact, but it is a fools errand to view “architecture” as something separate from development.  Software is built up piece-by-piece, and you make a thousand little decisions here and there, all of which end up in a final structure.  Developers can discuss “an architecture”, but it is just an abstraction, it doesn’t really “exist” outside of the system as it has been implemented, and it is more ceremony than anything else.

The problem with the whole idea is that it suggests an analogy to construction, you know “real” architecture. We’re not in the business of standing up houses and/or buildings, but, invariably, every time someone tries to create an architecture they end up drawing models that almost never line up with reality. Let’s retire the word “architect” as it applies to software development I’m not saying that there can’t be a plan, or a loose collection of rules, but it is the really bad developers that get stuck on this idea that there is “an architect” who is telling “developers” what to do.  (That’s the worst kind of place to work.)

That’s not how it works.  You can make it work like this, but, trust me, your software will suck as a result and you won’t be able to grow a committed team.  Why? Because of the next rule…

2. Everybody Grows

Why is there no architect?  A couple reasons. First, every real software architect I’ve come into contact with is a train wreck. They used to be a developer and they were either so incompetent they had to be promoted out development or they were the kind of individual that thought so highly of themselves they fell naturally into the role of dictating design decisions. Second, having someone be “the architect” gives everyone else a good excuse – something to blame when a system falls on its face. When you have an “architect” you’ll usually see disengaged teams, teams that don’t want to stay late because they are excited about the work they are doing, and a corporate culture ruled by political dysfunction.  Think of a room full of disaffected developers saying things like “Well, it wasn’t my decision to do it that way?” and “Well, I don’t know why we did that, go ask the architect.”

The working alternative is that everyone on the team makes a contribution, and everyone on the team is responsible for at least some portion of the architecture (which, remember, is a useless abstraction). The rule is that decisions are made by the people doing the work, and you should rarely override these decisions. This is essential for long-term success. If you force a strict hierarchy on to your development team, you’ll never give anyone the opportunity to grow. What you need is a self-aligning hierarchy (or as close to self-aligning as possible… people still need bosses, don’t get me wrong.)

3. Avoid Developers who think there is an Objective “Right”, There is only “Satisfies Current Requirement”

This is not to say that I believe in a sort of software relativity in which no solution is better than another.  I do believe that there are good ideas and bad ideas, but if anyone on your team decides that they are the arbiter of Right and Wrong you need to have a long conversation about how a team works. You need to remind them that there is no “architect” and that there are really just people responsible for different components of the system. Now, a little disagree is alright, but in my travels throughout developerland I see a lot of developers running around with this idea that they’ve discovered the One True Way. They haven’t, and there isn’t one.

(Note: there’s an exception to this rule: Subject matter experts – real subject matter experts.  For example, if you are in a meeting with someone who, say, wrote a book about Tomcat, and you say something about Tomcat that causes that person to object.  Listen.)

If disputes seem unresolvable, you should absolutely intervene and nudge a decision along.  This is what management is for, but if you have people on your team constantly debating software development dogma, you should stop them.  Yes, there’s a right way to do File I/O.  Yes, there’s a right way to use a UITableViewController in iOS, and, yes, there’s a correct way to configure Tomcat. No, there isn’t one right way to implement an API, there are several hundred, and you have to give team members enough room to screw up.

4. One Metric for Success == The Number of Times you’ve screwed up

If you are writing software worth writing then you are likely doing something that’s never been done before.  (If this isn’t the case, you are likely writing software that shouldn’t be written but that’s a whole different post.)   If you are forging new ground, there’s a good chance that you are going to screw up… frequently. Good software development teams talk about failures in a way that doesn’t ostracize the people responsible. You want a team that can joke about how they made a bad decision.  You want a team that isn’t constantly trying to look invincible to management.  Everybody screws up.

I’m not telling you to reward your worst programmer a special nerf gun because he’s constantly screwing up production, you shouldn’t “celebrate” screwing up, but you should set the tone that mistakes are valuable. Read some Petroski, he’s written good books on engineering failure (if I weren’t such a lazy bastard I’d link to them).   Long story short: if you don’t want to read his books is: failure is engineering progress.

5. “Is it done?” is to be met with Laughter.  There is no “Done” there is only “Satisfies Current Requirement”

I’ve worked on software long enough to realize that there is no “done”. Stop saying that word, it’s crazy. No matter how much you may want to forget about that code you wrote seven years ago, there’s no escaping it. Software is never done, and it will always break. There will always be something left to do.  If your management team is saying things like, “I thought we were done with that?” or “We don’t have to worry about that system it was done years ago?”  It is your job to remind them that software development is an endless pit of risk – an inescapable gravity well of budget allocation.

The two acceptable responses to “Is it done yet?” are “Software is never done” or “Welcome to the Jungle, we’ve got fun and games.”

6. Don’t Believe Anything You Read Online About Development (Avoid Dogmatic Process)

This is the most important rule of all.  Don’t find some easy software development dogma or list of rules and follow them… that includes this blog post. You should be very suspicious of any online manifesto that talks about ideal software development practices or anyone who takes time out of his weekend to write a blog post entitled “The 6 Rules of Software Development”. Please think for yourself.

One of the worst development teams I ever worked on was a team that was heavily influenced by a popular agile consulting firm that will remain nameless. First of all, the software being produced was awful, but I’m not sure that had much to do with the team as much as it had to do with the company that ran the project. There were architects, there was a guy running around telling everyone he knew what was right and wrong, the place adhered to dogma so much so that people quit over it. Including me.

If someone comes to you totally energized about some “Agile manifesto” they just read, or if they start preaching TDD as the One True Way.  Take them aside, let the know that the workplace is no place for religious proselytizing. While there are many good ideas in the software process community there are also many ideas designed to sell books and consulting services.  Every project and every team needs a slightly different process.  There is no cut and dried model for software development because there is endless variation in humanity.

7. When Everything in this Post is a Lie, It’s Time to Move On

Eventually, as the project matures you’ll experience a slow shift to maintenance mode.  “Management” will start focusing on “risk” and asking questions like, “Who is the architect of this system?” or “What is the Total Cost of Ownership for ActiveMQ?”  At this point, every single rule in this list will experience a rapid inversion point.  There will be an architect, there will be strict definitions of senior and junior developers, someone will start telling everyone else they know the One True Way, programmers will be fired for making even slightly wrong design decisions, you’ll start having status meetings that talk about “finishing” the project, and someone will show up with a bunch of cultish books on process and convince management send everyone to an Agile re-education retreat.

Let’s Start Lifting Each Other Up (…no, Seriously)

My constant affliction is my Inbox, several constantly-conflicting Google Calendars,  Skype, a cell phone that’s constantly beeping with Twitter, Facebook, and LinkedIn activity… and a lot of this activity is negative.  People whining about technology, society, government, people complaining about everything.  The internet is a 2/47 snark carnival and Twitter is the main stage. (I’m not innocent of this by any means, my popular posts are me calling Ruby on Rails names.)

In this overwhelming storm of negativity I find myself asking, “Where’s my list? When do I get to keep track of the tasks I promised myself I’d complete?”  For this, there is Lift.   Go check it out: http://lift.do – if you don’t like web browsers you can search for the Lift application on your phone, it’s there.

Here’s my Lift at the moment.  It’s simple, that’s what I like about it.  There’s not a lot of bells and whistles being thrown around, the app isn’t constantly trying to give me trophies or convince me to attend a webinar.  There’s no upsell, and I’m not getting overwhelmed by automated emails telling me to pay attention to it.  It’s simple, and it’s focused on the positive.  I’m trying to keep my list manageable at the moment: take some walks, learn Spanish, talk less, listen more, meditate, and write more. Clearly, I haven’t been using it enough, I pay for a gym membership, but I average two gym visits a year (that’s $550 per 20 minute recumbent bike session).

lift-dashboard

What’s great about Lift is that it is based on this idea that the best motivators are each other.  Some people use the platform to quit smoking, others use it as a way to set some personal goals.  It sounds a bit corny to the uninitiated, but you set some positive goals, you record that you’ve met them and random strangers fall out of nowhere to give you “props”.

Right, so in the last five minutes two people just decided to like the fact that I learned a bit more Spanish today. Doesn’t that sound incredibly silly? But, it isn’t. I’m motivated to keep on learning Spanish by these random acts of “drive-by” support.  Who could dislike a site that has this Aristotle quote on it?

Happiness does not consist in pastimes and amusements but in virtuous activities.

Lift was started by Tony Stubblebine some time ago.  Tony Stubblebine was the one who helped build that ancient O’Reilly social network that never panned out, and he was also a very close witness to the founding of Twitter at Odeo (maybe even a participant, at this point he probably doesn’t want to be asked about Twitter very much – I’m just guessing.)  Anyway, he went on to start a conference-focused social networking startup, and he’s the kind of guy who, if you’ve met him (even if only for a moment), you want him to succeed.

So, do it.   Let’s all Tweet a little less; let’s all focus on pastimes and amusements a bit less and let’s focus on “virtuous activity”.  If you were about to go think of the next super snarky Tweet that would gain you another follower: stop what you are doing, and go sign-up for Lift.  I dare you, and I promise to give you mad props (I’m too old to say that in person BTW):

http://lift.do

Getting Off the Analytics Treadmill

Years ago, I had the idea that I should put Google Analytics on my own web site.  You know, why not track the readership, find out why people show up, track top refers, maybe even define a couple of conversion goals.   At the time, maybe it made more sense than it does today.  I had this open source book that I had made available, it got a lot of traffic, and I was thinking about trying to convert readers into newsletter signups. Whatever. My plans were nebulous, and, predictably, those plans were put aside for paying client work, a couple of kids, and life in general.

These days, the idea of tracking my organic vs. direct vs. refer traffic and locating the top metropolitan areas of my blog’s audience just seems silly so I got rid of it.  I turned off analytics, and now I’m realizing that there is one less dial to check.  One less meaningless number to pay attention to.  One less game to play every day when I’m looking for ways to waste time.  Here’s what freedom looks like.

freedom-from-analytics

After a week of this, I’m finding it easier to write.  I’m not tempted to go and dive into these meaningless traffic patterns.  Like some distracted data scientist, asking myself: “What is it about Parisians that attracts them to my complaints about Maven?”   No, I’ll write what I write, and if that garners an audience, great.  If it doesn’t, great. In some ways, who cares? This new idea is writing for writing’s sake – I’m not selling advertising, I’m not paying myself to write this blog – but, then it occurs to me…

…why not do the same for the businesses I help with blogging?  Why not turn off analytics for a month (or maybe two)?  Take a radical approach of just doing interesting things.  Produce content and don’t focus on bounce rate or returning vs. new browsers, just do it.  If someone asks about lead generation form conversions, laugh at them and say, “not my job.” Here’s the thing, I’ve worked for companies that have had amazing growth in traffic. (That Maven book had millions of unique viewers.)  I’ve been responsible for that growth over 24-36 months, and it didn’t correlate to us doing interesting things or even generating revenue. You can make traffic go up and you can make people like you and get high off of your Analytics graphs, but it’s really so worthless.  And, there’s so many graphs to look at you will always find one that is going up.  I’m starting to wonder if analytics is just a silly distraction.

What I’m wondering after this personal experiment of turning off statistics is if: analytics, marketing automation, conversion tracking, ad words… all of this is just detracting from what should be the Prime Directive for a technology startup (or any startup).  Connect with some customers, do what they want you to do efficiently, and iterate on what works.  The only statistic that really matters is revenue, so what I’m contemplating is just turning off (or maybe more accurately not paying attention to) analytics not only for a personal blog but for a business blog as well.

At the end of that day, if you need some silly HTML counter to tell you if your ideas are working, you are not going to succeed. If you are deciding what to do based on a focus group or a poll, you should quit now.

(Some disclaimers.  There’s still a bit of statistics gathering on WordPress.  Wordpress tells me how many readers I get, but it isn’t something I give more than a cursory glance.   (In fact, I wonder if there’s an option to turn it off.  I’m tempted.))

How Java Programmers “Feel” in 2013

My summary of general Java sentiment after attending JavaOne 2013.

“Everyone’s all excited that Java didn’t die. Yay! We made it!”

Ok, that’s not fair, how about:

“Everyone’s excited that Java has new found energy and that Twitter ultimately had to migrate everything to Java. I mean Twitter is using Java! The fact that twenty-somethings are using the platform makes us feel a lot less old. Thanks.”

Ok, let’s try again…

“Everyone is excited that the Ruby on Rails kids are playing defense, that Oracle is paying a lot of attention to Java, and Java EE officially doesn’t suck anymore. Let’s go.”

I’m going to go with that last option.  While it is true that Oracle’s taking the platform in an incrementally more commercial direction (you can’t get 1.6.0_51 without a subscription and there are some new tools only available to paying subscribers), the platform does appear to be healthier than ever.   My own theory is that Oracle is doing a much better job enabling people like Rheinhold and Gupta to innovate than Sun Microsystems ever could have.   There was a lot of pain between the Setting Sun and this New Java Renaissance, but we’re here.

After several industry luminaries predicted the death of Java, we’re still here and not only that, we’re still innovating.  We’re beyond that difficult period of ex-Sun employees griping about Oracle’s takeover and we’re now moving on to things like Java 8.  I wouldn’t have said this four years ago, but I’m generally optimistic about Java as a language and a platform.

My First 7th, 8th, and 9th Tweets Ever – Still Somewhat Accurate

Now that Twitter is going public, I was wondering what my first few tweets were in April 2007.  Some of them were funny.  Like my 6th tweets is: “Abandoning Continuum – Maven Blows” and my 14th Tweet which was: “OMG, like, using Twitter I can keep up with all my coolest friendz.   This is so totally awesome!  Twitter you are my hero”

But, my 7th through 9th tweets 

2007-04-23 twitter is for jerks
2007-04-23 twitter is for the hundred or so people who revolve around the SFBay area and give a crap
2007-04-23 oh twitter, can’t get enough of twitter, blahblahblah – twitter is a web20 dream….

With all the publicity that Twitter is getting this week I’m coming back to the same sentiment.  Twitter, while useful and interesting, has overplayed its hand of late.  It’s a great marketing tool, don’t get me wrong, but it’s also an echo-chamber of super Narcissists (myself included, I mean I do have a blog, don’t I?) 

“I’m not the Internet’s Cafeteria”

Was eating at a local restaurant a month ago. (A good one, but I won’t mention which one so that the owner won’t get in trouble with OpenTable.) We sat at the bar because I had a hard time getting a reservation, and since we’ve “sort of” known the owner for a decade we started talking about “whatever”. Conversation eventually ended up on, “So, Tim, what do you ‘do’?”  I do a bunch of different things, but I usually say, “Internet.”   This answer is obtuse, but it conveys two things: I do technology and also I don’t really want to go into the details. I want to drink wine, eat good food, and not have to talk about technology.  That didn’t work because his question reminded me that I did have something to ask him…

“Why did you take this place off of OpenTable? It was convenient.”

He sort of let out one of those “pull up a chair because I have a lot of things to say about how that whole experience sucked” sort of sighs.  Here were the reasons he gave me:

  • It wasn’t right for his business and he lost some of his ability to control reservations.
  • It turned that whole reservation process into a contest to have tables free at just the right time.  So he had to “play games.”
  • He prefers that people call, because he and his partners can offer alternatives on the phone.  “People that would have otherwise been fine waiting 30 minutes end up being pushed to the competition.  That’s not good. This isn’t Expedia.”
  • There was a lot of ridiculous technical overhead, like some custom hardware that had to be involved.

From what I could gather it sounds like the restaurant business is about developing personal relationships with dedicated clientele and that moving to an automated approach was causing more problems than it was solving.  He then showed me his computer and it was a blank notebook with pencil marks in it.  He said it’s been working well for years and it didn’t cost very much. “Upgrades are free.”

At the end of it he summed up his experience: “I’m not the Internet’s cafeteria.”

Screenflow: Unpredictability of Bad UI and a Wasted Afternoon

frustrationScreenflow is one of those essential tools for people like me who fancy Macs and need to create screencasts.     Matthew McCullough introduced me to this tool years ago, and I haven’t looked back since recording the equivalent of many man-months of video for various clients.   It works well, when it works that it, but it also suffers from some instabilities.   Previous versions used to just decide to randomly exit and crash, and I’ve had my fair share of “Unable to open video” errors.

If you do any video work at all, and if you record your own soundtracks, you’ll understand how long it takes to record a good video and to make sure that everything is aligned to make that video work.  When you do screencasts and voiceovers you run through a checklist:

  • Is my voice rested enough to do a good voice track?
  • Have I had too much caffeine and am I going to sound like Jim Cramer?
  • Are there people in the office who are going to look in your office window and see just how strange you look when you are doing a screencast?  (I move quite a bit, it’s a strange process.)
  • Is your crazy expensive microphone plugged into the Tube Amp and the iMic just right?
  • are the levels set correctly?  Also, what does that even mean? you are not an audio engineer.
  • Have you unplugged your fridge and everything else that can add electronic noise to the audio line?
  • Have you made sure that your screen recording isn’t going to be interrupted by someone’s curse laden Skype message box?
  • Will the guy from the back of the office knock on your door to ask you another question about Apache httpd in the middle of a voiceover recording session?
  • and, most important for me, is the conference room next door free of that loud-talking narcissistic idiot who likes to make too much noise for your recording sessions?

Recording 10 minutes of audio?  Maybe it will take you 15 minutes, but it’ll more likely take you an hour (or two).  I’m never surprised that it takes 20 takes to get it right.  Now, the last thing in the world you need to have happen after the moon and stars have aligned is have Screenflow die on you as it decides it doesn’t want to be a piece of software.  I use this tool frequently, and I’m constantly cursing it because it just disappears.  Maybe, if I’m lucky, it shows me the rotating rainbow wheel of “I’m about to just crash on you”, but, more often than not, the thing just, blip, and disappears – no record of the previous project or, worse yet, a corrupted recording.

It’s enough to make me stop what I’m doing and write a blog rant about the tool… Although, it isn’t like I’m moving to anything else, Screenflow is the best tool for the job (even though it has caused me to scream as loudly as possible twice in the last two weeks…. and if you know me, that’s pretty loud – I’m a singer.)

GitHub, Deploy Hooks Could Use Some Love

GitHub offers integration with a billion zillion services, and they just present them all in one big list.  Here’s the problem. Who knows what all these services do?  Notifo, Pushalot, Grove, DeployHQ, Depending, Jaconda, Irker, Planbox, Pipplydoodah, Circleci, JimmyBox, and one hundred more… I’d love to know what these services do, I would, but I feel like this list of options is absurd.

Here’s the combined screenshot of the deploy hook screen, if it were an artwork I would name it, “Do we really need another damn issue tracker?”  After this massive screenshot, some suggestions for GitHub…

github-awful-integration

You see how that doesn’t scale well?  It’s the phonebook effect and it gives every open source project or company in the space an incentive to name themselves something that starts with “A”.

  1. Sorting alphabetically by default, can we have other options? – Imagine if Google Play gave you a list of applications starting at A?
  2. Provide some categories for deploy hooks – I’d like to see all the issue trackers or all the notification services.   This would help make sense of the list.
  3. Give me some statistics about usage – Look at how the Drupal project displays statistics about adoption with graphs over time, do that.
  4. Provide users with the ability to comment on deploy hooks – If I’m contemplating using one of these hooks, it would help to know if “the crowd” had an issue with it.
  5. Make an editorial choice and feature some hooks over others – Why not?  Maybe you could charge these companies money to say good things about them?

If you made this page more of a showcase of the deploy hooks that people are actually using, if you exposed statistics about adoption, and if you made this page friendlier for novices you’d get even more signups per day than you are getting now.

Big Data Evaporates into Nothing. News at 10.

big-data-headlineOk, “Big Data” is going the way of “Information Superhighway” and “Web 2.0”.  How do I know?  I know this because I’ve seen a number of posts by people both attacking and defending it as a “thing” that could be considered real.  There’s a piece in the NYT attacking it, a bunch of people are now talking about “Data Skepticism.”  Don’t mistake the meme for reality.  Big Data is just a name invented to refer to nothing, and the marketing department has ruined it.

Here’s some short fictional dialogue that illustrates the way in which Big Data evaporates during the budget process at a large corporation:

CIO: When you say Big Data you mean Hadoop, right?  I was looking at the budget and I don’t see Big Data anywhere, but I do see this huge line item for Hadoop.  Jack wants a promotion, and he tells me everything you say BTW, he said you called me a “greyhair that doesn’t get Big Data”… I get it: Big Data is Hadoop.

Manager: Big Data isn’t just Hadoop, it’s a whole concept. It’s bigger than just a single technology.  Big Data is a new approach to data at scale, and I’m sorry about what Jack told you I thought making fun of the boss was essential mid-level management team building.  I apologize, I said that during the offsite meeting I thought he’d keep that to himself.

CIO: Forget about it, Jack’s not getting the promotion because I think he’s an idiot.   Back to this Big Data thing… ok, so it’s analytics, but we’ve had analytics from quite some time.   Is this just another name for Business Intelligence.

Manager: Dear God. Don’t call it that! You’ll scare away all the kids we just hired.  We need them. They are the only ones who understand Map/Reduce.  Big Data can also cover analytics, yes, but it is a different thing.

CIO: Wait, so the idea that businesses collect data that needs to be processed into reports that then drive the decision making process, that’s a new thing?  I thought we were…

Manager: …It isn’t that it is new it’s different.  Big Data is a different way of thinking about old problems in a new light.

CIO: Ok, so we’re talking about a analytics at a different scale, right?

Manager: Right, and different tools. Big data is more than just working with data at scale it is about asking different questions. It is about realizing the the value is the data and making sure you are taking advantage of it.

CIO: Paul, if you keep on talking like that, maybe Jack will get the promotion. You realize that you just said nothing to me. You said it very eloquently, but if I wanted bullshit I’d go read eWeek. Back to the budget… so Big Data is stuff like Hadoop and Map/Reduce?

Manager: No, stop it. That’s too limited. Big Data is transformational, it covers not only offline reporting but any systems that need to operate at internet scale: Google is Big Data, Facebook is Big Data, but the government is also starting to adopt Big Data.

CIO: Internet scale… our technology budget is a lot less than Google or Facebook, and we’re not indexing the web.   On another note, this Big Data thing sounds like a religion. Are there churches?

Manager: Why aren’t you taking this seriously?  There’s a lot of money being spent on Big Data.  I’m not the only one saying this stuff.

CIO: There’s a lot of money being spent on religion as well and there are many people that believe the world sits atop an infinite stack of turtles, that doesn’t mean I should take it seriously.  I’m asking because you are proposing that we spend a lot of money on it.

Manager: You know what, Big Data is Big Data you’ll know it when you see it.  Haven’t you ever worked with a Data Scientist.

CIO: Hold on, a Data Scientist, you mean a Statistician? Years ago, we hired people that knew SAS – that software has been around since the 1970s. Did you put money in the budget for a Data Scientist?   And, if so, do I need to install Bunsen burners and fume hoods in that office?

Manager: No. Maybe you should come with me to this conference. A Data Scientist is someone who knows how to work with Big Data.

CIO: You know what? This is fascinating, maybe we should schedule a meeting to talk more about this, but I’ve got to go talk to our analytics guy about the report he just sent me. We’re having a problem with the budget because we spent all this money on machines to support this Hadoop initiative, and it doesn’t look like we’ve found something to use them for yet.

Manager: Right, I wanted to talk to you about that.  Our Hadoop initiative isn’t paying off just yet because we need more data to analyze.

CIO: Thanks Paul. A quick tip from upper-management for future reference.  When I end a conversation telling you that “we should schedule a meeting to talk more”, that’s just a creative way of saying, “You are full of it, and I’m walking away now.”