What’s The Big Deal About Big Data

What do Moneyball, Twitter, and the US Government have in common? They’re all involved in Big Data. If you don’t believe me check out this NYT article, it’s got my back.

I started transitioning to a new job last fall. I’m one of the chosen few that get to research the latest technology trends and see how (if at all) they fit with Partner’s mission. One of the first ones I came across was Big Data. However, the enormity of the such a simple concept intimidated me and I focused on other things. Well, last week in our team meeting my fearless leader requested that we all look into it. And so I have. The concept is simple: never before has there been so much data available to so many people. But the concept isn’t the problem, it’s what to do with it that gets a little dicey.

I subscribe to the concept that data leads to information which can turn into knowledge which finally results in wisdom (if you’re lucky). One of the nuggets of information that Big Data can track is that which comes from car sensors. So if we use this as an example, the data would be a nearly empty gas tank. The sensor would turn that into that pretty little fuel tank light on your dash, there’s your information. Once that’s lit, you can turn that into the knowledge that you really should stop at Jack’s Gas on the way home. Finally, when the car doesn’t start in the morning because you didn’t pay a visit to the pump, you gain the wisdom there is in paying attention to the information your car is giving you.

Now, let us leave the DIKW model and turn our attention to another area where information is thriving: the internet. For a while now, folks have turned here to find the answers to questions of all shapes and sizes. If they want to know how to bake a cake, which king followed Edward IV, or how cold fusion theoretically works, you turn to your favorite search engine. It’s become so common that a new word popped up: google. What started out as a new spelling for an old word, googol, has now become both a noun and a verb. Got a question? Google it.

However, it’s not surprising that I know hear the public clamoring for some authority behind these answers. Oh sure, Wikipedia has a peer review of sorts, but how many times has a page been vandalized?  So new services like ChaCha and KGB are popping up to answer this call. But my question is, why are we trying to reinvent the wheel? We already have an established service for this: Reference Librarians.

I wonder how much the “uncool” nature of libraries come into play here. Is this push towards internet question answering born out of how cool the internet is or is it a by-product of the cool kids wanting to steer clear of stodgy old libraries? Or, alternatively, is it the result of resistance to a paradigm shift? After all, I remember using a physical card catalog when I was in high school. So, prevalent online cataloging is a couple of decades old while those cabinets with their nifty little cards go back to the French Revolution.Either that or I’m older than I though…

Whatever the reason for the shift, it won’t be long before the cool kids on the streets are tricked into using the magnificent powers of reference librarians and the technophobes in libraries are pulled kicking and screaming into cyber space. I’ve mentioned the DPLA and the Internet Archive in these pages before and I do so again to reinforce my point: the best place to go for answers is your friendly neighborhood library. These two ambitious projects have tapped the well-honed skills held by librarians and we’ll all benefit from their forethought.

Big Data is only going to get bigger. How we harness this vast cornucopia of data is up in the air. One thing that’s firmly grounded, though, is that technology is going to play a vital part. There’s simply too much to process manually. Which means another thing that should be planted right alongside said technology is the aforementioned reference librarians. They have the training and experience to handle the job. The question shouldn’t be whether or not to use them. No, the question should be: what color cape we should get them.