Big Data Only Tells Half the Story, If You’re Lucky.

We’re living in a world where our devices are tracking our every move. Google, Yahoo, Amazon, eBay, Facebook, Pinterest, Twitter and countless other services are collecting crazy amounts of data on us. All this information is analyzed to uncover patterns of behavior so they and those they share their information with can better target us at every turn.

Many articles and books have been written that offer advice on how to improve an experience through design changes based on this big data. In fact an entire industry, Conversion Rate Optimization (CRO), has sprung up around the idea that big data can improve site performance. Can big data (quantitative data) alone be enough to realize the full potential of any experience? Can we forego in-person user research (qualitative data) now that we have so much data on WHAT users are doing? No.

Big Data only points to symptoms.

Big data comes tells us what people did, but it rarely every tells us WHY they did it. This quantitative data can show us patterns of behavior indeed but it almost never uncovers the underlying cause of the behavior.

Consider this crazy scenario (ignore the obvious ethical implications and play along for the moment) – a doctor opens the doors to the ER waiting room and finds a thousand patients waiting, all doubled over and clutching their bellies. This behavior is symptomatic of severe abdominal distress but alone is not enough for the doctor to safely prescribe a course of action. He’ll need at least 2 data points even begin to uncover a pattern to consider safe treatment options.

So our dutiful doctor looks around the room and visually examines the patients and notices that none of them are showing signs of external bleeding. This is good because he can now, with reasonable certainty, rule out gun shoots and stabbings. Now with 2 data points he can focus on internal ailments and begin to consider treatments.

If data alone were enough our doctor might begin by offing some sort of antacid or other drug designed to sooth and calm an upset stomach. But that would be crazy and possibly fatal to the patients. What options does our doctor have?

A/B testing will help but will only get us so far

Imagine now that our doctor takes an intuitive leap and assumes that not everyone is ailing from the same cause. He could, like any good CRO professional might do, treat half the patients with one remedy and the other half with another and then wait to see which group responds most favorably. The problem is some people could die by the time he recognizes the effects.

What if neither group responds well, what now? Does our doctor try to split the groups again and try two new treatments? What if some of the people in one group respond and some in the other do as well, then what? There are so many possibilities and variables. How can our doctor narrow this down?

To get to the cause you need to talk to the patient

Clearly our doctor can’t single handedly interview each and every one of the 1000 patients to gather a complete medical history and still help everyone. What if this is something more serious than a case of severe indigestion? Some patients would die before he got to all of them. What’s a good doctor to do?

He could begin with a small random sampling of patients and a short list of insightful questions. What was the last thing you ate, and where and when was that? When did you begin to feel the pain? Where do you feel pain and how intense is it?

Pulling out as few as 10 patients could begin to uncover a pattern that sheds light on the cause of the ailment. Let’s say that after interviewing 10 patients our doctor finds out that 7 out of the 10 had all attended the same social function and ate the same food. Now he get on the ER intercom and ask the other 990 patient who else attended the function and had the same food, pull them aside and treat accordingly.

With this small amount of additional qualitative data our doctor can now begin treat all his patients more quickly and with greater effectiveness than he could solely relying on big data. Why do I bring this up?

Treat the cause not the symptom for maximum results.

This how we go about making many decisions these day with web design, minus the possible fatalities and questionable ethics concerns of course. Conversion Rate Optimization (CRO) is based almost entirely on the use of big data. Don’t get wrong, CRO can be a highly effective way to improve the performance of a given page, albeit usually in small incremental percentages.

For some companies a shift as little a quarter percent can equate to hundreds of thousands, if not millions of dollars lost or gained. But radical shifts come from gaining deeper insights into user motives and pain points than big data will usually ever uncover.

To fully understand how powerful the combination of big data and just enough qualitative data can be look no further than Jared Spool’s story of the 300 Million Dollar Button.

The gist of Jared’s real life story is that big data pointed to a symptom for one major online retailer, high cart abandonment rate. Now they could have gone the CRO only route of A/B testing the hell out of the existing design but they would have never uncovered the real cause of abandonment. With a little in-person usability testing Jared’s team was able to discover the real source of pain.

“I’m not here to be in a relationship” was the response from one test subject and he was not alone in this feeling. Turned out that asking people to set up accounts at the start of the checkout process was putting them off. This kind of insight would likely never have surfaced through typical CRO practices. So what was the result of the changes they made based on that insight? A 45% reduction in cart abandonment and about 300 million dollars in gained revenue.

I encourage you to read Jared’s article to get full story. The reduction in cart abandonment and increased revenue were only two of many benefits the online retailer gained from the doing just enough qualitative research.

Quantitative data + Qualitative data = Maximum ROI

Big data (quantitative data) will help you spot symptoms and optimize your experiences so you need to utilize what’s available. Qualitative data on the other hand can transform the experience entirely by identifying the causes of pain. Used together, as illustrated by Jared’s true story this is a combination that will yield the greatest results for both your business and your users.

To paraphrase a bumper sticker: Know users, know success. No users, no success. It’s not enough to know what the user did, you need to know why they did it. While quantitative data requires a fairly large sample to be useful, qualitative data can uncover impactful insights from a relatively small sample.

If you think you can’t afford to do that kind of research I’d say, you can’t afford not to.

Published by

Mike Donahue

Just a guy that's passionate about creating useful, usable and desirable experiences for all humans. I love nature and wildlife photography, it's my source of artistic expression and inner peace. I live and play in South Florida with my wife Nikki and our girls (dogs) Layla and Cassidy.

One thought on “Big Data Only Tells Half the Story, If You’re Lucky.”

Leave a Reply

Your email address will not be published. Required fields are marked *