Contrary to hype, hopes and dreams, big data visualization is generally not helping humans make novel discoveries. Data visualization has two primary purposes: exploration and storytelling.
Hype. Imagine, for a moment, a beautiful, big ass graph – all four walls in the room covered with icons of people, companies, cars, boats, planes with lines between these entities depicting how they are related. Some icons are blinking. Lines of different colors and widths are undulating at varying rates to signify volume and pace of the flows (e.g., fast moving money). So much information is encoded in this magnificent representation that it took hours to study the legend in order to fully understand what all the color and commotion mean. The big graph can be re-scaled down to the size of a postage stamp fitting nicely on one screen (where at this density it appears as if a solid). Then with one click, the visualization instantly expands back to its previous form; consuming the four walls of screens. Reset to a specific date/time; play events forward or backward, fast or slow; all at a whim. Maybe this visualization system can even recognize hand gestures and voice commands to crawl through the picture – or better yet, this can all be experienced from within an immersive virtual reality environment.
Visualizations like this are magnificent and impressive; delivering true shock and awe.
Question: What are the odds this form factor and experience will help someone find novelty – true weak signal; that proverbial ‘needle in the haystack?’
Answer: Slim to none.
Such claims are likely pure hype.
Data visualization is best suited for exploration and storytelling, not ‘discovery of novelty.’
The most common use of data visualization is “exploration.” In this use case, someone has a specific place to start already in mind; an entrance point. The inspiration for the starting point may have been an alert from a fraud detection system, the boss has asked a question, a personal epiphany followed by a hypothesis, and so on. In each case there is an entrance point in hand before one enters the visualization.
Starting from this entry point one thoughtfully drills in and out, navigates up this branch or down that, etc. This exploration process helps one get their hands around what is known and how things are oriented (context). In conjunction with human expertise and intuition, this visual exploration assists investigations, audits, forensics, diagnostics, etc. Simulations e.g., 3D structural models that help imagine the house before it is constructed make for another good example that falls into the exploration category; the starting point being the front door of your house.
The next most common use of data visualization is “storytelling.” In this case visualization is used to deliver on the “picture is worth 1,000 words” axiom. For example, following much research one has an exciting finding worth sharing; maybe a call to action is in order. What better way to share the findings than an artistically finessed and professionally packaged visual aide. Visualizations used this way help make a case more compelling; whether one is presenting evidence to the judge or enlightening their boss as they ask for more resources.
What to do if one accidentally discovers important novelty when gazing at a big visualization: First, it is probably a good day to go buy a lottery ticket. Next, find out how long this condition was knowable before it was accidentally stumbled upon. Had this been knowable for years? Has this been discovered too late to matter? Also ponder how long it might take a future ‘visualization gazer’ to miraculously stumble upon such a discovery, if it were to happen again?
NOTE: It is for these reasons that systems which allow the data to find the data and the relevance to find you are so essential. Such systems detect actionable insights (new starting points) the moment they become knowable, e.g., supplying an eager fraud analyst that next lead worth exploring in a beautiful data visualization system.
RELATED POSTS:
What Came First, the Query or the Data?
Context: A Must-Have and Thoughts on Getting Some …
G2 | Sensemaking and its 7 Secret Sauces
Hi Jeff
I am a student studying in SMU. I attended your talk last month and loved it. I have been following your blog after that.
But I somewhat disagree with this post. I was thinking that maybe the definition of discovery is too narrow here. We may not be discovering new things by big data but we are definitely discovering new ways and methods of finding those new things. In order to support my argument, I would actually use an example from your own work. You told us that using Geo Spatial Analytics you were able to forecast the location of the asteroids which those astronomers couldn't with their current powerful computers. So finding this new method is a discovery in itself I believe. What do you think about this perspective?
Posted by: Rikin Khanna | February 14, 2016 at 11:42 AM
Placing this InfoWorld counter-point piece here: http://www.infoworld.com/article/3040708/analytics/data-visualization-showing-isnt-always-telling.html
And my counter-counter-point: I think you have missed my point while at the same time making a case for my point. For example, when you point out that “The heat map on the bottom right illustrates that default is more likely to occur on mortgages with more than 10 years employment length.” This visualization helps tell that story aka storytelling. But the question is how did you find this? Did you have a ‘big data visualization’ covering the walls containing a near endless combination of possible X and Y axis features? And with each, different time scales, geographical regions, and other often relevant dimensions? Did you pick out the one on the lower left because over the thousands of representations this one was more significant than the heat map showing mortgage defaults for those which considered age range, career, and marriage stability? Was this more interesting than the graphs on employee turn-over trends quarter to quarter in contrast to stock price?
My point is really: big pictures of all the data are pretty, but not that useful for finding novelty. Once something interesting is found then a bite-sized consumable picture (whether sourced from big data or not) makes for a useful way to explore for more understanding and storytelling.
I’d enjoy debating you on this one day. Albeit once we cleared up what we each meant it is likely we would be on similar page.
Posted by: Jeff Jonas | March 24, 2016 at 06:15 PM