Contrary to hype, hopes and dreams, big data visualization is generally not helping humans make novel discoveries. Data visualization has two primary purposes: exploration and storytelling.
Hype. Imagine, for a moment, a beautiful, big ass graph – all four walls in the room covered with icons of people, companies, cars, boats, planes with lines between these entities depicting how they are related. Some icons are blinking. Lines of different colors and widths are undulating at varying rates to signify volume and pace of the flows (e.g., fast moving money). So much information is encoded in this magnificent representation that it took hours to study the legend in order to fully understand what all the color and commotion mean. The big graph can be re-scaled down to the size of a postage stamp fitting nicely on one screen (where at this density it appears as if a solid). Then with one click, the visualization instantly expands back to its previous form; consuming the four walls of screens. Reset to a specific date/time; play events forward or backward, fast or slow; all at a whim. Maybe this visualization system can even recognize hand gestures and voice commands to crawl through the picture – or better yet, this can all be experienced from within an immersive virtual reality environment.
Visualizations like this are magnificent and impressive; delivering true shock and awe.
Question: What are the odds this form factor and experience will help someone find novelty – true weak signal; that proverbial ‘needle in the haystack?’
Answer: Slim to none.
Such claims are likely pure hype.
Data visualization is best suited for exploration and storytelling, not ‘discovery of novelty.’
The most common use of data visualization is “exploration.” In this use case, someone has a specific place to start already in mind; an entrance point. The inspiration for the starting point may have been an alert from a fraud detection system, the boss has asked a question, a personal epiphany followed by a hypothesis, and so on. In each case there is an entrance point in hand before one enters the visualization.
Starting from this entry point one thoughtfully drills in and out, navigates up this branch or down that, etc. This exploration process helps one get their hands around what is known and how things are oriented (context). In conjunction with human expertise and intuition, this visual exploration assists investigations, audits, forensics, diagnostics, etc. Simulations e.g., 3D structural models that help imagine the house before it is constructed make for another good example that falls into the exploration category; the starting point being the front door of your house.
The next most common use of data visualization is “storytelling.” In this case visualization is used to deliver on the “picture is worth 1,000 words” axiom. For example, following much research one has an exciting finding worth sharing; maybe a call to action is in order. What better way to share the findings than an artistically finessed and professionally packaged visual aide. Visualizations used this way help make a case more compelling; whether one is presenting evidence to the judge or enlightening their boss as they ask for more resources.
What to do if one accidentally discovers important novelty when gazing at a big visualization: First, it is probably a good day to go buy a lottery ticket. Next, find out how long this condition was knowable before it was accidentally stumbled upon. Had this been knowable for years? Has this been discovered too late to matter? Also ponder how long it might take a future ‘visualization gazer’ to miraculously stumble upon such a discovery, if it were to happen again?
NOTE: It is for these reasons that systems which allow the data to find the data and the relevance to find you are so essential. Such systems detect actionable insights (new starting points) the moment they become knowable, e.g., supplying an eager fraud analyst that next lead worth exploring in a beautiful data visualization system.