Visual data discovery seems to be all the rage this year, with new products and high-growth companies. What’s driving this interest – the pretty pictures or the self-service? And will this new category of tools finally take BI mainstream or are we simply trading spreadsheet chaos for
another kind of chaos?
There is still a fair amount of confusion about what is visual data discovery and what it is not, so I’ll start with a definition:
Visual Data Discovery tools speed the time to insight through the use of visualizations, best practices in visual perception, and easy exploration. Such tools support business agility and self-service BI through a variety of innovations that may include in-memory processing and mashing of multiple data sources.
Some befuddled BI teams though are shrugging their shoulders, and asking, “Isn’t that what ad hoc business queries were supposed to do?” Well, yes, to a degree. Two of the biggest differences in business query tools and visual data discovery tools are the use of graphs and the degree of user autonomy. In a business query tool, a user can certainly add a bar chart to a dense page of numbers. But the chart is an after-thought. In fact, according to a TDWI survey last year, users spend two-thirds of their time analyzing data in tabular versus chart form. This may be appropriate when you need a precise number (“how many widgets do we have on hand?”), but not when you are trying to identify patterns, trends, and anomalies. With visual data discovery tools, the query and visualization process are one in the same. Drag a time period onto the page and up pops a trend line. Add a product category, and perhaps that trend line is now automatically converted to a trellis or small multiple chart. Research has shown that when data is represented graphically, we use less cognitive resources to make a decision and retain information better. So these graphs are more than just pretty or engaging; it’s about speeding the time to insight.
The other big distinction with visual data discovery tools from business query tools is the degree of user autonomy. Business query tools generally require a meta data layer that IT will often design and build. This meta data layer provides a layer of abstraction from the physical database schema, with potentially hundreds of tables. With a visual data discovery tool, business users are often working with a subset of data, either a flat file or spreadsheet, so IT is not a bottleneck. If a real-time query is involved, the visual data discovery tool may automatically model a meta data layer, giving its best guess at what’s a metric and what’s a dimension, again with little to no IT support. Somebody would have to write the SQL for the initial query and define the joins, but once extracted, the data is often loaded into an in-memory engine. (See BI Scorecard’s Visual Discovery evaluation framework for detailed features to consider.)
If the pretty pictures and degree of business autonomy make you want to rush to buy these tools and throw out your business query tool, keep in mind that there are gaps the visual data discovery tools don’t fill (yet?): many are desktop authoring and don’t work as well with multiple fact tables in a single query, for example. Others provide a high degree of interactivity and exploration on the desktop, but not on the Web so just how to share analyses is a work in progress for some products. So right now, I recommend companies add visual data discovery tools to their BI tool portfolio and view them as a complement to the business query module. Time will tell if the modules fully converge.
Wait! Haven’t we been here before with desktop OLAP tools? The industry pendulum does seem to be swinging back to the late 1990s – think Microsoft Analysis Services, Cognos Power Play, Essbase, TM1, all with their beautiful, highly visual front ends (Proclarity, Wired for OLAP, Executive Viewer to name a few). These departmental initiatives grew into chaos, so the desktop OLAP tools became enterprise grade … and IT once-again became a bottle neck.
The visual data discovery vendors, then, should take note of history: The more successful products and vendors will empower users without overwhelming them. The tools will be agile while also being scalable. And the savvy IT departments will embrace them rather than run from them. After all, in this era, it’s survival of the smartest and the fastest, not the perfectly controlled and architected.
How important is visual data discovery in your organization? Take the Successful BI Survey here.
Cindi Howson, BI Scorecard