I would have thought that by now business intelligence tools and Microsoft Excel would be happily coexisting. In some cases they are, but a larger number of Excel users, managers, and BI professionals simply seem battle weary.
Last week at TDWI in Washington, DC, I taught an updated course on "BI & Excel: Friends or Foes?” I last taught this course four years ago. I learned my first lesson about BI and spreadsheets the hard way back in the early 1990s. At the time I was the project manager for a reporting system based on a custom transaction system. Typical of many IT projects, I gathered business-user requirements, went away for a couple months (at least it wasn’t a year!), and we developed some parameterized reports on the mainframe. The final solution was flexible, interactive, and exactly what the business users asked for. We launched the new reporting app in a training class I had personally developed and was thrilled to be teaching.
We were only half an hour into the class, when the power user in the group, Frank, declared, “I don’t want any of this. I just want all my data in Excel.” Frank was the statistician in the group -- the trusted analyst. Normally, he and I were on the same side. We would swap notes on how to tweak Lotus macros or how to extract data from mainframe sources. But he basically just told me I had wasted months of effort and that what my team had built was crap.
The business managers in the room liked the reports, but they relied on Frank for all things data. Frank had planted a seed of doubt in their minds that our whole approach was wrong. Frank had become my foe. I tried not to get teary-eyed (hey, I was in my 20s!) and decided to cancel the rest of the class.
Frank felt fixed reports constrained him. He needed flexibility. For the managers in the group, who were not yet proficient in spreadsheets, the mainframe-based reporting system might have been fine. But Frank could make any data looking prettier in Excel, with better formatting, colors, charts, and so on.
Even though that lesson about the role of spreadsheets in BI was learned 20 years ago, BI leaders, managers, and Excel gurus continue to grapple with BI and Excel’s coexistence.
At the start of last week's TDWI class, about a third of the attendees agreed with the suggestion that Excel and BI are friends; Excel helps improve decision-making and fulfills the vision for business intelligence. “I'm on the friend side because I know if I treat Excel as a foe, I will lose," said one attendee. "It's easier to embrace your enemy and gain trust and creditability.”
The remaining two thirds felt that BI and Excel are foes: too many spreadmarts (some in the hundreds of megabytes) and multiple versions of the truth undermine the BI team’s efforts to provide a single version of the truth. One attendee who started the day thinking spreadsheets were the enemy later said the class had changed her view.
To fully appreciate Excel’s role in BI, start by understanding why users love their spreadsheets. Sometimes it may be personal and job security. Knowledge is power, and being the designated source of data is an enticing role. Beyond job security, there are a number of valid reasons that Excel plays a powerful role in business intelligence, such as:
- Familiar user interface: Attendees acknowledged that Excel is the preferred interface for power users, but managers and front line workers prefer dashboards and easier- to- use tools (they were wowed by SAS’s Email integration).
- Ability to tweak a report, whether to sort, filter, pivot, or remove a column
- Extensive formula library
- Access to multiple data sources: Excel’s ability to combine data from multiple data sources is a must-have requirement for many types of analyses
- Ability to “massage” the data: few attendees said they use Excel for data cleansing, but several people spoke of changing group roll ups.
- Ideal prototyping environment where users themselves can build applications.
For any given requirement, assess whether Excel is the best solution or if the BI tool would be better. For example, newer version of BI software allow users to filter, sort, and interact with a report via a browser. Habits are sometimes hard to change, and if users don’t know about these capabilities, they will fall back to what they have done for years (click that Export button).
The multiple-data-source issue is probably the biggest challenge. BI teams don’t like personal or departmental data to be accessed from a central business view and certainly not from within the data warehouse. Excel is ideally suited for joining data from multiple data sources for a one-off analysis. The scalability improvements with the PowerPivot feature in Excel 2010 make it even more suitable. However, for recurring analyses that require joining data from disparate systems, multi-source becomes a problem for the entire organization. The BI team has to recognize this or they risk being less relevant and the cause of spreadsheet chaos. Don’t back business users into a corner.
Most BI platforms now offer an Excel add-in that supports data integrity while giving users their data in the familiar Excel interface. It surprized me, though, how few attendees were either aware of or using these add-ins.
Why aren't add-ins being used? In some cases, it’s performance and licensing problems. In other cases it seemed to be lack of awareness. There also seemed to be a degree of fear: BI teams shy away from Excel because they’ve been burned in the past.
My recommendation is to recognize Excel as part of the BI environment. Manage the spreadsheets, whether via SharePoint, a content management system, or the BI portal. Consider carefully when it’s appropriate to use Excel versus the BI tool; and in all cases, make available add-ins part of your BI tool portfolio.
By the end of the morning, attendees were more positive, with two thirds agreeing that Excel and BI can be friends when managed. The remaining third are still wary. Where do you stand? Have you successfully deployed these Add-Ins? I'd like to hear from you.
And if you followed my last blog about a few Washington, DC, mishaps, for this trip, the cherry blossoms were in full bloom. The view of the Jefferson Memorial, in 80 degree sunshine, was nothing short of spectacular ... and I didn't get kicked out of a single cab!