Next week at TDWI in San Diego, I’ll be facilitating the 2011 BI Bake Off with IBM Cognos, Microsoft, and SAP Business Objects. With all three vendors recently releasing major new versions of their BI platforms, the bake off will be the usual exciting, nerve wracking, and enlightening session.
I conducted the first BI bake off at TDWI back in 2005, at a time when few companies had BI standards, and the market was highly fragmented.
Today, the BI platform market is more mature. The market has evolved into mega vendors that have BI platforms, numerous smaller vendors that focus predominantly on BI platforms, and dozens of specialty vendors that focus on a single segment within the BI space. With greater market segmentation, it might seem that BI buying has gotten easier, but it hasn’t. Smart companies make BI investments wisely, and yet business users often demand the newest innovations, regardless of who is touting them.
Most large companies now have a BI platform standard. Even though customers have made an initial investment, these companies continue to evaluate solutions, because:
- Vendors constantly release new versions, begging the question if it is better to upgrade or to make a switch.
- Customers are looking for opportunities to lower licensing costs.
- Some are not satisfied with support from the incumbent vendor.
- They were acquired or merged with another company, with a different BI standard.
- They’ve outgrown the capabilities of the product initially purchased.
- Budgets tightened in 2009 and 2010 have loosened. There’s pent up demand to expand BI capabilities.
For small and mid-sized companies, BI may be a new endeavor. They’ve outgrown their fixed, operational reporting solution and need something more robust. For many of these companies, BI platform breadth may be less important than ease of deployment and time to value.
When I conducted the first BI Bake Off, I kicked the session off with dry-ice emitting smoke, a bow to the smoke-and-mirrors in many vendor demonstrations. I promised attendees that we would try to cut through that smoke, with a candid look at side-by-side differences.Sadly, I have had to abandon the smoke as it violated the hotel’s fire code (killjoy, right?).
In the first few bake-offs, I also thought it would be sufficient to politely stand when a vendor’s time was up (I told them in the agenda they would have only six minutes, surely they would stop talking, wouldn't they?). Well, I learned that I needed something more akin to a gong, or at least a timer, as vendors will take as much air time as possible. Six years later, and it still feels rude to beep someone mid sentence, but it certainly makes for a fairer session. The more seasoned vendor participants now know never to run over time.
In the bake off, I give vendors a carefully scripted demo of the most critical features and differentiators that customers often look for. Attendees then vote on which product best met their requirements for that capability. Everyone knows that it is indeed just a demo and that a real-world test may yield different results. Still, what is clear is that the TDWI attendees are a sophisticated audience, equally tolerant and unforgiving. Attendees give high marks to vendors who candidly say “we don’t do that out of the box, but let me show a work around.” Conversely, one vendor who started with a marketing video was unilaterally panned, thank goodness with lack of votes, and not out-and-out booing.
Surprizing to me is when a vendor sells their product short. I’ve tested all of them, so I know their strengths and weaknesses. As the goal is strictly education – but some fun in the process – I do point out when a demo just didn’t go well or when I disagree with something a vendor said. I have sometimes chalked up these incidents to a vendor’s not being prepared (another thing attendees are unforgiving about), but frankly, I also think it’s that all the product lines have gotten more complex. While the marketing folks may be able to articulate product differences well, it is often the technical consultant who knows the product nuances. Hence, many vendors now bring two participants to the bake off.
For companies large and small alike, an ongoing challenge in managing your BI tool portfolio is in knowing when to standardize and when to mix and match. Standardization has the lure of lower cost of ownership, better support for business users, and integration across modules. That’s the lure; reality varies vendor to vendor. And yet, there are dozens of emerging vendors bringing often-times superior functionality than what may be available from a BI platform vendor. This is true of dashboards, visual discovery, mobile, search, and SaaS offerings, to name a few.
To help companies better evaluate these specialty solutions, in the last year, we have added detailed scores on BI Scorecard for a number of specialty products and vendors . At TDWI, we introduced the Dashboard Bake Off in February 2011. This class is a deeper dive on an increasingly important module; BI platform vendors and specialty vendors compete side-by-side on dashboard capabilities. Dashboards are also part of the BI platform bake off in August. We also give vendors a chance to showcase something cool and innovative that most differentiates them. It’s the one topic the vendor chooses; the rest of the scripted demos are topics I and the attendees choose.
With Cognos previewing their new visualization product in the labs and Microsoft whispering about project Crescent, I suspect visualization may be what these two vendors showcase during the “cool topic.” With SAP being one of the few vendors to natively supporting the BlackBerry Playbook, perhaps they will choose Mobile BI for their cool topic … but there has been a fair bit of noise about collaboration as well, so we’ll see.
While the BI Bake Off provides side-by-side product insights for three vendors, the other goal of the course is to help companies improve their own selection and bake off process.
To make the most of your software evaluation:
- Define your requirements and prioritize them, differentiating between near and long-term needs
- Understand the trade-offs between a solution from a single-vendor and pursuing a predominant standard
- Get educated about BI pricing and packaging early on.
- Ask vendors to follow scripted, not canned, demos
- Gather feedback from a broad constituent of BI stake holders
- Recognize that requirements change over time as users and software capabilities evolve. Never assume your BI tool strategy will stand still.
Making smart software investments and managing your BI tool portfolio is important to BI success. But as I often say, ultimately the question of which tool is less important than how you use your BI tools.