Boris Evelson, of Forrester, has a post up asking whether or not the famous 80/20 rule applies to BI.

I get tons of questions about "how much it costs to develop an analytical application." Alas, as most of us unfortunately know, the only real answer to that question is “it depends.” It depends on the scope, requirements, technology used, corporate culture and at least a few dozen of more dimensions. However, at the risk of a huge oversimplification, in many cases we can often apply the good old 80/20 rule.

He goes on to list some of the ways in which the 80/20 rule applies (80% of the costs are up-front vs. 20% on-going, 80% are for new people, services, processes, integration, etc. vs. 20% for direct costs like software or hard ware). 

This sounds about right to me--but I think the question we should ask is whether or not it has to be that way.  Is it really necessary that 80% of the cost goes into getting systems set up?  If you've got to figure everything that you want from the system beforehand, then probably so.  But it seems to me that a little flexibility in a BI or analytics system goes a long way to reducing that 80%.  Developing an analysis application in Spotfire is a matter of minutes to hours rather than days to weeks, and that's because the end results don't need to be specified up front (and shouldn't in order to get maximum benefit from Spotfire).

UPDATE: I see that the folks at LucidEra are thinking along the same lines, though for different reasons (SaaS reducing the up-front costs in their case, simplicity of analysis in ours.)

So it turns out that my doubts about the review of the Spotfire Professional client that I touched on here, have been largely repudiated. I read through the whole review, and it provides a good nuts-and-bolts overview of the Spotfire Professional client. There are things which I would quibble with, but overall a nice product review.

And, perhaps not surprisingly, one that Spotfire was rather heavily involved in the creation of. It turns out that we worked on the report several months back with BARC, who are-- previously unbeknownst to me, but perhaps beknownst to others--the parent company of The OLAP Report. So my apologies to Barney Finucane of BARC, who produced quite a nice review of the Spotfire Professional client.

The OLAP Report recently published a report reviewing Spotfire. I haven't had a chance to read the whole review, but there were a couple of things that jumped out at me that I wanted to mention.  The first is simply to note that Tibco wasn't involved in any way with the product review, which I gather the folks at The OLAP Report believe gives their judgments some additional credibility.

There's something to be said for keeping an unbiased perspective, but I think that by keeping the vendor totally out of the process, an evaluator ends up with a view of a software product that no one who would actually implement or use that product would ever have--I mean, when was the last time that anyone bought enterprise software without any interaction with, or support from, the vendor?

For instance, the first line in the report description labels Spotfire as "A very interactive, specialist analysis tool with advanced graphics."  I'll give them the advanced graphics and interactivity (these are general characteristics of the platform), but do have to take issue with the "specialist analysis" description as well as the "tool" appellation.  Regarding the former, it's true that we have a product which is geared to analysts (Spotfire S+), but it's also true that we have a web client designed specifically to make analytics broadly available and easy to understand, Spotfire Web Player.  Concerning the latter, calling Spotfire a tool significantly misrepresents the way that the various products work together to form an analytics platform.  Some of the elements built on that platform could be fairly described as tools, but either the OLAP Report folks really misunderstood Spotfire, or they only reviewed a single element of the platform (I strongly suspect the second, given that the title points to "Spotfire DXP" the former name of the Spotfire Professional client).

Either way, they've done their readers a disservice by not helping them understand the full spectrum of use cases that the Spotfire platform can address.  I don't object to an unbiased review of the platform--as with all software, there are things that Spotfire doesn't do, or doesn't do as well one might hope, but it's a bit unfortunate to simply not include key parts of the platform in the review.

To be fair, I haven't yet read the full report--it's possible, though given the report description, I think unlikely--that the full platform is accurately described therein, in which case, I'd be curious to know where the idea that Spotfire is for specialists comes from.  I'll have some more thoughts once I've read the whole report.

UPDATE: So it turns out that Spotfire did in-fact work on this report, and it's pretty good.  More details here.

There's an interesting article around R in the NYT.  TIBCO is mentioned briefly, and Lou Bajuk-Yorgan, my colleague, and a Sr. Director of Product Management here at TIBCO, wanted to offer some additional information on S+ (which as the NYT mentions is now part of the TIBCO Spotfire product family).  Here's Lou:

The burgeoning interest in R demonstrates that there’s demand for analytics to solve real, business-critical problems in a broad spectrum of companies and roles, and that some of the incumbent analytics offerings, in particular SAS and SPSS, don’t sufficiently meet the growing need for analytics in many major companies.

S+ (now TIBCO Spotfire S+) is a commercial software package based on the S language, which was a forerunner of R, and has been widely adopted. It is currently used in a wide variety of areas, including Life Sciences, Financial Services, and Utilities, for applications such as speeding the analysis of clinical trial data, optimizing portfolios, and assessing potential sites for building wind farms.

We welcome, respect, and appreciate the vitality, creativity, and sheer productivity of the R community, and the high quality of statistical methods the community creates. Because of the close historical ties between the two products, it is generally easy to port most R statistics into the commercial S+ environment, and we have worked to make that easier in recent releases.  

Once in S+, these analytic methods can be incorporated into intuitive tools for business decision makers and deployed to automated environments, using visual workflows, web-based applications (using standard web services), Spotfire Guided Applications for dynamic visual analysis, and scalable, event-driven architectures using TIBCO's IT infrastructure. S+ also provides some unique offerings, such as the ability to flexibly and efficiently analyze very large data sets. In this way, we feel companies can maximize the value of their analytic investments to make rapid business decisions, whether those analytics are developed in R or S+.

The Bits blog of the New York Times has a nice look at the Spotfire Gift Finder today:

Spotfire, a division of the software maker Tibco, has used its statistics expertise to come up with a tool for sorting potential presents. With the Spotfire Holiday Gift Finder, you can churn through thousands of products, including apparel, electronics, jewelry and tools. The software lets people narrow down their choices based on price and reviews and then points to the appropriate spot on Amazon.com where the product can be purchased.

Have a look!

Happy Holidays from everyone at TIBCO.  We've put together an application which should simplify your gift shopping this year, have a look!

A tounge-in-cheek, yet insightful, article from John Myers in which he refers to Analytics as the "Soft Drugs of Business"

The gray area that has emerged is the class of “soft-drugs” known as analytical business intelligence. To the quants in the organization, analytical applications are viewed as free choice that provides enlightenment to the user and benefit to the organization to “see outside the box” of existing standard operational and financial reports. The business stakeholders or quants fight for the freedom of the analytical applications since they usually provide greater value over the existing IT-sponsored operational or financial reports. To the data governance organization, analytical applications are viewed as something that should be controlled and regulated since they could lead to the “destruction of the youth of the company”

Well worth reading the whole thing.

I've got an article up at MyCustomer.com which reviews some of the interesting pieces available to organizations who want to compete on analytics:

Because of the importance being placed on analytics, it’s no wonder that the noise from technology vendors can be deafening. The sheer volume of players can complicate the technology selection process and mask the best practices around building and deploying analytics in the enterprise. 'Pervasive analytics', 'prescriptive analytics', 'predictive analytics' and 'business analytics', all have the same stated benefits, so what makes them different? To make the right selection, it's important to understand the various technology options available and be able to separate the wheat from the chaff when it comes to how vendors market themselves.

Instead of focusing on marketing terminology, more light may be shed by breaking down the categories of vendors vying for the analytics pole position. These categories include statistics vendors, vertical application vendors, business intelligence (BI) vendors and visual analytics vendors. In the end, most organisations will likely find that some combination of these approaches will be optimal, as no single approach can solve all needs.

You can read the rest here.

Earlier this week, we announced the release of Spotfire 2.2, the latest update to the Spotfire platform.  It's always good to get an update to the platform into the market, and we've made some great strides with Spotfire this year (a topic for another post), but I'm particularly pleased with some of the things we've added in this release.

Spotfire has historically (10+ years) been a leader in in-memory, interactive visualization, and given how much end users like being able to actually understand their data, it's not surprising that we've started to see other vendors adding some data visualization capabilities to their offerings.  Not-unrelatedly, we've started to get some questions about whether or not our core historical strengths were enough to continue differentiating ourselves from the rest of the market.

Without getting into the other things that we're doing to marry user-driven analytics with predictive analytics, event-processing and other enterprise technologies, the 2.2 release of the Spotfire platform provides a great response to questions about how Spotfire is different.

The two biggest additions are both new visual analysis tools:

The 3-D Plot allows for the visualization of multiple dimensions on a single plot.  While you can add multiple dimensions to a 2-D dot plot with the use of color, shape or size, or by trellising multiple plots, it's not always easy to identify trends within groups, or across different plots.

The 3-D plot addresses some of those challenges, and provides an understandable visual framework for displaying results from statistical techniques such as Principle Components Analysis (a dimension reduction technique for highly multi-variate data).

It's also great for cases, such as the example shown here--measurements from the drill hole of an oil well--where the actual data are measurements made in three dimensions.

Network Analytics, a extensible visualization tool for navigating and analyzing networks, is built entirely using the Spotfire public SDK, and it's something that I'm really excited about.

Wearing my analysis-loving geek hat, I think that analysis of networks is going to be one of, if not the, hottest area of data analysis in the not-too-distant future.  It's been used extensively for years in a few areas such as intelligence and other specialized fields, but its value is becoming more and more evident as everything becomes ever-more connected.

For instance, I'm a member of a Harvard-sponsored working group on Food Safety (last meeting detailed here), and it's absolutely critical for the FDA to be able to quickly traverse the immense network of food suppliers when there is an outbreak of food-borne illness, not only to identify the source, but to quickly clear the suppliers whose products aren't at risk.

That's not something that can be readily done with other types of visualization or analysis techniques.

Similarly, social networking sites such as LinkedIn, Facebook, Twitter and others create networks, the analysis of which is interesting to many, and a real business opportunity for folks who would like to advertise to targeted groups of consumers.  Such networks are only going to proliferate in the future, and the ability to understand them will be key to decision making across industries and disciplines.

Beyond those two items, there are a number of other improvements to the platform, but it's these two pieces that I'm really excited by.

"Analyzing data in aggregate is a crime against humanity."

That's according to Avinash Kaushik, Analytics Evangelist (hey, cool title!) at Google.  He goes on to say:

Bold statement, but the reality is that a “monolith” does not come to your website. Your site does not exist for a singular reason either. The core drivers of traffic are magnificently different for each core group of visitors.

So your website’s really a mix of Visitor Sources, Visitor Behavior and your Desired Outcomes.

When you look at all that in aggregate you get nothing. You think Average Time on Site means something. No! You think All Visits and Overall Conversion Rate gives you insights. Nyet! You think understanding Keywords without drilling down to each search engine will be awesome. Non!

If you want to find actionable insights you need to segment your web analytics data.

(emphasis mine)

The only thing that I'd change is that his comments are applicable to all data, not just web analytics data.  If you want to find actionable insights, aggregations just won't cut it.  You've got to move beyond the cube.

I'd suggest that anyone who thinks that it's not possible to cause all manner of trouble with uncontrolled spreadsheets read this:

A formatting fubar involving an Excel spreadsheet has left Barclays Capital with contracts involving collapsed investment bank Lehman Brothers than it never meant to acquire.

Working to a tight deadline, a junior law associate at Cleary Gottlieb Steen & Hamilton LLP converted an Excel file into a PDF format document. The doc was to be posted on a bankruptcy court's website before a midnight purchase offer deadline on 18 September, just four hours after Barclays sent the spreadsheet to the lawyers. The Excel file contained 1,000 rows of data and 24,000 cells.

Some of these details on various trading contracts were marked as hidden because they were not intended to form part of Barclays' proposed deal. However, this "hidden" distinction was ignored during the reformatting process so that Barclays ended up offering to take on an additional 179 contracts as part of its bankruptcy buyout deal, Finextra reports.

(HT: Andy Hayler)

I was recently asked to participate in the Harvard University Executive Session on Food Safety--hosted by the Kennedy School of Government--dedicated to enhancing cooperation and data sharing between the various components of industry and the agencies responsible for preventing and responding to outbreaks of food-borne illness.  It was attended by senior people from the FDA and State health agencies, as well as leaders from all points in the food-supply chain.  I was invited to provide some insight about how analytics might be useful in tracing products back to their origins in the case of outbreaks, and how such outbreaks could be predicted and prevented.
Interestingly, and perhaps unsurprisingly, the challenges aren't predominantly analytic, but related to data integration.  Think for a moment about what the FDA needs to go through to trace an outbreak:

From a set of cases, they need to track down where those who are ill ate or bought their food, and from each of those locations, track the implicated food back along its supply chain to its source, looking for points at which multiple cases converge to identify the problem.

If you’ve got the data on who bought what from whom and when, it’s a pretty easy problem.  However, the required data don’t conveniently live in someone’s data repository, but are diffused across all points of the food-supply chain.  Based on some quick googling, it seems that there are roughly 1 million restaurants in the United States and nearly 200k grocery stores.  They are sold to by a vast and complex network of suppliers, distributors, wholesalers, shippers and producers.  There is no standard for keeping shipping records, nor standard for describing which items are shipped—you wouldn’t believe how many varieties there are of a single vegetable there are, and how many more names those varieties go by.

The challenge of being able to navigate this data is immense—literally millions of different silos of information, much of it stored only in paper documents such as invoices.  Being able to do it under the kind of time pressure the FDA faces when there is an outbreak of food-borne illness is tougher still.
However, it is a tractable problem, and the session yesterday was a step towards a solution, and I’m looking forward to further sessions with the group.

On Friday, I visited the Boulder BI Braintrust with Spotfire's Sr. Director of Marketing, Mark Lorion.  Mark and I were invited to give the folks in the Braintrust an overview of Spotfire and get feedback from some of the brightest people in Business Intelligence and Data Warehousing.

Though the weather wasn't the perfect Colorado blue sky and fall air that I, being a CO native, bragged to Mark about, the visit with the folks at the Braintrust more than made up for the rain.  It was great to have so many smart people together in a single room, and get their feedback on some of the things that we're doing and planning here at Spotfire.  One thing that the group found particularly interesting was the on-going integration of Spotfire with TIBCO's event-processing and BPM software, currently sold as Operations Analytics.  Richard Hackathorn blogged the meeting, and describes the integration:

Operational BI is seeking advanced analytics that operate upon event streams. The gaps are quite apparent between mainstream BI sitting on top of the enterprise business data versus CEP (like TIBCO) sitting on top of the enterprise business processes. Spotfire can act as an integrating component that bridges those gaps. If Spotfire moves beyond the pixels-on-the-screen, its integration value will be based upon consuming data from and generating data to the BI infrastructure.

It was great to see other people as excited about this as I am.  As I've mentioned before, I think that the way that BI becomes pervasive is to embed itself into business processes, and doing analytics on the event stream presents an obvious opportunity for such an integration.

I also recorded a podcast with Claudia Imhoff, which you can find on the Braintrust's podcasts page. 

 I was pointed to an interesting post on the growing prevalence of R support in statistical packages.

In terms of OSS, we are seeing wholesale integration of R into such packages as Spotfire and SPSS.  SPSS it seems is even offering a menu system to access R routines! I’ve also heard rumors that SAS is demoing an R interface in their SAS/STAT Studio product.

In my opinion, integrating R into each of these packages will have the effect of making statistical code and models portable across packages. This will eventually dilute the value of the packages statistically and make their value being evaluated on ability to manipulate and import data, connect to databases, and how effectively they put together their menu systems.

On one hand, I woudn't say that Spotfire has a "wholesale integration of R," but with the recent addition of S+ to the Spotfire platform, it's clear that our support for R is stronger than ever before. 

The point of the original comment stands.  But even with the growing stature of R, there's a whole lot of value packaged up in the "ability to manipulate and import data.. and how effectively they put together their menu systems."  Offering end-users the ability to easily manipulate their data, and effectively interact with it is the entire value proposition for several software vendors, and a big part of the value for others (including Spotfire).  Statistical and predictive analytics are becoming a bigger and more important part of Business Intelligence, but even though they comprise a relatively small part of most BI vendors' offers, BI is still a multi-billion dollar market.

In any case, if R becomes the defacto language for statistical modeling, I like Spotfire's chances of competing with SAS and other BI vendors on quality of user experience!

How often have we heard the phrase "We don't want to be reactive, we want to be proactive" with the implication that unless we're able to take action in anticipation of events which will impact our business, we're going to be in trouble?

It may be true that taking pre-emptive action seems like a great thing to do when the information enables one to do so.  But as we seem to relearn every few years, past performance is no guarantee of future results, and models which use the past to predict the future are vunerable to changes which make those predictions invalid (just ask anyone at Bear Stearns).  Given that, I think it's worth asking what's wrong with developing really good reactions as an alternative (or compliment) to being proactive?

In responding to events once they've happened, we have the advantage of potentially full information on them, and beyond that, we're not locked in by a system built on what we thought would happen, but are free to react to what actually has happened.  We're given the freedom to fully analyze and understand the situation.

This needs to be balanced against the fact that if we wait too long to react, it's not much better than not reacting at all, but with the proper tools, I believe that reacting in real time can be even better than committing yourself to moving in one direction and hoping that reality accommodates you.

What makes for tool set that allows for real-time reactions?  Though there may be others, in my mind there are two key features of such a tool set:

1. A responsive, flexible analysis environment.  Given that one of the advantages of reactive analysis is that we're not limited by preconceived models, it's important that we not be limited by our tools.  The idea is to know what's happened as fast as possible, and if your tools limit the sorts of analysis you can perform, or can't perform them quickly, your reactions aren't as good as they could be.
2. An infrastructure which supports real-time data capture and distribution.  Obviously to do any sort of analysis, appropriate input data is required.  If that data needs to go through a lengthy process of being captured, cleansed and transformed before it's available, reacting in real-time is essentially impossible.  The data needs to be available when it's fresh and it needs to be easy for analysts to get to.

 I'll go into more detail on the details of each of these items in subsequent posts.