Add to Technorati Favorites

 Subscribe in a reader

Trends and Outliers

TIBCO Spotfire's Business Intelligence Blog

Category Archives: Data Warehousing


Not All Big Data is Created Equal – Best Practices for Taking Action

The volume of data that’s now available to decision makers is mind-boggling. In fact, 2.5 quintillion bytes of data is being collected every day and the growth in data is simply astounding: 90% of all of the data that’s ever been created has been generated in just the past two years.

shutterstock 1519206711 300x271 Not All Big Data is Created Equal – Best Practices for Taking ActionHowever, experts warn that more data doesn’t necessarily result in better decision making.

“Even the best data doesn’t lead to the right decisions if you don’t know how to look at it,” says Rachel Kennedy, associate professor and director of the not-for-profit Ehrenberg-Bass Institute.

Continue reading »

Print post


Data Warehousing Development Standards = Efficiency, Quality and Speed

business man writing graph of industrial product and service improvement concept by increased qualit 150x150 Data Warehousing Development Standards = Efficiency, Quality and SpeedBuilding a data warehouse by following established standards will help your organization achieve a competitive advantage, lead to quicker development cycles, and realize a higher ROI.

Have you ever tried to fill every corner of a box with a single ball? The ball might fit but there will always be gaps.

Have you ever compared the characteristics of an apple to an orange? The results will always be the same, but the only conclusion that can be drawn is that they are different. You can’t glean any additional information from the apple-to-orange comparison.

Now think of a data warehouse design. The data warehouse will leave gaps, make comparative analysis difficult, and it won’t lend itself to self-service business intelligence if it’s built without:

  • a properly formatted physical structure
  • data that’s been subjected to a rigorous filtering and transformation process
  • a data warehousing schema that’s easy for an end user to use and understand

    Continue reading »

Print post


How To Develop A Top-Notch Data Warehousing System

DataExplosion How To Develop A Top Notch Data Warehousing SystemUnlike just a few years ago, each of us generates a “ton” of data every day.

Almost everything we do generates date including surfing the web; buying groceries from the supermarket; and sending text messages. In fact, with the right mobile technology, simply walking into our local malls generates data.

With this explosion of data, developing a top-notch data warehousing system is paramount to the success of any company. Processing data in a well-developed data warehousing system provides the competitive edge that companies are striving for. The question then becomes: What steps should you take to ensure you build a data warehouse that enhances and supports the decision-making process?

Mike Ferguson makes the point that you and your data warehouse development team have to understand certain things including:

  • The corporate business strategy: For your decision makers to make the best use of the data, your data warehouse development team must understand the corporate business strategy. Once they do, they can work with the decision makers to determine which objectives are priorities. They can also figure out how the data can lead to a higher return on investment. They must continue to work with the decision makers to join the data elements to the objectives then determine how to capture the appropriate data and build the necessary dependencies to make the data meaningful.
  • The data requirements: Your development team must work with the corporate data scientists and analysts to define data requirements, data sets and desired data visualizations to ensure that the data warehouse is highly interactive and that it allows users to customize the data sets, charts, and graphs. And they must make that information available in a variety of formats including dashboards, scorecards, and reports.
  • The technical environment: You must learn as much as you possibly can about the proposed technology and use that knowledge to draft the data warehouse technical architecture. Then you should participate in all facets of the technology selection process and work with the team to develop an implementation plan, which should include:
    • A metadata repository – to track information about both the data and the system including processes. Define the business vocabulary, store it within the repository and share it across the organization.
    • An ETL process – to extract data from transaction systems, transform the data into something suitable for the data warehouse and then load the data into the target system; i.e. the data warehouse. Pay close attention to how the data is manipulated, how long the process takes to completely execute, and the accuracy of the data. Be prepared for data that may be incomplete, incomprehensible, and inconsistent and develop processes to handle the issues.
    • Security – what level of security will be required? How will the data warehousing system be maintained to the level required by the organization, internal compliance, and external laws and regulations?
    • Data integration – combine data from several disparate sources into a unified data warehousing system. Then attempt to embed the system within existing corporate software for quicker adoption as the user may feel familiarity with the look and feel, leading to less end-user training.

Next Steps: 

  • To learn more about how analytics can improve your business and increase your bottom line check out these complimentary guides:
    • 5-Minute Guide to Business Analytics,” to find out how user-driven “analytic” or “data discovery” technologies help business and technology users more quickly uncover insights and speed action.
    • 5-Minute Guide to HR Analytics,” to discover the three critical capabilities a modern analytic environment must provide to the entire spectrum of HR staff so they can adequately support the enterprise.
    • 5-Minute Guide to CRM Analytics,” to learn how agile analytics technology can help you deliver critical value to executives and front-line marketers.

Dennis Hardy
Spotfire Blogging Team

Print post


How to Turn Your Data from Archenemy to Ally

How to turn data from archenemy to ally 217x300 How to Turn Your Data from Archenemy to AllyLast week we had some fun comparing a data analyst to a superhero. But we all know every superhero has an archenemy – Batman has The Joker and Superman has Lex Luthor.

As a data analyst, you’ll have to confront your share of villains too. They may even have terrifying names like Big Data, Dirty Data or Data Chaos. OK, maybe these aren’t such villainous names but left unchecked they can cause major problems for data analysts and company executives alike.


Well, for one thing, big data poses big challenges for traditional analytics approaches, according to

That’s because while your company’s focused on the volume and variety of “big data,” it doesn’t really spend enough time figuring out how to transform the massive amounts of stored, raw, structured and unstructured data into useful, real-time business intelligence to make better decisions.

Continue reading »

Print post


Twitter Roundup of Last Week’s TDWI Conference

Last week, The Data Warehousing Institute (TDWI) held its annual conference in beautiful San Diego.  This year’s focus was on the evolution of Agile BI. Below is a summary of the tweets coming from the event’s hashtag #TDWI.

Top Tweeters:

Here’s our leader board of top Tweeters taken from a data visualization of the event’s tweet stream.

Screen shot 2011 08 15 at 12.44.49 PM2 Twitter Roundup of Last Weeks TDWI Conference

Of the 298 tweets:

Philip Russom,Research Director for Data Management at TDWI had 15.9%

Claudia Imhoff, a BI consultant and Founder of the Boulder BI Brain Trust, had 10.83%

Tony Carrini, publisher of SourceMedia’s Information Management, Health Data Management and MDM & Data Governance Summit, came in third with 6.37%.

Tibco Spotfire was next with 5.3%.

DBA-Alex, BI Engineer at iContact, came in a close fourth with 5.10%.

Scott Humphrey, a PR consultant, also had 5.10% of the tweets.

And Quest, an IT management company, rounded out the top users with 4.46% of the tweets.

Top Links:

Many links were tweeted and re-tweeted during the event.  Some of the top links were a link to a mobile BI app for the iPad and links to two youtube video interviews during the event.

Continue reading »

Print post


Top Considerations When Working with Big Data

big data opportunities and challenges Top Considerations When Working with Big Data Daily Finance columnist Anders Bylund (@TMFZahrim) made a big deal about Big Data investments this past week and showcased how BI software is the solution for dealing with all the data once you have your “enormous database.”

While this column was directed at companies to invest in, Bylund provided a good example in calling data analytics software “the muscle to wring meaning from it [Big Data].”

However, it’s not as simple as purchasing software when you consider today’s deluge of data, according to a Search Data Management article. The traditional data warehouse is full of data that’s mostly structured, says Forrester analyst Brian Hopkins (@practicingEA). That’s not the case anymore – today’s data is semi-structured at best.

Continue reading »

Print post


Top Five Benefits of a Data Warehouse

Benefits of a Data Warehouse 300x202 Top Five Benefits of a Data Warehouse

According to The Data Warehouse Institute, a data warehouse is the foundation for a successful BI program. The concept of data warehousing is pretty easy to understand—to create a central location and permanent storage space for the various data sources needed to support a company’s analysis,  reporting and other BI functions.

And it’s really important for your business.

But a data warehouse also costs money — big money. The problem is when big money is involved it’s tough to justify spending it on any project, especially when you can’t really quantify the benefits upfront. When it comes to a data warehouse, it’s not easy to know what the benefits are until it’s up and running. According to, here are the key benefits of a data warehouse once it’s launched.

Continue reading »

Print post

Filed under: Data Warehousing


Landscape: Data Warehousing and Business Analytics

Spotfire Landscape:  Data Warehousing and Business AnalyticsThe Terrain

This landscape has changed dramatically in recent years.  Not so long ago, IT investment was focused on building data warehouses to store large amounts of static data in a structure designed to facilitate reporting.  The reports were usually designed and delivered by BI workers inside the IT department.  But as organizations began to recognize the value of their warehoused data, demand grew for broader and more flexible access among business users.

Continue reading »

Print post


How “Dirty Data” Derails Your Company’s Data Analytics and ROI

Data Analytics and Dirty Data How Dirty Data Derails Your Companys Data Analytics and ROIHow much of what your company knows is useful? Which data is current? In a recent webinar about the dangers of “Dirty Data” Jay Hidalgo of The Annuitas Group says many companies simply don’t know. He estimated that 30 percent of companies have no strategy for “data hygiene” – removing duplicates or obsolete information. He said 34 percent of companies ask the front-end sales team to update customer and prospect files but even that can be flawed among multi-product or different “views” of the same company, client or transaction.

Continue reading »

Print post


A New Kind Of Data Warehousing Will Emerge in 2011 According To Gartner

data warehousing 150x150 A New Kind Of Data Warehousing Will Emerge in 2011 According To GartnerAn article in summarizes the latest data warehousing report from Gartner entitled “The State of Data Warehousing in 2011.” In the report, Gartner predicts a turning point in the evolution of the data warehousing market this year.

The concept of the ideal data warehouse is changing. Data loading into data warehouses is fast becoming a continual process and warehouses must increasingly support new and extreme types of information asset formats. At the same time, the focus of business intelligence and data analytics is shifting, as organizations place business intelligence and data analytics tools into the hands of end users. In response, the focus of the data warehousing market is changing from one of storage and access to one of delivery and comprehension.

Continue reading »

Print post