lørdag den 31. december 2011

Lessons from the real DW/BI life

Based on my latest experience from the field, I'd like to mention a few things that are close to my heart.
This is not a thorough check list and I'm not going to plunge you into lots of details for each single issue.
Although your experience may be different, please give them a thought anyway and you're welcome to share your opinion.
 
 
Be driven by adding or enhancing the business value. This is easier said end done, but I've seen that often the focus is on technology.

The long-term vs. short-term goal of your DW/BI team regarding building of an enterprise information infrastructure. And for it's data modeling freaks : there are pros and cons regardless of your usage of an Inmon, Kimball or Data Vault approach or not. Sooner or later - just like in marriage - there is a chance that you will regret it.

The ETL part does not equal the whole DW/BI, but it's a very important part of your DW/BI solution. Although the ETL part is 'transparent'  for the users, I faced situations tendencing to emphasize the metadata part of the ETL or the 'bureaucracy' of data stewardship in stead of listening to the business users and deeply analyse their requirements.   

The Toolset: When it comes to the architecture and product selection for your DW/BI system choose one vendor. If you choose a mix, then there is a huge risk that you'll spend your time integrating the systems or platforms. Microsoft DW/BI platform delivers according to Gartner the best ROI and TOC, but you may feel free to choose IBM, Oracle, SAP or another platform  if you really have strong arguments for it.

The BI solution fails in targeting users of different profiles and the natural flow of information between them: from top level management to analysts and information workers and to fullfill both strategical, tactical and operational business.
I worked on solutions where the ad-hoc or selv-service BI was not taken into account and where the entire solution tried to fulfill the daily operational business insight. No implementation of a Strategy Map, a Six Sigma, a top 5 or bottom 4 culture or even a deep understaing of the notion of KPI or KPI objectives as the underlying pieces of the business perspectives.

Data Mining is still one of the most underestimated parts of the DW/BI solution and I do not have any explanation. According to a IDC study, data mining should be the fastest growing business intelligence segment surpassing any other BI field. Unfortunately I can not recognize this trend. Customers prefer to explorer the yesterday's figures and not to discover patterns of data or predict the metrics with a huge potential to improve their business for a low cost.    
    
Heppy New DW/BI Year.

onsdag den 21. december 2011

Compare Cube space in an indirect way

A couple of weeks before, as we migrated our BI solution to different environments, an important task was to ensure that everything was in sync.
This makes sense as we run separate ETL flows on a daily basis in each environment, that in turn pulls out data from their own system of records, not to mention that the SSAS databases were transferred out of our control.
How can we check that both schema and the cube space for the SSAS databases is the same ?
I was not aware of any tool that does this job. On the other side, there are tools that accomplish the task for a relation database.
So, what should do the trick : to compare schema and data for the underlying databases and compare the SSAS XMLA schemas.

I used my Visual Studio Studio 2008 Team Database Edition GDR for the first task
When started, I saw duplicate dropdown items appear under Data menu. I proceeded any way but any attempt failed with an error message like:   
"The operation is not supported within this release."
A schema or data compare between two SQL 2008 R2 databases not supported in a Visual Studio 2008 SP1 release ?
I realized that the incompatibily issue sounds misleading and that the 'error' is related to the duplicate menues.
I was afraid of the idea to repair / uninstall / install the products in the 'right' order but the following steps solved my problem :

  • I closed all instances of Visual Studio Team System 2008 editions.
  • At the Windows Command Prompt, typed the following command:
  • %ProgramFiles%\Microsoft Visual Studio 9.0\DBPro\DBProRepair.exe RemoveDBPro2008 and pressed Enter
  • At the Windows Command Prompt, typed the following command: %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe   
A word of caution here: data compare is naturally based on a primary key constraint for the corresponding tables. If your fact tables do not have a primary key constraint, then you have to figure out another way in order to compare their data.

For the SSAS part, I used the BIDS helper. I strongly recommand this free add-on to BIDS and you can download it from: http://bidshelper.codeplex.com/
I scripted first out my SSAS databases in SSMS by right click -> script database as --> create to --> and saved the XMLA files
Then I used BIDS helper "smart diff" to compare the XMLA files and tracked the differences that may influence the final result from a user perspective..