Comments on: ETL Subsystem 26: Version Migration

By: ETL Subsystem 29: Data Lineage and Dependency « Tod means Fox

ETL Subsystem 29: Data Lineage and Dependency « Tod means Fox — Fri, 22 May 2009 14:33:00 +0000

[...] front-office sales database. Not only that, because you practice good logging, version control and release management, you also happen to know exactly when the front office system gave you the sales data, when it was [...]

By: Maintaining a DW/BI Environment « Tod means Fox

Maintaining a DW/BI Environment « Tod means Fox — Fri, 22 May 2009 11:16:23 +0000

[...] = ‘TodmeansFox’;In posts Version Control and Version Migration I glossed over one of the more complex and challenging aspects of data warehousing: Once the DW/BI [...]

By: 34 Subsystems of ETL Data Integration « Tod means Fox

34 Subsystems of ETL Data Integration « Tod means Fox — Thu, 21 May 2009 21:54:43 +0000

[...] Version Migration [...]

By: Tod McKenna

Tod McKenna — Wed, 04 Mar 2009 12:27:57 +0000

Hi Jagan,

I will soon! Check back in a few days

By: Jagan

Jagan — Wed, 04 Mar 2009 08:44:09 +0000

Hi Tod McKenna,

I’m waiting for your post on maintaining DW/BI environment with respect to deployment and versioning them (as u said in above column). I appreciated if you provide me those details thru mail. Thanks,
Jagan S.
India.

By: Tod McKenna

Tod McKenna — Sat, 31 Jan 2009 09:42:45 +0000

Hi Hans,

Sorry for the delay in responding. Here are a few ideas:

1.) Start thinking of you DW/BI environment as a publisher would think of a monthly magazine. You should release complete and targeted “issues” of your DW/BI project to your subscribers. You can easily do this by breaking your dimensional modeling projects strictly by business process. Release all the code and packages for that business processes only.

2.) Script everything — jobs, procedures, dml, ddl. Then organize your scripts by numbering them based on their order of installation. Each script should be smart enough to check for the existence certain objects and dependencies, and if not available, create them.

3.) Create a versioning matrix that shows the currently installed version of each component in the DW/BI environment. This will give you a birds eye view of how the different pieces of the system work together.

I think I’ll write a new entry on this topic over the next week or so. I’ll include some real examples and explain how I do it. One of our challenges is that we often have to release partial or incomplete systems to the business as fast as possible. This means that version control is complicated as multiple overlapping parts are released on different schedules. Tricky, for sure.

By: Hans Geurtsen

Hans Geurtsen — Wed, 07 Jan 2009 14:41:50 +0000

You say “Version control is tricky”, and I couldn’t agree more. I think particularly for BI/DW projects, versioning is tricky as it also involves large amounts of data. Also packaging the entire solution in some sort of a release, puzzles me. Could you provide us with some pointers were to go? You give us a lot of things to think about, but it would be even better to have some starting points to make these things easier.