When I sat down at my desk and began putting together my notes on this subsystem a couple of weeks ago, I was reminded how important version control can really be. In fact, as I was looking through a few articles and books on the matter, I realized that I don’t do a great job of version control management. And unfortunately for data-warehoused businesses around the planet, I’m not alone.

What I currently do

I use version control software for all packages, database scripts, and configurations. Although not the greatest of the bunch, MS Visual SourceSafe does the job (we’ll be switching to Serena Dimensions soon, though). From reading various articles and from a few first-hand experiences I can tell you that many teams don’t even do this much. Ouch.

I tag each package, job, and source with a version number and relate these versions to the database and BI application versions. I will also label the source code in VSS to match. For example: version 4.0 of the database aligns with version 3.1 of the data integration systems which supports version 1.7 of the BI Portal. I currently coordinate this using a spreadsheet in Excel, with high hopes to get this into the database in the form of metadata that can be queried and used for planning and deployment purposes.

I should mention that versioning is particularly important in compliance-laden environments. If it is important to show an auditor how data got from point A to point B, then you’ll need to dig out the exact versions of the software that loaded the data at the time. This is why including version numbers in your logging and auditing routines is very important. I wrote earlier this year on this topic in my article “Auditing Your Warehouse for Sarbanes-Oxley Compliance“. Take a look if you want more information and to see how I implemented this using SSIS.

At this time, I could segue into impact analysis; I’ll leave that for another post.

In addition to source control software and versioning, I use an LDM (Logical Data Mapping) that is stored as “business” metadata in the database. This metadata explains all source-to-target mappings and is sensitive to start and end dates. For example, if a vendor adds a column to one of our input files, I add the new column to the LDM and set the begin/end dates appropriately so that the historical mappings are left in tact. In this scenario, I can load an old file or a new file with little effort. When time permits, I will build logic into my SSIS packages so that, based on the date of the file being processed, the appropriate child package will be loaded.

What I don’t do yet

So far so good, but this is where I fall short. I have source control. I use major and minor versions and connect them to all other elements in the data warehouse environment. I even have a time-sensitive, table-driven LDM system. But I am lacking in the “control” area which has me worried. I have a lot of to-do items and half implementations.

What I need is a (fully automated, metadata-driven) system to coordinate different releases on different schedules and allow the packages to work (or not work) under certain database versions and environments. It would be nice to have some indicator in the ETL system that tell me if a certain version of a certain input file will load with a certain version of the database. To complicate matters more, much of what I do is based on certain mathematical models that sometimes change (yep, they have versions too). The question becomes, what version of what package works with models X, Y, Z for what database. Normally (99% of the time), everything is simple: the latest version of the package works for the latest version of each model on the current version of the data warehouse. That other 1% of the time is what makes my job interesting!

Assuming that you already use source control…

SQL Server 2005 Integration Services (SSIS)

Some elements of version control are simple using SSIS. For starters, you are able to set the VersionMajor and VersionMinor properties to your ETL version. When you make a major change to the package — so that it will no longer work with historical data, for example — then you can also update the package ID (see property “ID” under the Identification group in the Properties window). As a general rule, you should update this ID if the package works fundamentally different than it did before.

When I get around to implementing the system for selecting the correct version of the package that matches the input file version, I’ll be sure to write about it. For now, it’s merely a figment of my imagination and needs some fleshing out.

What I have done, though, is create a custom Task which examines the input source file against the LDM. If a column name or type is out of place, the file is rejected. This prevents loading a file that might have an extra column suddenly or if columns were swapped for some reason. We deal with input data that is (mostly) trustworthy, but sometimes dirty. As a result, this extra checking has become necessary. If anyone is interested, I would be happy to share the code and logic of this custom Task in a future post. Just let me know!

The above is not a scenario that exists in every environment. For me, detecting the file version is very important. For you, it might not be a challenging issue.

I have also linked the package’s ID to the LDM and my logging system so that each package — as represented by its GUID — can relate to one or more versions of my mappings. This allows me to modify packages, but keeps the modifications tied to a particular mapping.

Hand Coding with Visual FoxPro (VFP9)

Good developers will appropriately version their software. This should be part of How to Write Software 101, but is surprising to me how many times I’ve seen poor or non-existent versioning schemes. Hand-coding ETL is also a software development project and deserves the same versioning-attention you would give to any other good software.

In addition to versioning and source control, you should also consider versioning your metadata as I discussed above. I have written about and given presentations on this topic in the past with respect to the Logical Data Mappings. Check the following for more details:

• Hand Coding the Data Warehouse with VFP
• 2007 Fox Forward Presentation Logical Data Map Database
• Warehouse Data with Microsoft Visual FoxPro

The LDM is not the only metadata to add to your version control, but this should give you a real good start!

From Here

So you have a good version control system in place. Now you need to migrate your current versions from your development environment into production. That’s what Subsystem 26 is all about. And lucky enough, that’s next on my list!

As eluded to in my previous post (Backup System), bad things happen to good people. There are a ton of reasons to justify having a good backup procedure. A lot can go wrong with the hardware and software that make up your system. From network failures to bad service pack installs, a backup system is a must. But the best backup system in the world only gets you so far if you don’t have a system to recover or restart a failed process.

This is an ETL / Data Integration topic, so I’ll leave out discussion on restore capabilities found in backup tools. Of course, there may be times when restoring entire files, intermediate results, and staged data is necessary, but this is a job for your IT department. I’ll focus instead on how the ETL system should handle exceptions and failures.

At its most basic level, a recovery and restart system must allow you to stop, start, and retry your ETL processes.

Some might argue (and I have been in the room with these debaters) that it is simpler and more manageable to simply restart the jobs from the beginning and not bother with the added complexities of recovery and restart. While this might be true for simple environments or small jobs, it is certainly not true for larger, more critical projects. Restarting, for example, saves you from having to re-acquire data from your sources. This is a huge benefit, as some sources are only available to you during certain load windows. In some cases — like in my environment — some source files are overwritten each day! Other times, your source data might come from an FTP site. Downloading the same (large) data over and over takes time.

Keep in mind that your recovery and restart systems must be smart enough to clean up partial or damaged work. You’ll need the ability to back out of processes that partially finished, or even to rollback committed changes in order to ensure continued quality and integrity of your data warehouse and staging area. This is not an easy task and requires a lot of planning. Make your life easier by planning well ahead of time and try to link your recovery and restart procedures directly to the times when you write data to disk (”staging”).

SQL Server 2005 Integration Services (SSIS)

To start, use transactions in your packages. Transactions are an often underused capability of SSIS. They should be a part of your package designs so that blocks of tasks can be handled together as logical units. Transactions can be defined on the Task and container level, giving you the flexibility to nest and work with transactions in parallel. To get this working, you will need to set TransactionOption on the parent container to “Required”. This will initiate a new transaction if its parent hasn’t already done so. Child Tasks can participate by also setting TransactionOption to “Required” or “Supported”. A common technique you could employ: Use a Sequence Container to group related Tasks together, setting the Sequence Container’s TransactionOption to “Required” and all of its contained Tasks TransactionOption to “Supported”.

But transactions only get you part of the way there, and are not suited for every situation.

SSIS comes with a powerful and also underused feature called “Checkpoints”. Checkpoints allow you to restart failed packages from the point of failure, instead of rerunning the whole package again. The process is rather simple. SSIS will record information about its current execution to a special log file (this log file is overwritten each time the package is started from the beginning). When the package fails, and is subsequently rerun, the package will read the checkpoint log and attempt to pick up where it left off.

What’s particularly handy about SSIS’s implementation is that you can control checkpointing on Tasks and containers using the FailPackageOnFailure property. Set this property to True for each Task you want to participate. Furthermore, checkpointing is smart enough to recognize and handle transactions.

Hand Coding with Visual FoxPro (VFP9)

VFP supports transactions for both database tables and free tables (the latter works in VFP9 if you use the MAKETRANSACTABLE() function). This is nice, but doesn’t allow you — without some creative coding — to wrap transactions in the same way SSIS does with its containers and tasks.

You also don’t get built-in functionality to resume processes in Visual FoxPro after a failure. You might be able to use the RETRY command in combination with an error handler to fix problems as they are occurring, but for the sake of this subsystem, you need a “retry” mechanism that can span multiple sessions (i.e. restarting a process after the executable has already exited). The consequence is that you will need to design your ETL system in such a way that it can pick up where it left off, or go back to a previous step. You will need to:

• Create “markers” in the workflow of your code (the most natural place for any marker is right after data has been staged to disk, but you could also create intermediate markers as well)
• Log successful passes through each marker so the system will know where to pick up where it left off
• Make each marker smart enough to clean up any partial work left over by a previous failed run

The easiest way to implement the third point above is to use transactions in combination with cursors for all temporary work. If a job fails (the lights go out) then your cursors will be lost, but you should be able to restart from the last good marker. Since the cursors didn’t do anything permanent, there is nothing to clean up. This creates a natural and easy-to-manage recovery and restart system.

The bottom line: You’ll need to be creative when implementing a recovery and restart system using VFP (or any other hand coded solution).

From Here

Next, I’ll talk about version control (Subsystem 25) and version migration (Subsystem 26). Both of these subsystems deal with deployment and have tripped up many teams (I’ve been a part of such teams and have hopefully learned from the mistakes).

You’ve just spent the better part of the year designing a perfect ETL system - from data profiling to data propagation. You’re in production, and all jobs are running on schedule in what seems to be an eerie harmony. You lean back in your chair. Victory, you think.

Then comes the call: Your database server crashed. Now what?

You must have a reliable backup system in place — and not just overnight tape backups conducted by your IT department. Those are a good first (baby) step, but they are usually not good enough, especially to help you recover from failures in mid ETL stride. Even with a fully redundant environment, you have got to be sure that you can re-run jobs from your staging area no matter where they failed. This is an ETL responsibility not to be passed off to IT. This means that an accessible backup of all staging data is essential. Remember, you should stage data at each major step in the ETL process (after load, after cleansing, conforming, and just before delivery). Doing so allows you to return to these staging points in order to recover from failures when they occur during ETL processing. I’ll discuss this more in my next post “ETL Subsystem 24: Recovery and Restart System”.

If you use a staging area, then you’re already part of the way there. The only thing you need to manage at this point is to plan and implement your backup strategy. Backing up staged data as it is being staged can be tricky. You can write to two different locations, on different machines or on different drives of the same machine. Consider this the poor-man’s approach to staging area backups. I’ve done this often; as it is easy and inexpensive (disk space is cheap and getting cheaper). More sophisticated mechanisms exist (for example, check out IBM Tivoli). Your policies, existing services, compliance concerns, and your ability to recreate lost data will all play a factor in determining what level of backups you need to make.

Storage of these backups, including the question on how long to keep the archives, is something you’ll need to consider as well. For critical data that cannot be reproduced, you will want to store it for a very long time in a safe location. For easily reproduced data, you might want to keep it long enough to appease the auditors. For the staging area, it might be safe to keep it only until the next run.

From Here

I won’t frame this subsystem using SSIS or Visual FoxPro, as I’ve done with the other subsystems to date. Backing up your data and files — unless you use the poor-man’s approach — is not a function that you can do using the ETL tooling. And doing a manual copy of staging data might be as simple as moving files from one directory to another, or using database replication to replicate changes from your main database to a linked, backup database on another machine.

You have a lot of decisions to make when it comes to your backup system. Next post, I’ll discuss recovery and restart, ETL Subsystem 24.

This subsystem addresses one of the more basic management-related functions of ETL: Scheduling. You must provide a way to reliably run your packages and scripts. Data moves into and out of the Data Warehouse continuously; scheduling provides a structured and predictable way of managing the process.

I use the term “predictable” because it is often necessary to provide business users and other applications with data at specific intervals throughout the day and week. In my current project, for example, we provide a reporting data mart to a specific analytical application every morning at 9am. This reporting mart must have data up until the close of the previous day (and is not refreshed throughout the current day). If the timing wasn’t right (if, for example, some of yesterday’s data came into the database at 9:30), then the reporting database would not be ready for the business users. All of these timing issues are solved with a scheduler (and of course some agreements with the data providers to ensure data is available to process by the time the scheduled task runs!).

Scheduling is also used to organize and arrange processes so that dependencies and relationships among disparate data sources are processed logically. If data A must be loaded before data B and C, then data A will be scheduled first (worth mentioning again: a prerequisite for this is that the data is available from the source at the right time as well!). If one load fails, a graceful exit from the process must be made with adequate notifications and alerts. Restarting at failure points (without having to rerun the entire process) is also a highly desireable feature of any scheduler.

I’ll get into these topics more in a few of my next subsystem posts.

SQL Server 2005 Integration Services (SSIS)

Scheduling SSIS packages is easy. The best option you have is also the most intuitive: The SQL Server Agent.

The SQL Server Agent is a Microsoft Windows service that executes scheduled tasks. These tasks are referred to as Jobs. A single job can consist of multiple steps and be run on multiple schedules. Schedules can be configured to respond to a specific event, at intervals, specific times, or on demand. In a single job, for example, you can run a few SSIS packages, some database stored procedures, and a few command line utilities. These can all be scheduled at various times throughout the day (perhaps each hour, or once in the morning and again at lunch), and you can get an email telling you when it finished.

The scheduling aspect of SQL Agent is quite powerful. Not only can you create a seemingly endless combination of schedules, you also have full control over how to handle each step if it fails or succeeds. All of these steps can be linked together in such a way that if any one job step fails, processing can halt or move on to another step (of your choice). If you think about it, you have built-in If/Else/EndIf and CASE logic available right in the scheduler. I find that developers who come from a programming background can make quick sense of this put together some rather elaborate and logical flows, with dependancies, prerequisites, and alternative paths built in.

Knowing this ahead of time can save you some effort in your ETL packages. So here’s a tip: Use the scheduler to manage package dependencies where you can, rather than trying to build these complexities into your Control Flows. Of course, every package should start out by examining the environment, to be sure that it has everything it needs to start processing. This might be the existence of a CSV file in some directory, a flag in a staging table that indicates a previous process has finished, or any other check-able condition. If you can get into the habit of using the scheduler for some dependency management, and you are familiar with using parent/child packages, then you really have full control over your ETL processes.

One drawback to using the scheduler for managing dependencies is that your packages are now dependent on the scheduler! If, for example, you need to debug a package, you may be forced to run other packages first, or you may need to manually create the environment ahead of time (set flags, move files, run other scripts, etc.). This is in contrast to having a single super master package call dozens of other master packages and SQL Tasks in a more visual way. The scheduler in this environment would simply schedule the super master package and that’s all.

Hand Coding with Visual FoxPro (VFP9)

Finding solutions for management-related and process-related development problems in VFP is easy if you utilize the wealth of information and tools available from the FoxPro community. The community has always been one of FoxPro’s greatest assets. And this subsystem, more than any other discussed so far, highlights this.

Scheduling is not something you can do out-of-the-box with VFP. One very good option is to simply use the Windows Task Scheduler to run your compiled executables or application files. If you’ve been following Doug Hennig’s blog, then you might have seen his excellent Task Scheduler class. He writes:

This provides classes that can be used to create tasks in the Windows Task Scheduler. There are separate classes for Windows Vista, which uses Task Scheduler 2.0 and earlier Windows versions, which use Task Scheduler 1.0. Updated 2007/04/26.

TaskScheduler.zip is available for download on Stonefield’s Whitepaper and SourceCode page.

If you’re using SQL Server for your data warehouse database, then you could (and rightfully should) use the SQL Agent to run your VFP ETL applications from the command line (see my previous section on SSIS). In fact, in one system I worked with, I used a combination of SSIS packages with VFP compiled applications to handle a fairly complex VFP/SQL Server data integration environment. I used VFP much like SQL Server would rely on stored procedures and the CLR to conduct some complex row and set-based analytics. The SQL Agent — as with any scheduler — is a great way to bind different technologies together.

An option you should not explore is using the VFP Timer control in a running VFP application to schedule tasks. There are better options out there. If you disagree, feel free to start a discussion in the comments section of this post. I’ve attempted to use the Timer a few times in the past for these sorts of things, only to scrap it and go another way. One area worth exploring, however, is using a Timer in an application designed for the SysTray. Microsoft provided a SysTray class as part of their Solutions Samples when VFP8 was released. But even still, the scheduling capabilities of both SQL Agent and the Window’s Task Scheduler are very good and built in to both Windows and SQL Server environments.

From Here

I’ll continue discussing the management-related ETL Subsystems in my next series of posts. The next one is the Backup System, ETL Subsystem 23. Feel free to pass along any questions in the meantime!

I joined this project in mid-stride back in January when I was hired as an external consultant. I was coming over fresh from the United States, to a new and exciting land, Belgium (where the waffles are hot and the chocolate is worth dying for) and a new industry, financials. After a year and a half in healthcare at Cardinal Health, I felt I was ready to enter the financial world — particularly investment management. And so I joined the quant team in the quantitative research department at ING Investment Management.

I was hired as a BI/DW consultant (functioning primarily as an Integration Architect) and was immediately charged with supporting the in-place data integration (ETL) processes for the then un-released DW/BI system. I began my work by documenting (some functional, bust mostly technical) the entire system. In a few weeks I had uncovered many issues and offered a long list of improvements to the team. Many items on my list were directly related to many of the ETL Subsystems that I’ve been discussing. Taking these to heart, we began the long process of refactoring, renaming, and redeveloping some of the processes that were in place. The focus was on future maintainability and performance. I’ve chronicled some of these in a few of my recent posts.

In addition to this, I slowly took over the project management duties and have now been managing the development activities for the entire team. By mid-summer, and due in part to several converging circumstances, I was hired as a full time member of the quant team. Now, my stake in this project’s success is even greater!

I expect to be just as busy over the next two weeks as the final touches are added to this release of the project. I’ve also started to compile a list of “lessons learned”, most of which will be relevant-enough to share with you all. I think that some of these might surprise you!

This week I’ll continue my posts on the ETL Subsystems. I’ll also likely start working on another article for Advisor (topic yet unknown). Lastly, planning for the next release of our DW/BI system is already underway, with development starting as early as November!

PS: If you ever find yourself feeling a bit overwhelmed or too engaged in your work, I highly recommend re-reading Walden (ahh, come on — everyone’s read Walden, right?!?). It has always managed to level me, reminding me how important it is to just sit back and relax once in a while.

It will often become necessary for you to move data from the data warehouse to other applications, databases, and servers. This is a little different then scaling out your data warehouse to support multiple servers (an integration server, relational database server, reporting server(s), analytic server(s), etc.); typically, distributing data in this way is done via replication techniques. But this subsystem addresses another important need.

I usually refer to this need as “Data Provisioning“. Provisioning is a term borrowed from the telecommunications industry meant to describe the process of providing various telecom services to a customer. A data provision is simply a service that provides data to some client. The client can be an Excel sheet, XML file, application, operational system, or perhaps a master (MDM) database.

This subsystem is not the only place that you’ll be provisioning data to various client applications. BI Applications often allow various extracts and downloads that can be made available to the same list of clients. This usually comes by way of a downloadable report or extract. However, the data propagation subsystem provides a few additional benefits above what BI Applications can usually deliver. First, as an ETL process, you can utilize the full power of the ETL system to perform additional transformations on the data before delivery. Second, propagating data can be part of the ETL pipeline so that provisions can be made available as soon as the dimensional models are loaded (highly desirable if the provision is meant to update an operational or master data system).

Consider though that as an ETL function, these data provisions are best suited for well-defined, almost-never-changing requests for data. You would not use this subsystem to provide data in an ad hoc way to an analyst! You would, however, use this subsystem to provide a daily extract to your risk department for compliance reasons.

SQL Server 2005 Integration Services (SSIS)

The only challenge that you’re bound to face is providing the data in the various formats required by the client. I’ve found that Excel is particularly tricky to work with and as a consequence, I usually push my clients to accept CSV or Tab delimited files instead!

Hand Coding with Visual FoxPro (VFP9)

You’ll have no problems propagating data with FoxPro. In many cases, you can use SQL to get the data you need and simply write it to a table or file on disk. The COPY TO command supports different formats and it isn’t difficult to create outputs in other formats such as XML. Using the same techniques you would employ to connect to various servers (using ODBC drivers for example), you can push data into almost any environment. This is a strength of FoxPro that sets it apart from other ETL scripting languages.

From Here

There’s no real magic to propagating data. It’s an ETL function with the typical extract, transform, and load operations. You can treat it as part of your current ETL system; perhaps running provisions after your facts are loaded. Alternatively, provisions can be run on a schedule independent of normal ETL processing or can be run on demand as needed.

…

This post wraps up the “Data Delivery” function in the ETL environment. In my next post, I’ll move on to “ETL Management” functions including scheduling, backup, security and metadata.

One of the reasons I picked up this title was because I’ve been itching to redesign Tod means Fox. I use WordPress (and like it very much) and have all the tools (and some questionable skills) for a redesign. But don’t worry, any redesign will have the same glorious orange that helps remind me of FoxPro, Firefox, ING, Packt, and Holland! But honestly, It’s been a while and I’m restless. I also want a redesign to be done entirely on my new iMac. If I do it quick enough, it will be my first real work performed on the new machine.

It turns out though that the book wasn’t very helpful. I’m sure it will sit on my shelf until some day in the distant future when its time to “clean up” the office. I’ll likely be on revision n of Tod means Fox, and who knows where WordPress will be. But I suppose that’s the nature of buying technology books.

Very few tech books stand the test of time. Those that do are usually pure reference and theory books and not how-to books.

I keep reference books (like the PHP Cookbook, The Complete Reference: MySQL, and of course the Hacker’s Guide to Visual FoxPro) at arms length. These books I go to often. Then there are the theory books. These are books I keep further away but access often — and usually not in front of the computer. Books like Kimball’s The Data Warehouse Toolkit and Zeldman’s Designing with Web Standards fit into this category.

OK, enough babbling. I’d be interested to hear your thoughts on a redesign, WordPress, time-tested tech books, or the fabulous color orange.

Specifically, these FoxPro versions as installed on Microsoft Windows 2000 Service Pack 4:

• Microsoft Visual FoxPro 8.0 Service Pack 1
• Microsoft Visual FoxPro 9.0 Service Pack 1
• Microsoft Visual FoxPro 9.0 Service Pack 2

For SQL Server:

• SQL 2000 Reporting Services Service Pack 2
• SQL Server 2005 Service Pack 2
• SQL Server 2005 x64 Edition Service Pack 2
• SQL Server 2005 for Itanium-based Systems Service Pack 2

Be sure to check out the full report when issued on September 9th if this concerns you (.NET, MS Office, and IE users, you had better have a look too!).

Security Bulletin website

In my previous post, I discussed OLAP, the Analytics Server, and Cubes as a primer to this post. If you’re new to this topic, please have a look there for more details. This post is specific to building cubes from the ETL environment.

Cube building is simply the process of populating the analytical server database with data from the relational database. The relational database can contain relational or dimensional models.

You will want to re-build your analytical (OLAP) database after loading data into your dimensional model. This will keep your analytical data stored in the analytical database in sync with your dimensional models stored in the relational database. From the ETL point-of-view, the basic process works like this:

1. Extract data from disparate sources for a business process
2. Cleanse and conform the data
3. Load Dimensions
4. Load Facts, rebuilding any aggregate fact tables
5. Build cubes and other analytical server objects (like aggregates and dimensions)

These steps are repeated for each atomic business process dimensional model in the data warehouse. The first four are familiar, as I have discussed them at length. The last one should seem natural: You need to devise a way to alert the analytical database that the data warehouse has just been updated. The best place to do this is directly in the ETL system. How you handle the rebuild process is dependent upon how the OLAP vendor’s database works (I offer some suggestions below).

One thing to consider before building cube-updating code in your ETL workflow is latency. Latency, in this context, describes the time delay between the moment the atomic business process dimensional model is up-to-date, and the moment the related cube is up-to-date. It might seem natural to keep this in sync as close as possible. Certainly placing the code to update the cubes right after processing the fact tables in the ETL system is the way to go. But check your requirements. You may only need to have data refreshed once a day, in the early mornings for example (whereas your ETL system might run every 4 hours). You may also have a research environment where OLAP data need only be one week fresh. You can save yourself a lot of processing time by getting these requirements right.

To accommodate this and other scenarios, I highly recommend putting any cube-building or OLAP-related update code in a separate ETL package or module. As with fact and dimension processing, a master package or program (depending on your tool) will “call” the child packages or modules. You could embed some logic in the final step to only process cubes after the morning ETL run, for example.

I do not recommend processing OLAP cubes on a fixed schedule unless you can predict with certainty when your dimensional models will be ready. Normally, it makes most sense to kick off this process as soon as the facts have been loaded.

SQL Server 2005 Integration Services (SSIS)

If you’re using SSIS to update your cubes, then you are likely using SQL Server Analysis Services as your analytical database. If not, and you use SSIS, I’d like to know what you’re using for your analytic server and how you’re building your cubes!

SSIS comes with a powerful task that handles cube-building for you. It’s called “Analysis Services Processing Task” and can do the following: (a) connect to a SSAS database through the Analysis Services connection; (b) specify various objects in the database to update (measure groups, partitions, dimensions, cubes, mining models, mining structures, and databases); (c) define batch-mode or sequence mode to process objects together or one after another; (d) perform an impact analysis on objects associated with the cube; and (e) specify whether dependent and affected objects are also processed.

The Analysis Services Execute DDL Task also allows you to work with cubes by giving you the ability to run DDL statements, as defined by the Analysis Services Scripting Language (ASSL). These statements are included in an XMLA command (more on XMLA below). Although this task can’t process a cube, it can manipulate it. For details on the ASSL language, check out the Analysis Services Scripting Language Reference.

Hand Coding with Visual FoxPro (VFP9)

FoxPro, like all programming languages, just isn’t an analytical database engine so there is no native cube or cube builder. So a lot here depends on what type of technology you are using to handle your cubes.

Your first stop should be to check with the vendor to see if there is an api or technique available to update and rebuild cubes. Even if this functionality exists, it might not be the best option.

For your second stop, look into running XML for Analysis (XMLA) scripts against the database. XMLA is a SOAP-based protocol for interacting with OLAP databases. Through web services, you simply issue commands (like “process”) and receive responses. For more information on XMLA, please check out the XML for Analysis website.

From Here

It is normal for an ETL developer to be involved in generating cubes for analysis services, so this is a skill you should try to foster. Understanding dimensional modeling will give you a huge head start.

Next post (I’m getting closer and closer to the magic 34 — I just hope Ralph and the gang don’t decide to renumber them on me or something), I’ll go through ETL Subsystem 21: Data Propagation Manager!

It used to be that I could code in my favorite Phish t-shirt wearing sandals. I had a key instead of a badge, and lunch usually meant a few greasy pizzas or clam cakes. In those days, my attire only meant something if there was an off-site or if clients were coming to visit “the shop” (which was a tiny building several miles from the heart of the big city). I could easily bounce back and forth between long and short hair and between full beard and cleaned-shaved. Ahh… those were the days.

Now I work in a major international city for a rather large bank. I code in a suit when I’m not in meetings, wear nice shoes, carry a badge, and eat salads and yogurt for lunch. *sigh*

To be fair, I enjoy the new challenges and the big city. And if wearing a suit on occasion is a consequence, I can live with it. So while a suit is not fully mandatory, I still wear one at times. Here’s why:

1. It easily puts me in line with the dress code
2. Dressing is simpler in the morning (although sometimes it takes a couple of tries to get the perfect knot in my tie)
3. My wife tells me I look great
4. Dressing down on Friday never felt so good
5. I look more important than I am
6. I feel more important than I am
7. My jacket flaps behind me in the wind when I ride my bike to the train station, which makes me feel like a super hero with a cape

Other than those fantastic reasons, wearing a suit is a real drag.

(I do admit, there is something rather Monty Pythonish about wearing a suit on a bike. I bet I look pretty silly to the folks driving past me. But riding my bike gives me more than 30 minutes a day of much-needed exercise, and on top of that, the price of gas here in Europe would blow your mind!)

The OLAP Story

OLAP stands for Online Analytical Processing. It is a terrible name to describe multidimensional analytical databases. Instead of a more meaningful name, like FASMI (Fast Analysis of Shared Multidimensional Information) proposed by Nigel Pendse of “OLAP Report” we seem to be stuck with OLAP. With great pain, I will continue to discuss analytical databases using the OLAP terminology.

Basically, an OLAP (*sigh*) database has certain characteristics that set it apart from relational databases. And, coincidentally, Pendse included all characteristics in the FASMI acronym. Check out the OLAP Report for more details. If you’re just getting into business analytics, or are in the process of evaluating analytical database software – you must visit this site!

The Smartness Factor

I like to think of the relational database as “dumb” and the analytic database as “smart”. Let me explain:

The primary focus of Analytic servers is to get the (often pre-calculated) data out of the database as quickly as possible, allowing the user to zoom in and out along different hierarchies. Contrast this with the Codd’s relational model which seeks to eliminate data anomalies at transaction time through normalization: data retrieval is slow and often complex. With this in mind, you can get a feel for how a database engine can store, catalog, and retrieve data differently. (In fact, this is what makes dimensional modeling in general so favorable for querying and analytics – it is not bound by the restrictiveness imposed by normalization. I’d also like to note that as with dimensional models, OLAP databases are multidimensional.)

While the relational database plays a very important role, so does the multidimensional (OLAP) database. It’s simply built differently to service a different need.

Primarily, analytic servers can manage aggregates, aggregate navigation, and complex calculations and summaries across hierarchies at blazing speeds. These skills were borrowed from both navigational database designs (think of the DOM or a Wiki) and hierarchal designs (trees with leaves and branches). A lot of this has to do with how the data is stored.

Storage

Analytics servers offer a different type of storage. MOLAP, or Multidimensional OLAP, can be much more efficient than relational engines. Some tools (like SQL Server Analysis Services) allow you to store your analytical databases in a relational way (ROLAP) or using a hybrid approach (HOLAP). Personally, I see no benefit at all with ROLAP, aside from real-time systems where using the relational database is a must because of the latency involved with updating the OLAP database. (Not to digress too much here, but even real-time environments can use MOLAP storage by using partitions and caches correctly.)

MOLAP is more natural and faster. This may differ depending on your tool choice, but I beg someone to tell me differently. For a detailed discussion on how MOLAP and ROLAP engines store data, you can try this resource “Analysis Services 2005 Performance Guide” (which defines the way Analysis Services does it).

HOLAP storage is an interesting option and could actually perform well –even better than MOLAP in some instances — with less disk usage. But a lot of this depends on how many aggregates are defined and how often the system needs to query the relational database. It might be a good consideration, but if you’re not sure and you have plenty of disk space (disk space is cheap), then go with MOLAP.

The smartness factor² (the MDX language)

I like to think of SQL as “dumb” and MDX as “smart”. Let me explain:

The MultiDimensional eXpression (MDX) language was created about 10 years ago by Microsoft to work specifically with multidimensional data stored in analytical servers. This OLAP-specific language improves upon the SQL syntax, removing much of the bulkiness associated with the language. MDX is an elegant and highly relevant partner to the analytical database. While you could get out what you need using SQL, most every SQL statement will be a challenging one. It’s like cutting down a tree with a hand saw as opposed to using a chain saw. MDX is painless and intuitive. Concepts such as dimensions, hierarchies, and sets are built into the syntax. For more on MDX and the mdXML standard, please visit XML For Analysis.

Central to the MDX language is the cube concept, which deserves a proper introduction:

Cubes

OLAP data is stored in structures called cubes. As you know, a cube is a 3-dimensional solid that, given a point on three of its faces which form a vertex, can take you to a precise point somewhere within the cube itself. This point represents the aggregate metric you want to view. The faces represent the different dimensions (like Product, Customer, and Time) that are used to find the point. The dimensions are further broken down by its hierarchies on each face.

Cubes are sometimes — and more aptly — called hypercubes (or a tesseract or “n-cube”) because analysis often makes it necessary to examine more than three points. As soon as you look at more than three faces, you need higher dimensionality.

I admit, when I first heard about cubes I was a bit intimidated. I felt that they were too complex and advanced. Now, working with cubes is natural. Notice though that the concept of a cube is almost the same thing as a star schema. The only difference (at least that I can really think of) is that a star schema generally stores atomic data, and barring any usable aggregate fact table, calculations need to be done on the fly. The cube theoretically stores the results of these calculations inside the cube.

Dimensional Modeling

If you read my blog, then you know I advocate dimensional models for proper data warehousing. If you also advocate and use dimensional models, then the concepts discussed so far will fit you well.

What’s great about using Dimensional Models in your relational database is that your OLAP database will almost mirror the design, making the development and deployment of your cubes so much easier. Not only that, but the cubes will likely load faster and be easier to maintain when built off of the dimensional model. Note you can still achieve most of what you can do without an analytics server, just by using a properly constructed set of business process dimensional models. The OLAP database is just smarter, making several tasks (navigating hierarchies, security, and calculations to name a few) easier.

I have, however, managed to place one very important piece of the migration puzzle: Getting FoxPro up and running.

Virtualization

To get Windows running on the Mac, you basically have two options: Dual boot via Boot Camp or use Virtualization. Boot Camp was not a great option for me because I want to work in the Mac and share items between the Windows install and OSX. For virtualization, I seemed to have three choices: VirtualBox by Sun, VMware Fusion, or Parallels Desktop for Mac.

Reviews for Parallels were slightly better overall, and I got a recommendation from a colleague, so I went with it. For less than 100 Euro, this software is simply incredible. Installation was as simple as downloading and running the installer. These were the steps I took:

1. Downloaded the latest version from parallels.com
2. Double-clicked “Install Parallels Desktop”
3. Went through the Wizard, clicking Continue through each step
4. Accepted the terms of the license
5. Selected my Mac’s HD for the install destination
6. After the install finished, I moved the icon to the Dock
7. Started the software and from the Help menu clicked “Activate Product”
8. Entered the activation code that Parallels sent me via email
9. Clicked ‘New’ from the File menu to create a new VM via the OS Installation Assistant
10. Selected “Windows Express Installation”, Next
11. Selected “Windows XP”, Next
12. Entered the XP product key, set some basic settings, and gave the VM a name
13. Chose to allocate more resourced to the VM than to Mac OSX
14. Clicked Finish

Next, I installed Windows XP Professional using the new VM window created by Parallels. This was the fastest XP install I’ve ever done. Not only that, but the entire process was the same as if I had been on a PC (just in a window — which is a bit ironic). When Windows booted for the first time, and I heard the familiar Windows theme, I felt a sense of cool calm overcome me. This was going to work.

Ejecting the CD

During install, I had to insert my Windows 2000 disk (I have an Upgrade copy of XP Pro). I think it took me about 30 minutes to figure out how to eject the XP Pro CD from the Mac! Here’s how I did it:

1. Right-click on the CD icon in the footer of the Parallels Desktop window
2. Choose ‘Disconnect’
3. The CD drive will re-appear in OSX’s Disk Utility application, where it can be ejected.
4. Right-click on the CD icon in the footer of the Parallels Desktop window to reconnect when ready

When the VM is running, it ‘owns’ the CD drive. Because I was installing XP Pro, I didn’t have any obvious way to eject the disk. There’s no button on the Mac and I wasn’t yet ready to straighten out a paper clip! The above steps are a bit of a hassle, but certainly doable.

Two-times the Charm

Of course, I had to do it all twice (my fault). I have an XP Pro version 2002 CD without any Service Packs. After Windows installed, I went straight for Windows update and let it install SP2. It failed with some read errors and then Windows would no longer boot. So, I deleted the VM and started over. No biggie. This was easy. This time, I installed SP1 and then went straight for SP3. That worked and now I’m golden. After getting all the other updates, I was ready for the Fox.

Installing Visual FoxPro v9

Ok, so there was nothing to installing FoxPro. I literally put the CD in, installed the prerequisites, and then VFP. No surprises, no issues, no problems. I ran some basic checks and so far everything seems to run fine. I’ve yet to do any development but if I have any issues, I’ll be sure to report them.

I recall at FoxForward 2007 seeing a few Macs. If you’re using VFP on your Mac — let me know about it. I’d like to know how you’ve done it (Boot Camp? VM Ware?) and if you’ve encountered any issues.

Lord Voldemort

Lastly, I’ve named my XP Pro Virtual Machine “Lord Voldemort” of Harry Potter fame. My entire home network is named after various Harry Potter characters. I named my iMac Dumbledore.

This seems totally appropriate.