Oct 15
Quick TmF update: Over the past 6 weeks, I’ve been in the process of moving to Holland. While it can be bad blog-form to make excuses for not posting in a while, I thought emigrating from Belgium to The Netherlands would be a good enough excuse. After another week or two, I expect to be fully settled and back on track.
In the 3 years that TmF has been online, I’ve never gone this long without posting. Feels weird.
There’s been a lot of talk lately about how the US economy is crawling out of recession. You may have heard terms like “bottom” and “trough”, seen graphs of GDP growth, and read articles referencing NBER.
You may be highly skeptical and unwilling to believe that the latest recession is a thing of the past; after all, you likely know someone still looking for a job and you likely know someone else who recently lost a home. Or, you may be very optimistic, and knowing how the business cycle revolves, you are beginning to invest and think about growth. Understanding the business cycle will help you make better decisions — individually and in business.
Off the bat, I should state that economics is not an exact science (but you likely knew that already). And while there are a few axioms (supply/demand, money supply) and indicators (GDP, unemployment) that can help paint a realistic picture, most of what you see and read is based on analyst insight and experience, backed up with historical data and predictive models. There are faults all along the way. When you consider the complexity of the global economy, it should be clear that economics is more of an art than a science. The Business Cycle, though, seems to be something you can count on.
Plainly put, the Business Cycle represents 4 phases of aggregate economic movement, ranging from periods of high growth to recession and back again. John Maynard Keynes (d. 1946) referred to these cycles as “waves of optimism and pessimism”, or to put it another way, waves of expansion and contraction. These phases were first written about more than 50 years ago by Arthur Burns in his book “Measuring Business Cycles”.
| Peak YYYY-MM | Recession Period | Trough YYYY-MM | Recovery Period |
|---|---|---|---|
| 1980-01 | 6 | 1980-07 | 12 |
| 1981-07 | 16 | 1982-11 | 93 |
| 1990-07 | 8 | 1991-03 | 120 |
| 2001-03 | 8 | 2001-11 | 73 |
| 2007-12 | ? | ? | ? |
Data Source: NBER
Notice that peaks and troughs are represented by month and year, while the other two phases are measured over a number of months. You’ll also notice that recession and recovery periods can vary greatly.
In the US, the task is managed by the National Bureau of Economic Research (NBER). NBER is a private, nonprofit, and nonpartisan research organization (stocked with Nobel Prize winning economists). They work on many economics projects and work closely with businesses and universities. They self-proclaim their dedication to promote “a greater understanding of how the economy works”.
For example, NBER most recently concluded that “the last [US economic] expansion ended in December 2007″. We know that after such expansion, according to the Business Cycle, will come a period of recession, followed by a bottom, leading to a new period of expansion.
In short, nope. Milton Friedman, to name one prominent example, believed that the economy fluctuates rather than cycles. The new classical framework states that the economy is much more flexible than that which is implied by the business cycle framework.
There’s also an issue of market equilibrium. Having a somewhat predictable business cycle implies that the markets will be out of sync quite a lot — allowing some speculators and investors to take advantage of price differences at different phases of the cycle (this is called arbitrage, take a look into Rational Expectations Theory as well). Other differing methodologies include the credit/debt cycle, political cycles, and Marxian cycles.
Even against dissenter argument and alternate viewpoints, the Business Cycle framework still works and is easily observable. Economists at NBER continue to assign dates to peaks and troughs. Individuals, businesses and organizations still base many of their purchasing and hiring decisions on what phase we’re currently in. This likely won’t change any time soon.
The only problem with relying on NBER is that they lag reality. For example, we have very likely reached the bottom of the current cycle and are now in a phase of recovery. NBER might make it official at some point this year, maybe next year. Investors waiting for official announcements will find that they’re missing the boat.
Data warehousing is a complex operation. From start to finish (if there is a finish), project teams are faced with many challenges. In all phases of the lifecycle, there are opportunities for derailment. The best way to mitigate potential issues and stay on time and within budget is to carefully define and manage scope. Managing scope can be an ongoing struggle (especially if requirements are not clearly defined or justified). While this is really a PM101-type of topic, I feel there are some fine points in a DW/BI environment that are not mentioned enough.
Consider the following:
I won’t get into a deep PM discussion here, but it is important to point out that data warehousing (or business intelligence, master data management, etc.) initiatives should be thought of as programs and not projects. This mindset will help in scoping.
A program (which might also be called a “project portfolio” in some circles) is basically just a set of related projects. With a program, the emphasis is on organizing, prioritizing, and allocating resources to the right projects. Program scope is more strategic, and answers long-term questions about what type of value the organization hopes to achieve from the initiative.
A project, on the other hand, is much more specific — with a set number of deliverables and goals that have a high immediate impact. The scope at the project level is therefore more tactical in nature: high impact, fast delivery. Be aware that some projects may never be given the green light (for example, if there is a low business impact or if there is a low feasibility rating because of data source or data quality complications).
What I find odd is that organizations still choose to tackle immense data warehousing initiatives in one or two shots, trying to deliver everything at once over a period of 18 or more months. This is the wrong approach (here’s why). Break this large initiative into individual projects and try to deliver functionality every 6 to 8 weeks.
The best way to break down data warehousing programs into high-impact projects is along business process lines. A business process, as defined here, is:
The complete response that a business makes to an event. A business process entails the execution of a sequence of one or more process steps. It has a clearly defined deliverable or outcome. A Business Process is defined by the business event that triggers the process, the inputs and outputs, all the operational steps required to produce the output, the sequential relationship between the process steps, the business decisions that are part of the event response, and the flow of material and/or information between process steps.
Some example of the above: inventory tracking, Internet sales, retail sales, marketing, tax assessment, tax collection, pitching, batting.
In any data warehousing environment, you can expect to have several business processes to model. Each business process you tackle will have elements touching upon different aspects of the data warehouse, including infrastructure, middleware, data modeling, ETL, business logic development, presentation elements, and so on. If you scope each project to the business process, you can deliver complete solutions in the shortest amount of time. (It should be obvious that the very first business process you implement will take the longest, as the team works out the core infrastructure. Most of this infrastructure will be reused by other business processes.)
Do not fall into the trap of scoping to a data source. Scoping to a data source is almost guaranteed to deliver mediocre outcomes. These projects typically include many unfinished or inadequate business processes all delivered at once some time in the distant future and long after the excitement over the initiative has subsided.
While it is true that only one or two data sources might exist in some organizations, it is not true that inventory, customers, sales, procurement, shipping, and other business processes need to be taken on at once. Create a single project for each business process, prioritize based on impact and feasibility, and then badabing badaboom, you deliver. Next.
Along the same lines, do not adjust your scope if the data source is unavailable, uncooperative, or lacking in quality. Instead, bring the fight to the data source (here is where a good, preferable C-Leveled, business sponsor can come in handy) and set things right. This is obviously a project risk, and also an organizational risk. If you are having problems extracting inventory data then maybe its time to put down your data warehousing gloves and get a new inventory system.
Scoping the data warehouse is a difficult problem. Troubles start early on with the initial idea, it moves on through requirement gathering, and finally into the development phase of the lifecycle. There is not a lot of good advice in this area for data warehousing (if you happen to know of a good source, please send me a link or title). But I do find that if you work towards business processes, think in terms of programs and projects, and avoid the data source trap, scoping decisions will settle into the real needs of the business.
Hiring a “data integration expert” or consultant for your next, greatest, data warehousing project? Don’t take it lightly. ETL personnel are critical to the success or failure of your project.
The following are what I deem to be essential technology-related aspects, or faces, of a good ETL developer and/or architect (herein referred to as an ETLer for lack of creativity). While you need to consider business and industry knowledge, personality, and experience in your team-building process, you should start by checking off the following on your interview sheet:
Programming must come natural to an ETLer. Objects, logical constructs, expression construction, program flow, and the like, must be well understood. The truth is that no matter how much your vendor proclaims that their tool does it all, chances are excellent that some hand coding will be required. On top of that, ETL tools work a lot like procedural programs. Technologists are very good at putting their right foot forward, and will generally think of things to make the ETL flow perform better. They also think about logging, auditing, and exception handling; all important.
But a solid programming background is not enough. Knowledge of Data Integration theory and best practices are equally important. While I believe in and use Kimball’s methodologies for integrating data into a dimensional data warehouse, other methodologies exist that may be more suitable to your business and integration needs. Following a proven methodology, with slight modifications to suit your environment will get you further, faster. Having little or no theory behind what you’re doing gets you somewhere, slower. Identify your methodology, and then find someone who understands it.
Knowing the ins and outs of your ETL tool (SSIS, OWB, Datastage, Talend Open Studio, etc.) is essential. I would venture to guess that a solid programmer who has a great understanding of ETL theory will be able to get by using most tools with little learning curve. What I worry about (and you should too) are the nuances in the tooling that can stump even the best. These nuances (SSIS, my tool of *ehem* choice — sorry, I needed to clear my throat, has many of these nuances) can cost you many project hours and force rewrites if blocking issues are encountered. Tool knowledge is also essential to know when it is appropriate to forgo the tool because of I/O issues, or because hierarchical data is better handled elsewhere, or because business logic is best not bundled within a data flow.
While junior members of your data integration team can be one or two-faced (that came out funny), senior members and architects must have more meat on the bone.
I suppose this is why good ETLers are difficult to come by. The ETLer needs to have a healthy mix of programming talent, an approach discipline, and tool knowledge. Trained DBAs and software developers might have a lot to offer, as might a troop of certified tool jocks and method junkies, but to get your project in on time and within budget, don’t settle.
For several years I’ve been following the Business-IT alignment movement. Alignment is necessary for fostering innovation and realizing greater profits as IT resources are used to their fullest to satisfy business goals and objectives.
But many organizations still struggle to make the connection.
Some suggest varying 3-, 5-, and 7-step approaches for better alignment. They typically consist of diving into new organizational structures, or implementing new frameworks and service models. Other sources say that the business is to blame, that IT is misunderstood and underutilized. While finally, some say it is a dead issue altogether.
I’m not convinced this is a dead issue, nor am I convinced that the business is to blame (especially nowadays). I’m also not convinced that 3-, 5-, and 7- step plans will magically work.
A few basic actions, initiated by IT, must happen first.
Business has long ago recognized that better alignment with IT is essential. We can argue about how well they’ve understood IT and how it can best be utilized, but across the board, the recognition is there. IT, after many years of making the case for alignment, seems to be coming up short during this crucial period. I hope that the following actions, from an IT-perspective, can help:
If IT can take action, better alignment can be achieved. This isn’t to say that the business can’t do more, but before the business can make IT a full partner, understanding, participation, and contributions from IT are a must.
This website uses valid XHTML, CSS2.1, and RSS markup because I belive in Web Standards.
Follow me on Twitter
![[Valid RSS]](http://blog.todmeansfox.com/wp-content/icons/valid-rss.png "Validate my RSS feed"){kind=icon}
![[Valid CSS]](http://jigsaw.w3.org/css-validator/images/vcss "Validate my CSS")