Mindmapping Dimensional Models

When presented with a data modeling problem, I start with a conceptual design and then create the logical and physical designs as each concept becomes more mature and stable. This is an iterative process that can take many hours. Using mind mapping software has always given me a head start.

Mind mapping has been around for a long time. It’s a visual technique that you can employ which allows you to diagram ideas based on some central theme. For dimensional modeling, the theme is some event in the business process, while the ideas are the dimensions and dimension hierarchies. Mind maps are quick to make, easy to follow and share, and will allow you to see all interconnected concepts in one place. Software developers and data modelers have been using mind maps for a long time, but its use (as far as I’ve seen) isn’t quite mainstream in the dimensional modeling space.

Orders 300x97 Mindmapping Dimensional Models

Mind mapping an Orders business process dimensional model

When I start to construct a new data model (ER/DM), my first attempt at a design is often a mind map. As I read through requirements and examine business processes and business entities, I start to draw out how they may relate. Traditionally I used a paper and pencil. But recently, I’ve switched to using mind mapping software that I can access and share on all my devices. MinDgo, for example, works on my Mac, iPhone, and iPad. Once I’ve sufficiently covered all key concepts and requirements, I complete the conceptual model in PowerDesigner.

I use mind mapping software for the following reasons:

  1. Quick, easy, and structured way of designing high-level business process dimensional models
  2. Available on all my devices (unlike the heavy case/modeling tools we use), so when inspiration strikes, or when that coffee-machine meeting concludes, I can quickly get the ideas into the design
  3. Very easy to show, explain, and help interpret the models to business and technology colleagues
  4. Organizing different iterations of a design, and interconnecting related designs, is easy using software (try this in your Moleskin)

I also use this technique when I am trying to understand existing data models. For example, if I’m analyzing complex database documentation (from vendors like WorldScope or Charles River), I can get a good feel for how things relate by mind mapping as I go.

Tags: , , , , ,

New Role as Data Architect

stereographic polytope 8cell 150x150 New Role as Data ArchitectI have recently taken the position of Data Architect.

As I wrote on LinkedIn: “As data architect, I am responsible for creating a strategic and international data architecture and vision. This includes implementing data architecture standards and guidelines across the value chain; serving as the lead data architect for all data warehouse, reporting, and analytic (BI/DSS) projects; and playing a lead role in establishing data ownership/stewardship within the organization. ”

This is quite loaded.

As I settle in, I’ll be looking to prioritize our most pressing needs while consuming all that I can about our existing data landscape. This is bigger than the data warehouse, data marts, and BI applications I’ve been involved with over the past year. This is also bigger than the integration project I undertook (with the same company) over the course of two+ years while working on the quantitative investment team. I need to get my head around all aspects of our data, from our CRM system through to our ODS, from our master data management capabilities to our transaction processing. In addition, these systems all talk to each other through a network of messaging, transferring, and batch processing. Our integration and service architecture is maturing quite quickly, and we are constantly improving our alignment across borders, which means more data, higher quality, faster delivery. In many ways, our environment is robust and mature; in other areas, we can do better.

I intend to continue writing about data warehouse topics. But I also expect to write more about my experiences in this new role. I hope you don’t mind!

Tags: , ,

Chaos Theory and the Data Warehouse

Have you ever considered the Data Warehouse as a chaotic system? The work of the Data Warehouse team is never complete: new requirements trickle in every day, and user feedback gets more and more sophisticated as time passes. Chaos Theory can help explain this, and in the end, offer us some insight into how we can better plan Data Warehouse development, deployment, and maintenance.

butterfly effect 150x150 Chaos Theory and the Data WarehouseThe Data Warehouse is a process which forms the center of an information supply supply chain, with several inputs and several outputs. Each input and each output is subject to change based on factors such as vendor upgrades, new interfaces, expanded interfaces, and perhaps most importantly end-user (client) evolution. All of these changes happen continuously. As people use the Data Warehouse, they become more inquisitive. They want their output and analysis rolled up or down in different ways. Predicting (i.e. planning) for Data Warehouse change can be as difficult as predicting (and therefore planning for) the weather. This environment of ever-changing needs fits neatly into the confines of Chaos Theory. But what is chaos in this context? What is Chaos Theory exactly?

From the book “Chaos Theory Tamed”, author Garnett P. Williams writes:

Chaos is sustained and disorderly-looking long-term evolution that satisfies certain mathematical criteria and that occurs in a deterministic non-linear system. Chaos theory is the principles and mathematical operations underlining chaos. (pg 9)

Meteorologist Edward Lorenz in the 1960s determined that even the tiniest differences in an initial measurement can have a huge impact on an outcome. In other words, as his butterfly effect posits, a butterfly flapping its wings in Africa can affect weather patterns in North America. Weather is a system which has a highly sensitive dependence on its initial inputs.

The foundation of the Data Warehouse is only as stable as how you control for the tiniest changes to the inputs into the information structure. As weather, it too has a highly sensitive dependence on inputs. One tiny change to a source system can have almost catastrophic effects on the Data Warehouse.

Finding Order

However, despite the chaos, we should be able to find some order. This is what Lorenz and scientists after him tried to do. The first step in this process is understanding that even seemingly random changes are not always as random as they seem. If we can understand that changes to our Data Warehouse are not random, then we can build a better Data Warehouse.

There are a few things you can do to tame the chaos:

  • Be consistent and systematic. The more predictable you and your Data Warehouse team are, the easier it will be handle change. In other words, control any and all variables that you can.
  • Adopt proven analysis and development methodologies that others have had success with. This is not to say that some level of adaptation to your environment, team skills, and situation are not required, but rather, start off with a good foundation and follow along where it makes sense.
  • Keep the team close. Quality and frequent interaction among the people who make and run the DWH is essential.
  • Stay in the groove like an improvisational jazz band. If your data modelers are not in tune with your decision-support analysts who are not in tune with your DBA, then you can’t expect to handle the challenges of chaos.
  • Feedback and evolution are two very important aspects of Data Warehousing. Keep your ear to the wall and try to anticipate changes before they occur. This takes practice, but (back to the improvisational jazz band analogy) practice makes perfect.
  • Keep in step. In the Data Warehouse world, change is natural and will come in waves. More significantly, if changes cannot be implemented quickly, your clients will lose confidence in your ability to keep up.
  • Think and act quickly. The longer you debate, the longer your client must wait. While they wait, they construct workarounds or look elsewhere. If you’re lucky and they do wait for you, their change may become outdated and no longer relevant; an opportunity might have been missed (and you’ve essentially failed them).
  • Don’t be afraid to be wrong. The consequence of acting quickly is that you might get something wrong. Just be agile enough to respond and deliver new change with urgency.

I’ll post more thoughts on this over the next weeks. I’m particularly interested in how users of the Data Warehouse become more and more sophisticated as they use its tools and applications.

Tags: , , , , , ,

The Client or the Customer?

The terms client and customer are usually used interchangeably. After all, the dictionary basically says they’re the same thing. But why do I feel more warm and fuzzy using the term client to describe my relationship with my, um, clients?

IT/Business Alignment
This is cliché-ville, but I’m going there. Thinking of the business part of your organization as your client, rather than as your customer, might actually help you better align your objectives and goals with them. There’s a difference with the terms and it all comes down to that warm and fuzzy feeling I mentioned a few seconds ago (more tangibly, I’m talking about how the relationship is perceived by both sides).

As technologists, we’re not often known for our people skills (come on, admit it). We like zeros and ones, placing things in nice neat boxes, and Gantting. While I might be generalizing — ok, stereotyping — I think you know what I mean. Essentially, we need all the help we can get. So here’s my take:

A client is someone who uses the professional advice or services of another, while a customer is a person who purchases goods or services from another. The difference might be subtle, but imagine that your relationship is one built on partnership. In this fantasy, the “customer” is not always right (still in cliché-ville) and they know it. They have come to trust your guidance and advice, and turn to you as the expert. They are your client, and you are providing them a valuable service.

How do you see the relationship with each of your business lines? Do you position yourselves as information partners (clients) or as information servants (customer)?

Tags: , , ,

Moving Day

Quick TmF update: Over the past 6 weeks, I’ve been in the process of moving to Holland. While it can be bad blog-form to make excuses for not posting in a while, I thought emigrating from Belgium to The Netherlands would be a good enough excuse. After another week or two, I expect to be fully settled and back on track.

In the 3 years that TmF has been online, I’ve never gone this long without posting. Feels weird.

Tags: ,

Understanding the Business Cycle

There’s been a lot of talk lately about how the US economy is crawling out of recession. You may have heard terms like “bottom” and “trough”, seen graphs of GDP growth, and read articles referencing NBER.

You may be highly skeptical and unwilling to believe that the latest recession is a thing of the past; after all, you likely know someone still looking for a job and you likely know someone else who recently lost a home. Or, you may be very optimistic, and knowing how the business cycle revolves, you are beginning to invest and think about growth. Understanding the business cycle will help you make better decisions — individually and in business.

Off the bat, I should state that economics is not an exact science (but you likely knew that already). And while there are a few axioms (supply/demand, money supply) and indicators (GDP, unemployment) that can help paint a realistic picture, most of what you see and read is based on analyst insight and experience, backed up with historical data and predictive models. There are faults all along the way. When you consider the complexity of the global economy, it should be clear that economics is more of an art than a science. The Business Cycle, though, seems to be something you can count on.

What is the Business Cycle?

Plainly put, the Business Cycle represents 4 phases of aggregate economic movement, ranging from periods of high growth to recession and back again. John Maynard Keynes (d. 1946) referred to these cycles as “waves of optimism and pessimism”, or to put it another way, waves of expansion and contraction. These phases were first written about more than 50 years ago by Arthur Burns in his book “Measuring Business Cycles”.

The four phases

  1. Peak – The highest point of economic output just before a downturn
  2. Recession – When the economy actually shrinks, or contracts
  3. Trough – The “bottom”
  4. Recovery – The economy has stopped shrinking is growing once more

Last 5 US Business Cycles

Peak YYYY-MM Recession Period Trough YYYY-MM Recovery Period
1980-01 6 1980-07 12
1981-07 16 1982-11 93
1990-07 8 1991-03 120
2001-03 8 2001-11 73
2007-12 ? ? ?

Data Source: NBER

Notice that peaks and troughs are represented by month and year, while the other two phases are measured over a number of months. You’ll also notice that recession and recovery periods can vary greatly.

Who determines when each phase begins and ends?

In the US, the task is managed by the National Bureau of Economic Research (NBER). NBER is a private, nonprofit, and nonpartisan research organization (stocked with Nobel Prize winning economists). They work on many economics projects and work closely with businesses and universities. They self-proclaim their dedication to promote “a greater understanding of how the economy works”.

For example, NBER most recently concluded that “the last [US economic] expansion ended in December 2007″. We know that after such expansion, according to the Business Cycle, will come a period of recession, followed by a bottom, leading to a new period of expansion.

Does everyone agree?

In short, nope. Milton Friedman, to name one prominent example, believed that the economy fluctuates rather than cycles. The new classical framework states that the economy is much more flexible than that which is implied by the business cycle framework.

There’s also an issue of market equilibrium. Having a somewhat predictable business cycle implies that the markets will be out of sync quite a lot — allowing some speculators and investors to take advantage of price differences at different phases of the cycle (this is called arbitrage, take a look into Rational Expectations Theory as well). Other differing methodologies include the credit/debt cycle, political cycles, and Marxian cycles.

Final thoughts

Even against dissenter argument and alternate viewpoints, the Business Cycle framework still works and is easily observable. Economists at NBER continue to assign dates to peaks and troughs. Individuals, businesses and organizations still base many of their purchasing and hiring decisions on what phase we’re currently in. This likely won’t change any time soon.

The only problem with relying on NBER is that they lag reality. For example, we have very likely reached the bottom of the current cycle and are now in a phase of recovery. NBER might make it official at some point this year, maybe next year. Investors waiting for official announcements will find that they’re missing the boat.

Tags: , , , , , ,