But just to be clear: The mess refers to a lack functional documentation, little to no experience of the team (basically, everyone but you) on data warehousing/data integration, and therefore no focus on implementing best-practices in the initial design. Sadly, this is indicative of many projects. On top of that, we’ve been stuck in the mud with the release (still).

The consequence of missing or limited data profiling (we had some pretty good talks about the data profiling issue in particular), no functional requirements, and a more-or-less ad hoc approach to managing the development processes is that oftentimes what works isn’t very optimized, easy to debug, or easy to repeat (which is the task I have in front of me).

I had the benefit of stepping in after all this, and saw several patterns in the data flow that I’ve been able to shake out and turn into some optimized C#. The code is rather simple — although I won’t post the actual implementation, I’ll be posting most of the foundation. Actually, it would have been great for you to still be here for this. I could have used your knowledge and expertise.

So my decision to go ahead with this was based on the following: ease of future maintenance, ease of future development, and performance improvements. I’ve achieved that at the expense of making it more difficult for business users to poke in and see. To compensate, the documentation for the new component will be kept up to date and easy to follow.

Quiet curious on how you wrote the custom transformation, with all the lookups needed to find the various keys based on the different combinations.

Aim was to clearly show the business logic/flow in the package.
OK it looks messy, but at least the test manager was able to scan through the matching process and understand which matching types were needed.
When you put all of this in a custom transformation you make it more complex for a non programmer to understand what is happening in the ETL flow.
I think you have to consider what is easiest, scan through 2 X 100 lines of code or the flow picture…. (as you know a pictures says more than a thousand words).
But you have a point in doing it like that, if you improved performance by 600%.
Anyway, Tod it was nice working with you on the project….
see you.

Ok, ok. So I wrote a custom transformation “component”! Now that I’ve re-read my post, calling it a “task” now gets under my skin a bit. I may just have to edit it….

Tasks are elements found in Control Flow, but you have written a Data Flow “component”, specifically a transformation.

Writing custom tasks is even easier than components so write those too!