Comments on: ETL Subsystem 7: Removing Duplicates http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/ Supporting decisions through sound data management Tue, 23 Aug 2011 03:10:04 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Tod means Fox | 34 Subsystems of ETL Data Integration http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1220 Tod means Fox | 34 Subsystems of ETL Data Integration Tue, 18 Mar 2008 14:03:31 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1220 [...] Removing Duplicates [...] [...] Removing Duplicates [...]

]]> By: Tod McKenna http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1206 Tod McKenna Tue, 11 Mar 2008 11:50:46 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1206 Nice to hear you Garrett. The de-duping process is a rather complex one in general with every solution having some significant downside. Soundex only works for names in the US for example (well, that is how I understand it anyway!). I would guess that de-duping is one of the areas that takes ETL developers the longest to get right -- or close to right. Nice to hear you Garrett. The de-duping process is a rather complex one in general with every solution having some significant downside. Soundex only works for names in the US for example (well, that is how I understand it anyway!).

I would guess that de-duping is one of the areas that takes ETL developers the longest to get right — or close to right.

]]> By: Garrett Fitzgerald http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1195 Garrett Fitzgerald Sun, 09 Mar 2008 07:15:51 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1195 Tod, when I worked in the direct mail industry, we had a program from PostalSoft/FirstLogic/Business Objects that did Match/Consolidate, Merge/Purge, or whatever other term for deduping that you want to come up with. :-) It had some pretty elaborate settings, and small tweaks could make great differences in the outcome. After using that, I'm not terribly surprised that the functionality doesn't come natively. :-) Tod, when I worked in the direct mail industry, we had a program from PostalSoft/FirstLogic/Business Objects that did Match/Consolidate, Merge/Purge, or whatever other term for deduping that you want to come up with. :-) It had some pretty elaborate settings, and small tweaks could make great differences in the outcome. After using that, I’m not terribly surprised that the functionality doesn’t come natively. :-)

]]> By: Tod McKenna http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1162 Tod McKenna Sat, 01 Mar 2008 07:36:03 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1162 Hi Josh, Please let me know when ready! Deduplication of names is particularly tricky. At least with addresses, you have a finite number of known 'Good ' values. Lists can be obtained from the USPS for example. Names, on the other hand are too variable. Hi Josh,

Please let me know when ready!

Deduplication of names is particularly tricky. At least with addresses, you have a finite number of known ‘Good ‘ values. Lists can be obtained from the USPS for example. Names, on the other hand are too variable.

]]> By: Blog about Process Checkers » Blog Archive » Fast Friday links http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1160 Blog about Process Checkers » Blog Archive » Fast Friday links Fri, 29 Feb 2008 20:28:53 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1160 [...] http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/The process of de-duping goes well beyond removing or identifying pure duplicates during data integration. This process actually seeks to remove redundant, misspelled, or otherwise ‘almost matches’ from the data stream, selecting the … [...] [...] http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/The process of de-duping goes well beyond removing or identifying pure duplicates during data integration. This process actually seeks to remove redundant, misspelled, or otherwise ‘almost matches’ from the data stream, selecting the … [...]

]]> By: josh http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1159 josh Fri, 29 Feb 2008 20:27:43 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1159 Years ago I worked for Social Services. Our biggest problem was reports of people that needed assistance (ok; not our biggest problem, but it was mine!) and spelling of names was questionable. Even clients (that's code for welfare folks) sometimes didn't know how to spell their name (either due to drugs/alcohol abuse or brain damage...) Anyway; I attended a coference in New York -- it's outcome was a long document; which I filtered out to be a soundex replacement like function specifically for matching names. For instance, my last name is Assing -- but it's pronounced Ausing the nyiis function calls that a match. That is just one example. I used to sell the function's source code; but am planning on making it available for free to anyone that wants it under GPL licensing. -josh Years ago I worked for Social Services. Our biggest problem was reports of people that needed assistance (ok; not our biggest problem, but it was mine!) and spelling of names was questionable. Even clients (that’s code for welfare folks) sometimes didn’t know how to spell their name (either due to drugs/alcohol abuse or brain damage…) Anyway;
I attended a coference in New York — it’s outcome was a long document; which I filtered out to be a soundex replacement like function specifically for matching names.
For instance, my last name is Assing — but it’s pronounced Ausing the nyiis function calls that a match. That is just one example.
I used to sell the function’s source code; but am planning on making it available for free to anyone that wants it under GPL licensing.
-josh

]]> By: I wanna talk about Plan Checkers » Blog Archive » Fast Friday links http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/comment-page-1/#comment-1158 I wanna talk about Plan Checkers » Blog Archive » Fast Friday links Fri, 29 Feb 2008 19:46:10 +0000 http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/#comment-1158 [...] http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/You can read more about this over at Sweet Potato Software, article “Spelling Checker and VFP” as well. As mentioned above, it is best to conduct your searches using atomic data. If the source data is not currently tokenized (did I just … [...] [...] http://blog.todmeansfox.com/2008/02/29/etl-subsystem-7-removing-duplicates/You can read more about this over at Sweet Potato Software, article “Spelling Checker and VFP” as well. As mentioned above, it is best to conduct your searches using atomic data. If the source data is not currently tokenized (did I just … [...]

]]>