Sorry about the confusion. I’ll try to make a better distinction between data profiling and data quality.

Think of data profiling as a precursor to everything else; an introduction. Before you design for or use any data in any of the disparate systems in your enterprise, you need to understand it completely. Data profiling is that process. You will build a ton of useful metadata that you can rely on in later phases of ETL and DW/BI development. Data profiling, consequently, is also a precursor to data quality management.

If you think of data profiling as part of data quality, then you may be inclined to skip a thorough data profile at the start of a new project. The reasoning might be: “I’ll learn the source system while developing the ETL and mappings, and I’ll dig even deeper once the business analysts tell me what to look for regarding potential quality issues. I’ll profile as I go.”

In my experience, this is a death knoll. Profile first and thoroughly. When you move into data quality-related activities (which includes Extract and Transform functions of ETL, as well as complete data quality and monitoring systems you might develop as part of your DW/BI layer), you will have a complete catalog of metadata compiled during your initial profiling to turn to.

Once I’ve been introduced to the data, I am free to test the quality of the relationship. During data quality screenings, I may conduct additional profile-like activities and compile more statistics. I wouldn’t call this a new “data profile”, but rather, simply a part of my data quality research.

Another difference is specificity. Data profiles are broad and should touch upon each and every element you intend to work with. Data quality management tends to be more specific, as you build rules and logic to question the integrity of particular attributes and relationships in the data, oftentimes crossing multiple business processes and systems.

I’d be interested to hear your additional thoughts on this!

-Tod

When Tod says data profiling is separate from data quality auditing, I think it seems to be the case.

Because data profiling is the process of examining the data available in existing data sources (e.g. databases, files…) and collecting statistics. Data quality entails more than helping companies get correct data into their information systems; it also means getting rid of bad, corrupted, or duplicate data.

Of course they are related, but are not complimentary IMHO.

Hope this helped!

thanks very much.
your answer is very helpful.
but one thing let me confused that you said data profiling is separate from data quality auditing/management.
do you agree that data profiling is the first and important phase of enterprise data improvement.
what do you think about the relationship between data profiling and data quality.
how do you define the data quality management. in my opinion, data quality management of a DW/BI project includes data profiling, correcting, standadizing, monitoring.

Data profiling is separate from data quality auditing/management. Profiling is a design and development tool (and process!) that will allow you to examine the content, context, and quality of data coming into your data warehouse. The results of a good data profile will tell you how to build your staging area, your dimensional models, and your data quality procedures.

Currently, I store all data profiling results in relational tables as metadata. You can read more here: http://blog.todmeansfox.com/2007/07/03/data-profiling/ and here: http://blog.todmeansfox.com/2007/07/10/getting-started-with-data-profiling/.

Regarding your business rules question: Data profiling is an exploratory process helping you to explain the data you are receiving. In my experience, there is no room or reason to include business logic in a data profile. You would, however, build these things into a data quality system — preferably one that runs in your staging area or one that executes on data as it is loaded into your data warehouse.

Hope this helps!

I have tried to documented my results using MS word, but i found that to much items need recorded, Even some explanation of the bad data that given by BA.

Do u agree that data profiling will sometimes refered to business rules, or we only did it merely a techical process.
eg.
two fields named vhcl_type and vhcl_color,
the BA told that :
when vhcl_type=’bus’, the vhcl_color data quality was quite good.and
when vhcl_type=’truck’, there r too many ‘NULL’ in vhcl_color.
how do u consider this problem?

thanks for ur advice.