Data profiling and data mining may seem similar, but they serve different data management purposes. The two processes must work together to ensure quality data.
In theory, using data should be simple. All data would be easily accessible in an organized database with clearly labeled and consistent types, ready to use for analysis.
In practice, it’s not that simple. An organization can store data across different departments in a variety of formats — often incompatible — and sometimes with no structure or organization at all. Understanding how to use data mining and profiling techniques together can help an organization extract valuable insights from its data.
Data profiling
Data profiling is the process that examines, analyzes, reviews and summarizes the quality of a data set. It identifies where the data is, what it’s about, who has access to it, and if it’s consistent, accurate and complete.
Data catalog, metadata management or BI and analytics platforms typically provide data profiling tools, said Doug Henschen, vice president and principal analyst at Constellation Research.
“[Profiling is] about developing a better understanding of data in preparation for its refinement,” he said. “It would help you better understand which data to choose and possibly combine for data mining analysis.”
Businesses use profiling when designing or coding data transformation and integration processes, said Boris Evelson, vice president and principal analyst at Forrester Research.
Data profiling increases the accuracy and use of the data used for data mining and generally improves the data mining results. Thomas CoughlinLife fellow and president of the IEEE, president of Coughlin Associates
They can also use it to create models and schemas for data warehouses, or to build semantic layers. Semantic layers simplify data use by isolating complex data lake or warehouse structures from business users.
Data profiling can improve the quality of data for any application, said Thomas Coughlin, life fellow and president of the Institute of Electrical and Electronics Engineers (IEEE) and president of the consulting firm Coughlin Associates. For example, it can help companies prepare data for training AI models. It’s also a key first step for data mining.
“Data profiling increases the accuracy and use of the data used for data mining and generally improves the data mining results,” Coughlin said. “Data mining can be used on unprofiled data, but the results may not be as accurate.”
Data mining
Data mining is the process of analyzing structured or unstructured data sets to identify patterns, relationships and correlations. Analytics models use the results to generate insights that enable data-driven decision-making.
“Most of the time, data mining is associated with deriving insights from unmodeled data,” Evelson said.
One of the most frequent use cases of data mining is text mining. Unstructured data types such as emails, contracts and social media posts aren’t ready for analysis by BI or other analytics tools in their raw form.
“First you need to mine text for structures, including entities, topics and sentiment, then you can analyze those extracted or derived structures,” Evelson said.
In a hospital, data mining could predict disease outbreaks or identify the most effective treatments based on historical data patterns using profiled data, said Ani Chaudhuri, CEO at Dasera, a data security and governance company.
Techniques
Data profiling and mining each offer a variety of techniques to get the job done. Organizations must choose which technique best fits their needs.
Data profiling techniques
Several profiling techniques that data teams can use include statistical analysis, data quality assessment and schema discovery. Identifying data inconsistencies or anomalies leads to higher-quality data. Better quality generates more reliable insights and helps companies comply with regulations, said Nick Kramer, vice president of applied solutions at SSA & Company.
Profiling large data sets can be time-consuming and resource intensive. Analyzing sensitive data may raise privacy and security concerns. Analysts must also address any quality problems discovered during data profiling. If left alone, quality problems lead to more issues that waste the time and money spent on the profiling process.
SSA & Company typically uses Python for profiling, Kramer said. It also uses tools to enable non-technical users to profile data, including data quality and purpose-built data wrangling tools.
Companies can start to extract insights from data after profiling is complete.
Data mining techniques
Data teams have a variety of data mining techniques at their disposal to identify relationships in data sets and organize data. Common techniques include anomaly detection, clustering, classification, regression, neural networks, decisions trees and K-Nearest Neighbor. The techniques can reveal customer insights, improve marketing strategies, increase sales, optimize business processes and reduce costs.
“Identifying hidden patterns and relationships helps in making data-driven decisions and gaining a competitive edge,” Kramer said.
For example, segmenting customers based on their behaviors and preferences can help companies deploy targeted marketing strategies.
Data profiling and mining uses
Data profiling and mining work best when used to complement each other. Different industries can apply both processes in different ways to achieve effective results:
Science and technology. Scientific researchers collect large amounts of research notes and data points. Data profiling can help organize the information, while data mining can help scientists make conclusions or predictions, or identify hypotheses for further research.
Fraud detection. Profiling can help companies identify relevant data for fraud detection, such as customer transactional records or employee expense reports. Mining can use that data to identify suspicious behavior through anomaly detection or pattern recognition.
Market analysis. Data profiling and mining techniques are key processes in marketing. For example, profiling can identify and classify sources of relevant social media posts, while mining can discover unmet customer needs to help with product development.
Customer retention. Data profiling can identify sources of key customer data related to customer satisfaction, loyalty and churn. Data mining can segment customers based on history and demographics, predict which customers may be in danger of leaving and suggest intervention strategies.
Maria Korolov has been covering enterprise technology for nearly 20 years and is currently focusing on artificial intelligence and cybersecurity.
