Cleaning up data quality

What is your vision on data quality?

I am particularly fascinated by the idea that there are insights hidden in a growing mountain of data. In a process similar to the mining and refining of iron ore, for example, the data is purified and processed, ultimately leading to a valuable end product.

The degree of purity of the raw material and the delivery of the right intermediate products are very important here; you don’t make things easier for a metal worker by handing over a container full of small different pieces of metal instead of a rectangular, homogeneous block of steel. I think this is obvious to a lot of people, but when it comes to data, people don’t seem to take it into account as much, with all the consequences that entails.

Ultimately, data quality is important because it is what people downstream depend on to make the right business choices. Ensuring data quality delivers value by reducing uncertainty and facilitating the setting of the right course for an organisation.

In my experience, poor data quality often arises at the start of the process, when data is gathered from different, often external, sources. The data is then initially packaged in a varied set of documents with various templates. Knowledge about the exact definition and nuances of each field usually does not survive this step and is hidden in the brains of the people involved in the process. Because data is often combined in complex ways, poor data quality can quickly spread through a process. An additional problem is that, unlike explicit programming errors, incorrect data cannot be easily spotted.

2. Getting started

How can you automate data cleansing?

There are various ways of cleaning up the data automatically and improving its quality. An example is correcting typos and spelling variations in place names. This can be done by comparing with a dataset of “correct” spellings, based on the processing distance. A second example is that missing data can be imputed, which means that it is filled in afterwards based on a data model. These methods make use of both specialist knowledge and the source data itself. What can you do with the time saved on data cleansing? However, polishing a dataset afterwards can cost a lot of time, knowledge and money. It is estimated that data scientists spend 60%-80% of their time preparing and refining data, something I personally underestimated when I started this work. As a result, organisation-wide improvements in data quality quickly generate value in the form of cost savings, more thorough analysis, clearer visualisation, and more motivated data analysts.

How do you ensure clean data from the ground up?

This is a typical case where “prevention is better than cure”, ensuring data quality is more effective and ultimately cheaper the closer it is to the source. Data must be validated immediately upon import. For this purpose, it is important that an organisation has a standardised data model on which validation can be based. With a standard data model, the data speak the same “language”, as it were, everywhere; this facilitates cooperation and makes it easier to maintain data consistency and quality. In order to form this model, it is important to sit down with stakeholders to create something that everyone supports, which goes hand in hand with making people more aware of the importance of data quality. This is complicated by the fact that the value generated by data is difficult to see at the point where it enters the organisation, while this is precisely the point where quality assurance is most effective.

3. Getting started

A practical first step towards improving data quality is to establish what this actually means in the context of your specific organisation. This requires drawing up clear rules and definitions, which together form the data model. This then forms the basis for mapping data quality within the process, by means of automated tests. The result is a clear framework within which data quality can be measured, discussed and improved.

What can we achieve in three months?
Become a more data-driven organization in three months. Extracting actionable insights from data, leading to better –more informed–decisions.