😩Uncertainty in data: common bugs
There is a lot of talk about “data preparation” and “data cleaning” these days, but what separates high quality data from low quality data?
Most machine learning systems today use supervised learning. This means that the training data consists of (input, output) pairs, and we want the system to be able to take the input and match it with the output. For example, the input might be an audio clip and the output might be a transcription of a speech. To create such datasets, it is necessary to label them correctly. If there is uncertainty in the labeling of the data, then more data may be needed to achieve high accuracy of the machine learning model.
Data collection and annotation may not be correct for the following reasons:
1. Simple annotation errors. The simplest type of error is misannotation. An annotator, tired of a lot of markup, accidentally puts sample data in the wrong class. Although this is a simple bug, it is quite common and can have a huge negative impact on the performance of the AI system.
2. Inconsistencies in annotation guidelines. There are often subtleties of various kinds in annotating data items. For example, you might imagine reading social media posts and annotating whether they are product reviews. The task seems simple, but if you start to annotate, you can realize that “product” is a rather vague concept. Should digital media, such as podcasts or movies, be considered products? One specialist may say yes, another no, so the accuracy of the AI system can be greatly reduced.
3. Unbalanced data or missing classes. The way data is collected greatly affects the composition of datasets, which in turn can affect the accuracy of models on specific data classes or subsets. In most real world datasets, the number of examples in each category that we want to classify (class balance) can vary greatly. This can lead to reduced accuracy, as well as exacerbating balance problems and skew. For example, Google's AI facial recognition system was notorious for not being able to recognize faces of people of color, which was largely the result of using a dataset with insufficiently varied examples (among many other problems).