✈️Math for the Data Scientist: Measuring Distance, Part 3
• The Jaccard index or Intersection over Union is a measure for calculating the similarity and diversity of multiple samples — the size of the intersection divided by the size of their union. In practice, this is the total number of similar objects between sets divided by the total number of objects. For example, if two sets have 1 common entity and only 5 different entities, then the Jaccard index will be 1/5 = 0.2. The main disadvantage of this measure is its dependence on the amount of data: the larger the sample, the larger the index value. The Jaccard index is often used in applications that use binary data. For example, the DL model predicts image segments. In this case, the Jaccard index can be used to calculate how closely the forecast matches reality. This exact measure can be applied to text similarity analysis to measure how often words are selected between documents and to compare sets of multiple patterns.
• The Sørensen-Dice index is very similar to the Jaccard index — it also measures the similarity and diversity of multiple samples. While they are calculated in a similar way, the Sorensen-Deiss index is a little more intuitive because it can be thought of as the percentage of overlap between two sets, which is a value between 0 and 1. Like the Jaccard index, the Sorensen-Deiss index exaggerates the importance of sets in which there are practically no reliable positive results, weighing each element in inverse proportion to the size of its sample. This measure is often used in image segmentation problems and in text similarity analysis.
• Haversine distance is the distance between two points on the sphere, taking into account their longitude and latitude. This is similar to Euclidean distance in that it calculates the shortest line between two points that are on a sphere. This is the main drawback of this measure - ideal spheres do not exist in reality. For example, due to the unevenness of the planet's landscape, calculations may be distorted. Instead, you can use the Vincenty distance, which works with an ellipsoid instead of a sphere. Unsurprisingly, Haversine distance is often used in navigation. For example, to calculate the distance between two countries when flying between them. It makes no sense to apply this measure at short distances, because a small radius of curvature has little effect.