🎯How to remove duplicates in a dataset with Apache Spark?
Use the following framework API methods:
• distinct() - the simplest and most frequently used way to remove identical duplicate strings from the dataframe
• dropDuplicates() - Unlike distinct(), which takes no arguments at all, you can specify a subset of columns in the arguments to dropDuplicates () to remove duplicate records. Therefore, dropDuplicates (Seq <String> colNames) is more suitable when only some of the columns from the original dataset need to be processed.
• reduceByKey() - returns a new RDD - a distributed dataset of key-value pairs (K, V), in which all values ​​for one key are combined into a tuple - the key and the result of the reduce function for all values ​​associated with this key. This method of removing duplicates is limited to the size of a Scala tuple, which contains between 2 and 22 elements. That's why you should not reduceByKey() if Spark RDD keys or values ​​have more than 22 columns.
• collect_set() - a function from the Spark SQL API. It collects and returns a set of unique items. It is not deterministic, since the order of the results depends on the order of the rows, which might change after shuffling, and is not "real" deduplication. Basically, collect_set () is about rolling up records by executing groupBy () and collecting unique values ​​for the column associated with each group.
• write your own window function to get around the size limitation of Scala tuples. For example, split the RDD by columns, sort them, and filter the values ​​you want.