如何像火花数据帧中的SQL那样实现EXISTS条件

Question

我很好奇，我该如何以Spark Dataframe方式实现sql类似于exist子句。

Answer 1

LEFT SEMI JOIN等效于Spark中的EXISTS功能。

val cityDF= Seq(("Delhi","India"),("Kolkata","India"),("Mumbai","India"),("Nairobi","Kenya"),("Colombo","Srilanka")).toDF("City","Country")

val CodeDF= Seq(("011","Delhi"),("022","Mumbai"),("033","Kolkata"),("044","Chennai")).toDF("Code","City")

val finalDF= cityDF.join(CodeDF, cityDF("City") === CodeDF("City"), "left_semi")