我的数据框架如下:
+-------------------+-------------+
| Nationality| continent|
+-------------------+-------------+
| Turkmenistan| Asia|
| Azerbaijan| Asia|
| Canada|North America|
| Luxembourg| Europe|
| Gambia| Africa|
我的输出应该如下所示:
Map(Gibraltar -> Europe, Haiti -> North America)
所以,我正在尝试将数据帧转换为
scala.collection.mutable.Map[String, String]()
我正在尝试使用以下代码:
var encoder = Encoders.product[(String, String)]
val countryToContinent = scala.collection.mutable.Map[String, String]()
var mapped = nationalityDF.mapPartitions((it) => {
....
....
countryToContinent.toIterator
})(encoder).toDF("Nationality", "continent").as[(String, String)](encoder)
val map = mapped.rdd.groupByKey.collect.toMap
但结果图有以下输出:
Map(Gibraltar -> CompactBuffer(Europe), Haiti -> CompactBuffer(North America))
如何在没有CompactBuffer的情况下获得哈希映射结果?
让我们创建一些数据:
val df = Seq(
("Turkmenistan", "Asia"),
("Azerbaijan", "Asia"))
.toDF("Country", "Continent")
尝试先映射到元组然后收集到地图中:
df.map{ r => (r.getString(0), r.getString(1))}.collect.toMap
输出:
scala.collection.immutable.Map[String,String] = Map(Turkmenistan -> Asia, Azerbaijan -> Asia)