scala 如何在 Spark 中平面映射嵌套的数据框
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/36784735/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to flatmap a nested Dataframe in Spark
提问by user2230605
I have nested string like as shown below. I want to flat map them to produce unique rows in Spark
我有如下所示的嵌套字符串。我想平面映射它们以在 Spark 中生成唯一的行
My dataframe has
我的数据框有
A,B,"x,y,z",D
I want to convert it to produce output like
我想将其转换为产生输出
A,B,x,D
A,B,y,D
A,B,z,D
How can I do that.
我怎样才能做到这一点。
Basically how can i do flat map and apply any function inside the Dataframe
基本上我如何做平面地图并在数据框内应用任何函数
Thanks
谢谢
回答by zero323
Spark 2.0+
火花 2.0+
Dataset.flatMap:
Dataset.flatMap：
val ds = df.as[(String, String, String, String)]
ds.flatMap {
case (x1, x2, x3, x4) => x3.split(",").map((x1, x2, _, x4))
}.toDF
Spark 1.3+.
火花 1.3+。
Use splitand explodefunctions:
用途split及explode作用：
val df = Seq(("A", "B", "x,y,z", "D")).toDF("x1", "x2", "x3", "x4")
df.withColumn("x3", explode(split($"x3", ",")))
Spark 1.x
火花1.x
DataFrame.explode(deprecated in Spark 2.x)
DataFrame.explode（在 Spark 2.x 中已弃用）
df.explode($"x3")(_.getAs[String](0).split(",").map(Tuple1(_)))
