scala Spark：分解结构的数据帧数组并附加 id
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/42354372/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Spark: Explode a dataframe array of structs and append id
提问by Steve
I currently have a dataframe with an id and a column which is an array of structs:
我目前有一个带有 id 和列的数据框，它是一个结构数组：
root
|-- id: integer (nullable = true)
|-- lists: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- text: string (nullable = true)
| | |-- amount: double (nullable = true)
Here is an example table with data:
这是一个包含数据的示例表：
id | lists
-----------
1 | [[a, 1.0], [b, 2.0]]
2 | [[c, 3.0]]
How do I transform the above dataframe to the one below? I need to "explode" the array and append the id at the same time.
如何将上面的数据框转换为下面的数据框？我需要“分解”数组并同时附加 id。
id | col1 | col2
-----------------
1 | a | 1.0
1 | b | 2.0
2 | c | 3.0
Edited Note:
编辑注：
Note there is a difference between the two examples below. The first one contains "an array of structs of elements". While the later just contains "an array of elements".
请注意，以下两个示例之间存在差异。第一个包含“元素结构数组”。而后者只包含“元素数组”。
root
|-- id: integer (nullable = true)
|-- lists: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- text: string (nullable = true)
| | |-- amount: double (nullable = true)
root
|-- a: long (nullable = true)
|-- b: array (nullable = true)
| |-- element: long (containsNull = true)
回答by user7595317
explodeis exactly the function:
explode正是这个功能：
import org.apache.spark.sql.functions._
df.select($"id", explode($"lists")).select($"id", $"col.text", $"col.amount")
