scala Spark - 行值的总和
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/36360540/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Spark - Sum of row values
提问by karoma
I have the following DataFrame:
我有以下数据帧：
January | February | March
-----------------------------
10 | 10 | 10
20 | 20 | 20
50 | 50 | 50
I'm trying to add a column to this which is the sum of the values of each row.
我正在尝试为此添加一列，这是每行值的总和。
January | February | March | TOTAL
----------------------------------
10 | 10 | 10 | 30
20 | 20 | 20 | 60
50 | 50 | 50 | 150
As far as I can see, all the built in aggregate functions seem to be for calculating values in single columns. How do I go about using values across columns on a per row basis (using Scala)?
据我所知，所有内置的聚合函数似乎都是用于计算单列中的值。如何在每行的基础上跨列使用值（使用 Scala）？
I've gotten as far as
我已经到了
val newDf: DataFrame = df.select(colsToSum.map(col):_*).foreach ...
回答by David Griffin
You were very close with this:
你非常接近这个：
val newDf: DataFrame = df.select(colsToSum.map(col):_*).foreach ...
Instead, try this:
相反，试试这个：
val newDf = df.select(colsToSum.map(col).reduce((c1, c2) => c1 + c2) as "sum")
I think this is the best of the the answers, because it is as fast as the answer with the hard-coded SQL query, and as convenient as the one that uses the UDF. It's the best of both worlds -- and I didn't even add a full line of code!
我认为这是最好的答案，因为它与硬编码 SQL 查询的答案一样快，并且与使用UDF. 这是两全其美的——而且我什至没有添加完整的代码行！
回答by Alberto Bonsanto
Alternatively and using Hugo's approach and example, you can create a UDFthat receives any quantity of columns and sumthem all.
或者，使用 Hugo 的方法和示例，您可以创建一个UDF接收任意数量的列和sum所有列。
from functools import reduce
def superSum(*cols):
return reduce(lambda a, b: a + b, cols)
add = udf(superSum)
df.withColumn('total', add(*[df[x] for x in df.columns])).show()
+-------+--------+-----+-----+
|January|February|March|total|
+-------+--------+-----+-----+
| 10| 10| 10| 30|
| 20| 20| 20| 60|
+-------+--------+-----+-----+
回答by Hugo Reyes
This code is in Python, but it can be easily translated:
这段代码是用 Python 编写的，但它可以很容易地翻译：
# First we create a RDD in order to create a dataFrame:
rdd = sc.parallelize([(10, 10,10), (20, 20,20)])
df = rdd.toDF(['January', 'February', 'March'])
df.show()
# Here, we create a new column called 'TOTAL' which has results
# from add operation of columns df.January, df.February and df.March
df.withColumn('TOTAL', df.January + df.February + df.March).show()
Output:
输出：
+-------+--------+-----+
|January|February|March|
+-------+--------+-----+
| 10| 10| 10|
| 20| 20| 20|
+-------+--------+-----+
+-------+--------+-----+-----+
|January|February|March|TOTAL|
+-------+--------+-----+-----+
| 10| 10| 10| 30|
| 20| 20| 20| 60|
+-------+--------+-----+-----+
You can also create an User Defined Function it you want, here a link of Spark documentation: UserDefinedFunction (udf)
您还可以创建您想要的用户定义函数，这里是 Spark 文档的链接： UserDefinedFunction (udf)
回答by Pawe? Kaczorowski
Working Scala example with dynamic column selection:
具有动态列选择的工作 Scala 示例：
import sqlContext.implicits._
val rdd = sc.parallelize(Seq((10, 10, 10), (20, 20, 20)))
val df = rdd.toDF("January", "February", "March")
df.show()
+-------+--------+-----+
|January|February|March|
+-------+--------+-----+
| 10| 10| 10|
| 20| 20| 20|
+-------+--------+-----+
val sumDF = df.withColumn("TOTAL", df.columns.map(c => col(c)).reduce((c1, c2) => c1 + c2))
sumDF.show()
+-------+--------+-----+-----+
|January|February|March|TOTAL|
+-------+--------+-----+-----+
| 10| 10| 10| 30|
| 20| 20| 20| 60|
+-------+--------+-----+-----+
回答by Himaprasoon
You can use expr() for this.In scala use
您可以为此使用 expr()。在 scala 中使用
df.withColumn("TOTAL", expr("January+February+March"))
