scala 通过检查值是否在列表中以及其他条件来过滤 Spark DataFrame
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/33981487/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Filter Spark DataFrame by checking if value is in a list, with other criteria
提问by Bamqf
As a simplified example, I tried to filter a Spark DataFrame with following code:
作为一个简化示例，我尝试使用以下代码过滤 Spark DataFrame：
val xdf = sqlContext.createDataFrame(Seq(
("A", 1), ("B", 2), ("C", 3)
)).toDF("name", "cnt")
xdf.filter($"cnt" >1 || $"name" isin ("A","B")).show()
Then it errors:
然后它错误：
org.apache.spark.sql.AnalysisException: cannot resolve '((cnt > 1) || name)' due to data type mismatch: differing types in '((cnt > 1) || name)' (boolean and string).;
What's the right way to do it? It seems to me that it stops reading after namecolumn. Is it a bug in the parser? I'm using Spark 1.5.1
正确的做法是什么？在我看来，它在name列之后停止阅读。这是解析器中的错误吗？我正在使用 Spark 1.5.1
回答by pschilakanti
Hoping this will help you:
希望这会帮助你：
val list = List("x","y","t")
xdf.filter($"column".isin(list:_*))
回答by zero323
You have to parenthesize individual expressions:
您必须将单个表达式括起来：
xdf.filter(($"cnt" > 1) || ($"name" isin ("A","B"))).show()
![将 Scala Iterable[tuple] 转换为 RDD](/res/img/loading.gif)