scala Apache Spark：java.lang.NoSuchMethodError .rddToPairRDDFunctions
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/40388450/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Apache Spark: java.lang.NoSuchMethodError .rddToPairRDDFunctions
提问by Daniel Kats
sbt packageruns just fine, but after spark-submitI get the error:
sbt package运行得很好，但在spark-submit我收到错误后：
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.rddToPairRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/rdd/PairRDDFunctions; at SmokeStack$.main(SmokeStack.scala:46) at SmokeStack.main(SmokeStack.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
线程“main”中的异常 java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.rddToPairRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering ;)Lorg/apache/spark/rdd/PairRDDFunctions; 在 SmokeStack$.main(SmokeStack.scala:46) 在 SmokeStack.main(SmokeStack.scala) 在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 在 sun .reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$ SparkSubmit$$runMain(SparkSubmit.scala:736) 在 org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 在 org.
Here is the offending line:
这是违规行：
val sigCounts = rowData.map(row => (row("Signature"), 1)).countByKey()
rowDatais an RDD Map[String, String]. "Signature" key exists in all items in the map.
rowData是一个 RDD Map[String, String]。“签名”键存在于地图中的所有项目中。
I suspect this may be a build issue. Below is my sbt file:
我怀疑这可能是构建问题。下面是我的 sbt 文件：
name := "Example1"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
scalacOptions ++= Seq("-feature")
I'm new to Scala so maybe the imports are not correct? I have:
我是 Scala 的新手，所以导入可能不正确？我有：
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import scala.io.Source
回答by maasg
java.lang.NoSuchMethodErroris often an indication that the version the code was compiled against is on a higher version than the libraries used at runtime.
java.lang.NoSuchMethodError通常表明编译代码的版本高于运行时使用的库的版本。
With Spark, that means that the Spark version used to compile is different from the one deployed (on the machine or cluster).
对于 Spark，这意味着用于编译的 Spark 版本与部署（在机器或集群上）的版本不同。
Aligning the versions between development and runtime should solve this issue.
在开发和运行时之间调整版本应该可以解决这个问题。
回答by Shrikant Prabhu
I Was facing the same problem while reading a simple oneline json file into a dataframe and showing it using .show() method. I would get this error on myDF.show() line of code.
我在将一个简单的单行 json 文件读入数据帧并使用 .show() 方法显示它时遇到了同样的问题。我会在 myDF.show() 代码行上收到此错误。
For me it turned out to be wrong version of spark-sql library in the build.
对我来说，结果是构建中的 spark-sql 库版本错误。
i.e. I was having in my External Libraries from SBT , instead of .
即我在 SBT 的外部库中拥有，而不是 .
Adding following line to my build.sbt resolved the issue
将以下行添加到我的 build.sbt 解决了该问题
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"
回答by Jake
If updating the version of one spark dependency, it is safest to update them all to the same version
如果更新一个 spark 依赖项的版本，将它们全部更新为同一版本是最安全的
