scala Spark2.1.0 不兼容 Jackson 版本 2.7.6
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/43841091/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Spark2.1.0 incompatible Hymanson versions 2.7.6
提问by Yang
I am trying to run a simple spark example in intellij, but I get the error like that:
我正在尝试在 intellij 中运行一个简单的 spark 示例，但出现如下错误：
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:819)
at spark.test$.main(test.scala:19)
at spark.test.main(test.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: com.fasterxml.Hymanson.databind.JsonMappingException: Incompatible Hymanson version: 2.7.6
at com.fasterxml.Hymanson.module.scala.HymansonModule$class.setupModule(HymansonModule.scala:64)
at com.fasterxml.Hymanson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.Hymanson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
I have try to update my Hymanson dependency, but it seems not work, I do this:
我尝试更新我的 Hymanson 依赖项，但似乎不起作用，我这样做：
libraryDependencies += "com.fasterxml.Hymanson.core" % "Hymanson-core" % "2.8.7"
libraryDependencies += "com.fasterxml.Hymanson.core" % "Hymanson-databind" % "2.8.7"
but it still appear the same error messages, can some one help me to fix the error?
但它仍然出现相同的错误消息，有人可以帮助我修复错误吗？
Here is spark example code:
这是火花示例代码：
object test {
def main(args: Array[String]): Unit = {
if (args.length < 1) {
System.err.println("Usage: <file>")
System.exit(1)
}
val conf = new SparkConf()
val sc = new SparkContext("local","wordcount",conf)
val line = sc.textFile(args(0))
line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)
sc.stop()
}
}
And here is my built.sbt:
这是我的built.sbt：
name := "testSpark2"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "com.fasterxml.Hymanson.core" % "Hymanson-core" % "2.8.7"
libraryDependencies += "com.fasterxml.Hymanson.core" % "Hymanson-databind" % "2.8.7"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"
回答by himanshuIIITian
Spark 2.1.0 contains com.fasterxml.Hymanson.coreas transitive dependency. So, we do not need to include then in libraryDependencies.
Spark 2.1.0 包含com.fasterxml.Hymanson.core作为传递依赖。因此，我们不需要将 then 包含在libraryDependencies.
But if you want to add a different com.fasterxml.Hymanson.coredependencies' version then you have to override them. Like this:
但是，如果您想添加不同的com.fasterxml.Hymanson.core依赖项版本，则必须覆盖它们。像这样：
name := "testSpark2"
version := "1.0"
scalaVersion := "2.11.8"
dependencyOverrides += "com.fasterxml.Hymanson.core" % "Hymanson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.Hymanson.core" % "Hymanson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.Hymanson.module" % "Hymanson-module-scala_2.11" % "2.8.7"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"
So, change your build.sbtlike the one above and it will work as expected.
所以，改变你build.sbt上面的那个，它会按预期工作。
I hope it helps!
我希望它有帮助！
回答by Leon
FYI. For my case, I'm using spark and kafka-streamsin the app, while kafka-streams uses com.fasterxml.Hymanson.core 2.8.5. Adding excludeas below fixed the issue
供参考。就我而言，我kafka-streams在应用程序中使用 spark ，而 kafka-streams 使用 com.fasterxml.Hymanson.core 2.8.5。添加exclude如下修复了问题
(gradle)
（毕业）
compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){
exclude group:"com.fasterxml.Hymanson.core"
}
回答by Thomas Decaux
Solution with Gradle, by using the resolutionStrategy (https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html):
Gradle 的解决方案，通过使用 resolutionStrategy ( https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html)：
configurations {
all {
resolutionStrategy {
force 'com.fasterxml.Hymanson.core:Hymanson-core:2.4.4', 'com.fasterxml.Hymanson.core:Hymanson-databind:2.4.4', 'com.fasterxml.Hymanson.core:Hymanson-annotations:2.4.4'
}
}
}
