java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/45412593/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
提问by us56
I have written a Spark Job in Java. When I submit the Job it gives below error:
我用 Java 编写了一个 Spark 作业。当我提交作业时，它会出现以下错误：
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
at com.thinkbiganalytics.veon.util.SparkSessionBuilder.getOrCreateSparkSession(SparkSessionBuilder.java:12)
at com.thinkbiganalytics.veon.AbstractSparkTransformation.initSparkSession(AbstractSparkTransformation.java:92)
at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.init(SDPServiceFeeDeductionSourceToEventStore.java:57)
at com.thinkbiganalytics.veon.AbstractSparkTransformation.doTransform(AbstractSparkTransformation.java:51)
at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.main(SDPServiceFeeDeductionSourceToEventStore.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
回答by T. Gaw?da
Probably you are deploying your application on the cluster with lower Spark version.
可能您正在使用较低的 Spark 版本在集群上部署您的应用程序。
Please check Spark version on your cluster - it should be the same as version in pom.xml. Please also note, that all Spark dependencies should be marked as providedwhen you use spark-submit to deploy application
请检查集群上的 Spark 版本 - 它应该与 pom.xml 中的版本相同。另请注意，provided当您使用 spark-submit 部署应用程序时，应将所有 Spark 依赖项标记为
回答by UserszrKs
when submitting
提交时
spark-submit
火花提交
check your project should have same dependency as of spark version in pom.xml,
检查您的项目是否应该与 pom.xml 中的 spark 版本具有相同的依赖项，
This may be because you have two spark version on the same machine
这可能是因为你在同一台机器上有两个 spark 版本
for this you can create two various soft link and can use the exact spark-version on which you have build your project
为此，您可以创建两个不同的软链接，并可以使用您在其上构建项目的确切 spark 版本
spark1-submit -> /Users/test/sparks/spark-1.6.2-bin-hadoop2.6/bin/spark-submit
spark2–submit -> /Users/test/sparks/spark-2.1.1-bin-hadoop2.7/bin/spark-submit
or may be because of you have build your project with higher spark-version and deployed on the cluster with lower spark-version
或者可能是因为您使用更高的 spark-version 构建了您的项目并部署在具有较低 spark-version 的集群上
for this you have to upgrade your spark version in the cluster. cloudera provides support for it https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same-cluster/td-p/39880
为此，您必须升级集群中的 Spark 版本。cloudera 为其提供支持 https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same-cluster/td-p/39880
回答by JavaLearner
回答by qian xi
leaking of jars of spark environment will lead to this problem if you were using Intellij IDEA,you can following the steps below: File -> Project Structure -> Modules -> spark-examples_2.11 -> Dependencies jars -> {spark dir}/spark/assembly/target/scala-2.11/jars/
如果您使用的是 Intellij IDEA，spark 环境的 jars 泄漏将导致此问题，您可以按照以下步骤操作：文件 -> 项目结构 -> 模块 -> spark-examples_2.11 -> 依赖 jars -> {spark dir} /spark/assembly/target/scala-2.11/jars/
回答by userab
I was facing this issue while running from the Intellij editor. I had marked the spark jars as provided in pom.xml, see below:
我在从 Intellij 编辑器运行时遇到了这个问题。我已将火花罐标记为 中提供的pom.xml，见下文：
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.0</version>
<scope>provided</scope>
</dependency>
On removing the provided scope, the error was gone.
删除提供的范围后，错误消失了。
On making provided spark jars they would be provided only on running the application with spark-submitor having the spark jars on the classpath
在制作提供的火花罐时，它们只会在运行应用程序时提供，spark-submit或者在类路径上有火花罐
