java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/45412593/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-11-03 08:40:54  来源:igfitidea点击:

java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession

javaapache-spark

提问by us56

I have written a Spark Job in Java. When I submit the Job it gives below error:

我用 Java 编写了一个 Spark 作业。当我提交作业时,它会出现以下错误:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
        at com.thinkbiganalytics.veon.util.SparkSessionBuilder.getOrCreateSparkSession(SparkSessionBuilder.java:12)
        at com.thinkbiganalytics.veon.AbstractSparkTransformation.initSparkSession(AbstractSparkTransformation.java:92)
        at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.init(SDPServiceFeeDeductionSourceToEventStore.java:57)
        at com.thinkbiganalytics.veon.AbstractSparkTransformation.doTransform(AbstractSparkTransformation.java:51)
        at com.thinkbiganalytics.veon.transformations.SDPServiceFeeDeductionSourceToEventStore.main(SDPServiceFeeDeductionSourceToEventStore.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

回答by T. Gaw?da

Probably you are deploying your application on the cluster with lower Spark version.

可能您正在使用较低的 Spark 版本在集群上部署您的应用程序。

Please check Spark version on your cluster - it should be the same as version in pom.xml. Please also note, that all Spark dependencies should be marked as providedwhen you use spark-submit to deploy application

请检查集群上的 Spark 版本 - 它应该与 pom.xml 中的版本相同。另请注意,provided当您使用 spark-submit 部署应用程序时,应将所有 Spark 依赖项标记为

回答by UserszrKs

when submitting

提交时

spark-submit

火花提交

check your project should have same dependency as of spark version in pom.xml,

检查您的项目是否应该与 pom.xml 中的 spark 版本具有相同的依赖项,

This may be because you have two spark version on the same machine

这可能是因为你在同一台机器上有两个 spark 版本

for this you can create two various soft link and can use the exact spark-version on which you have build your project

为此,您可以创建两个不同的软链接,并可以使用您在其上构建项目的确切 spark 版本

spark1-submit -> /Users/test/sparks/spark-1.6.2-bin-hadoop2.6/bin/spark-submit

spark2–submit -> /Users/test/sparks/spark-2.1.1-bin-hadoop2.7/bin/spark-submit

or may be because of you have build your project with higher spark-version and deployed on the cluster with lower spark-version

或者可能是因为您使用更高的 spark-version 构建了您的项目并部署在具有较低 spark-version 的集群上

for this you have to upgrade your spark version in the cluster. cloudera provides support for it https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same-cluster/td-p/39880

为此,您必须升级集群中的 Spark 版本。cloudera 为其提供支持 https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Multiple-Spark-version-on-the-same-cluster/td-p/39880

回答by JavaLearner

As per the exception you are getting ,I think required jar is missing you need to add the required jar in your classpath which will resolve the issue.

根据您收到的异常,我认为缺少所需的 jar,您需要在类路径中添加所需的 jar 以解决问题。

refer this linkto download the required jar

请参考此链接下载所需的 jar

回答by qian xi

leaking of jars of spark environment will lead to this problem if you were using Intellij IDEA,you can following the steps below: File -> Project Structure -> Modules -> spark-examples_2.11 -> Dependencies jars -> {spark dir}/spark/assembly/target/scala-2.11/jars/

如果您使用的是 Intellij IDEA,spark 环境的 jars 泄漏将导致此问题,您可以按照以下步骤操作:文件 -> 项目结构 -> 模块 -> spark-examples_2.11 -> 依赖 jars -> {spark dir} /spark/assembly/target/scala-2.11/jars/

回答by userab

I was facing this issue while running from the Intellij editor. I had marked the spark jars as provided in pom.xml, see below:

我在从 Intellij 编辑器运行时遇到了这个问题。我已将火花罐标记为 中提供的pom.xml,见下文:

<dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-sql_2.11</artifactId>
     <version>2.4.0</version>
     <scope>provided</scope>
 </dependency>

On removing the provided scope, the error was gone.

删除提供的范围后,错误消失了。

On making provided spark jars they would be provided only on running the application with spark-submitor having the spark jars on the classpath

在制作提供的火花罐时,它们只会在运行应用程序时提供,spark-submit或者在类路径上有火花罐