scala 如何在代码的任何位置获取当前 SparkSession?

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/44502872/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-10-22 09:18:00  来源:igfitidea点击:

How can I get the current SparkSession in any place of the codes?

scalaapache-spark

提问by PC9527

I have created a session in the main()function, like this:

我在main()函数中创建了一个会话,如下所示:

val sparkSession = SparkSession.builder.master("local[*]").appName("Simple Application").getOrCreate()

Now if I want to configure the application or access the properties, I can use the local variable sparkSessionin the same function.

现在如果我想配置应用程序或访问属性,我可以sparkSession在同一个函数中使用局部变量。

What if I want to access this sparkSessionelsewhere in the same project, like project/module/.../.../xxx.scala. What should I do?

如果我想sparkSession在同一个项目的其他地方访问它,比如project/module/.../.../xxx.scala. 我该怎么办?

回答by Tzach Zohar

Once a session was created (anywhere), you can safely use:

创建会话后(在任何地方),您可以安全地使用:

SparkSession.builder().getOrCreate()

To get the (same) session anywhere in the code, as long as the session is still alive. Spark maintains a single active session so unless it was stopped or crashed, you'll get the same one.

要在代码中的任何位置获取(相同的)会话,只要会话仍然存在。Spark 维护一个活动会话,因此除非它停止或崩溃,否则您将获得相同的会话。

回答by Andrei Boaghe

Since 2.2.0you can access the active SparkSessionthrough:

2.2.0 开始,您可以通过以下方式访问活动的 SparkSession

/**
 * Returns the active SparkSession for the current thread, returned by the builder.
 *
 * @since 2.2.0
 */
def getActiveSession: Option[SparkSession] = Option(activeThreadSession.get)

or default SparkSession:

默认 SparkSession

/**
 * Returns the default SparkSession that is returned by the builder.
 *
 * @since 2.2.0
 */
def getDefaultSparkSession: Option[SparkSession] = Option(defaultSession.get)

回答by Ramesh Maharjan

When SparkSessionvariable has been defined as

SparkSession变量被定义为

val sparkSession = SparkSession.builder.master("local[*]").appName("Simple Application").getOrCreate()

This variable is going to point/referto only one SparkSessionas its a val. And you can always pass to different classes for them to access as well as

此变量将point/refer仅将一个SparkSession作为其 a val。你总是可以传递给不同的类让他们访问以及

val newClassCall = new NewClass(sparkSession)

Now you can use the same sparkSessionin that new class as well.

现在,您也可以sparkSession在该新类中使用相同的内容。