scala 使用本地时增加 Spark 内存[*]

Increase Spark memory when using local[*]

提问by BAR

How do I increase Spark memory when using local[*]?

使用 local[*] 时如何增加 Spark 内存？

I tried setting the memory like this:

我尝试像这样设置内存：

  val conf = new SparkConf()
    .set("spark.executor.memory", "1g")
    .set("spark.driver.memory", "4g")
    .setMaster("local[*]")
    .setAppName("MyApp")

But I still get:

但我仍然得到：

MemoryStore: MemoryStore started with capacity 524.1 MB

Does this have something to do with:

这是否与以下有关：

.setMaster("local[*]")

采纳答案by BAR

I was able to solve this by running SBT with:

我能够通过运行 SBT 来解决这个问题：

sbt -mem 4096

However the MemoryStore is half the size. Still looking into where this fraction is.

然而，MemoryStore 是它的一半大小。仍在研究这部分的位置。

回答by Gillespie

Assuming that you are using the spark-shell.. setting the spark.driver.memory in your application isn't working because your driver process has already started with default memory.

假设您正在使用 spark-shell.. 在您的应用程序中设置 spark.driver.memory 不起作用，因为您的驱动程序进程已经使用默认内存启动。

You can either launch your spark-shell using:

您可以使用以下方法启动您的 spark-shell：

./bin/spark-shell --driver-memory 4g

or you can set it in spark-defaults.conf:

或者你可以在 spark-defaults.conf 中设置它：

spark.driver.memory 4g

If you are launching an application using spark-submit, you must specify the driver memory as an argument:

如果您使用 spark-submit 启动应用程序，则必须将驱动程序内存指定为参数：

./bin/spark-submit --driver-memory 4g --class main.class yourApp.jar

回答by fansy1990

in spark 2.x ,you can use SparkSession,which looks like :

在 spark 2.x 中，你可以使用 SparkSession，它看起来像：

        val spark= new SparkSession()
        .config("spark.executor.memory", "1g")
        .config("spark.driver.memory", "4g")
        .setMaster("local[*]")
        .setAppName("MyApp")

回答by bshyamkumar

Tried --driver-memory 4g, --executor-memory 4g, neither worked to increase working memory. However, I noticed that bin/spark-submitwas picking up _JAVA_OPTIONS, setting that to -Xmx4gresolved it. I use jdk7

试过--driver-memory 4g，，--executor-memory 4g都没有增加工作记忆。但是，我注意到bin/spark-submit正在接收 _JAVA_OPTIONS，设置它来-Xmx4g解决它。我用jdk7

回答by hamza tuna

You can't change driver memory after application start link.

您不能在应用程序启动链接后更改驱动程序内存。

回答by Glennie Helles Sindholt

The fraction of the heap used for Spark's memory cache is by default 0.6, so if you need more than 524,1MB, you should increase the spark.executor.memorysetting :)

用于 Spark 内存缓存的堆的比例默认为 0.6，因此如果您需要超过 524,1MB，您应该增加spark.executor.memory设置:)

Technically you could also increase the fraction used for Spark's memory cache, but I believe this is discouraged or at least requires you to do some additional configuration. See https://spark.apache.org/docs/1.0.2/configuration.htmlfor more details.

从技术上讲，您还可以增加用于 Spark 内存缓存的比例，但我认为这是不鼓励的，或者至少需要您进行一些额外的配置。有关更多详细信息，请参阅https://spark.apache.org/docs/1.0.2/configuration.html。

回答by Ryan Miao

Version

版本

spark-2.3.1

spark-2.3.1

Source Code

源代码

org.apache.spark.launcher.SparkSubmitCommandBuilder:267

org.apache.spark.launcher.SparkSubmitCommandBuilder:267

String memory = firstNonEmpty(tsMemory, config.get(SparkLauncher.DRIVER_MEMORY),
System.getenv("SPARK_DRIVER_MEMORY"), System.getenv("SPARK_MEM"), DEFAULT_MEM);
cmd.add("-Xmx" + memory);

1. SparkLauncher.DRIVER_MEMORY

1. SparkLauncher.DRIVER_MEMORY

--driver-memory 2g

--驱动内存2g

1. SPARK_DRIVER_MEMORY

2. SPARK_DRIVER_MEMORY

vim conf/spark-env.sh

SPARK_DRIVER_MEMORY="2g"

vim conf/spark-env.sh

SPARK_DRIVER_MEMORY="2g"

1. SPARK_MEM

3. SPARK_MEM

vim conf/spark-env.sh

SPARK_MEM="2g"

vim conf/spark-env.sh

SPARK_MEM="2g"

1. DEFAULT_MEM

4. DEFAULT_MEM

1g

1克

回答by Rajiv Singh

To assign memory to Spark:

为 Spark 分配内存：

on Command shell: /usr/lib/spark/bin/spark-shell --driver-memory=16G --num-executors=100 --executor-cores=8 --executor-memory=16G

在命令外壳上： /usr/lib/spark/bin/spark-shell --driver-memory=16G --num-executors=100 --executor-cores=8 --executor-memory=16G

回答by Rajiv Singh

/usr/lib/spark/bin/spark-shell --driver-memory=16G --num-executors=100 --executor-cores=8 --executor-memory=16G --conf spark.driver.maxResultSize = 2G