java TaskSchedulerImpl：初始作业没有接受任何资源；
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/29469462/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
TaskSchedulerImpl: Initial job has not accepted any resources;
提问by sandeep
Here is what I am trying to do.
这就是我想要做的。
I have created two nodes of DataStax enterprise cluster,on top of which I have created a java program to get the count of one table (Cassandra database table).
我创建了 DataStax 企业集群的两个节点，在它们之上我创建了一个 java 程序来获取一个表（Cassandra 数据库表）的计数。
This program was built in eclipse which is actually from a windows box.
这个程序是在 eclipse 中构建的，它实际上来自一个 windows 框。
At the time of running this program from windows it's failing with the following error at runtime:
在从 Windows 运行此程序时，它在运行时失败并出现以下错误：
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
初始作业未接受任何资源；检查您的集群 UI 以确保工作人员已注册并有足够的内存
The same code has been compiled & run successfully on those clusters without any issue. What could be the reason why am getting above error?
相同的代码已在这些集群上编译并成功运行，没有任何问题。出现上述错误的原因可能是什么？
Code:
代码：
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SchemaRDD;
import org.apache.spark.sql.cassandra.CassandraSQLContext;
import com.datastax.bdp.spark.DseSparkConfHelper;
public class SparkProject {
public static void main(String[] args) {
SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf()).setMaster("spark://10.63.24.14X:7077").setAppName("DatastaxTests").set("spark.cassandra.connection.host","10.63.24.14x").set("spark.executor.memory", "2048m").set("spark.driver.memory", "1024m").set("spark.local.ip","10.63.24.14X");
JavaSparkContext sc = new JavaSparkContext(conf);
CassandraSQLContext cassandraContext = new CassandraSQLContext(sc.sc());
SchemaRDD employees = cassandraContext.sql("SELECT * FROM portware_ants.orders");
//employees.registerTempTable("employees");
//SchemaRDD managers = cassandraContext.sql("SELECT symbol FROM employees");
System.out.println(employees.count());
sc.stop();
}
}
回答by ameyamm
I faced similar issue and after some online research and trial-n-error, I narrowed down to 3 causes for this (except for the first the other two are not even close to the error message):
我遇到了类似的问题，经过一些在线研究和试错后，我缩小到 3 个原因（除了第一个，其他两个甚至不接近错误消息）：
- As indicated by the error, probably you are allocating the resources more than that is available.=> This was not my issue
- Hostname & IP Address mishaps: I took care of this by specifying the SPARK_MASTER_IP and SPARK_LOCAL_IP in spark-env.sh
- Disable Firewall on the client: This was the solution that worked for me. Since I was working on a prototype in-house code, I disabled the firewall on the client node. For some reason the worker nodes, were not able to talk back to the client for me. For production purposes, you would want to open-up certain number of ports required.
- 如错误所示，可能您分配的资源超过了可用资源。=> 这不是我的问题
- 主机名和 IP 地址事故：我通过在 spark-env.sh 中指定 SPARK_MASTER_IP 和 SPARK_LOCAL_IP 来解决这个问题
- 在客户端禁用防火墙：这是对我有用的解决方案。由于我正在研究内部代码原型，因此我禁用了客户端节点上的防火墙。由于某种原因，工作节点无法为我回复客户端。出于生产目的，您可能希望打开所需的特定数量的端口。
回答by Sudipta Basak
My problem was that I was assigning too much memory than my slaves had available. Try reducing the memory size of the spark submit. Something like the following:
我的问题是我分配的内存比我的奴隶可用的内存多。尝试减少 spark 提交的内存大小。类似于以下内容：
~/spark-1.5.0/bin/spark-submit --master spark://my-pc:7077 --total-executor-cores 2 --executor-memory 512m
with my ~/spark-1.5.0/conf/spark-env.shbeing:
我的~/spark-1.5.0/conf/spark-env.sh存在：
SPARK_WORKER_INSTANCES=4
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_CORES=2
回答by phact
Please look at Russ's post
请看Russ的帖子
Specifically this section:
本节具体如下：
This is by far the most common first error that a new Spark user will see when attempting to run a new application. Our new and excited Spark user will attempt to start the shell or run their own application and be met with the following message
...
The short term solution to this problem is to make sure you aren't requesting more resources from your cluster than exist or to shut down any apps that are unnecessarily using resources. If you need to run multiple Spark apps simultaneously then you'll need to adjust the amount of cores being used by each app.
这是迄今为止 Spark 新用户尝试运行新应用程序时最常见的第一个错误。我们兴奋的新 Spark 用户将尝试启动 shell 或运行他们自己的应用程序，并会收到以下消息
...
此问题的短期解决方案是确保您不会从集群中请求比现有资源更多的资源，或者关闭任何不必要地使用资源的应用程序。如果您需要同时运行多个 Spark 应用程序，则需要调整每个应用程序使用的内核数量。
回答by vefthym
In my case, the problem was that I had the following line in $SPARK_HOME/conf/spark-env.sh:
就我而言，问题是我有以下行$SPARK_HOME/conf/spark-env.sh：
SPARK_EXECUTOR_MEMORY=3g
SPARK_EXECUTOR_MEMORY=3g
of each worker,
and the following line in $SPARK_HOME/conf/spark-default.sh
每个工人的，
以及以下行$SPARK_HOME/conf/spark-default.sh
spark.executor.memory 4g
spark.executor.memory 4g
in the "master" node.
在“主”节点中。
The problem went away once I changed 4g to 3g. I hope that this will help someone with the same issue. The other answers helped me spot this.
一旦我将 4g 更改为 3g，问题就消失了。我希望这会帮助有同样问题的人。其他答案帮助我发现了这一点。
回答by Sachin
I have faced this issue few times even though the resource allocation was correct.
即使资源分配是正确的，我也遇到过几次这个问题。
The fix was to restart the mesos services.
修复方法是重新启动 mesos 服务。
sudo service mesos-slave restart
sudo service mesos-master restart
