Java 在 Hadoop 2 上运行作业时无法初始化集群异常

Cannot initialize cluster exception while running job on Hadoop 2

提问by SpeedBirdNine

The question is linked to my previous questionAll the daemons are running, jps shows:

该问题与我之前的问题相关联，所有守护进程都在运行，jps 显示：

6663 JobHistoryServer
7213 ResourceManager
9235 Jps
6289 DataNode
6200 NameNode
7420 NodeManager

but the wordcountexample keeps on failing with the following exception:

但该wordcount示例继续失败，但出现以下异常：

ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1238)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1234)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1233)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1262)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
    at WordCount.main(WordCount.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Since it says the problem is in configuration, I am posting the configuration files here. The intention is to create a single node cluster.

因为它说问题出在配置上，所以我在这里发布配置文件。目的是创建一个单节点集群。

yarn-site.xml

纱线站点.xml

<?xml version="1.0"?>
 <configuration>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

core-site.xml

核心站点.xml

<configuration>
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml

hdfs-site.xml

 <configuration>
 <property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/yarn/yarn_data/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/yarn/yarn_data/hdfs/datanode</value>
 </property>
 </configuration>

mapred-site.xml

mapred-site.xml

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>Yarn</value>
  </property>
</configuration>

Please tell what is missing or what am I doing wrong.

请告诉我缺少什么或我做错了什么。

采纳答案by Thomas Jungblut

You have uppercased Yarn, which is probably why it can not resolve it. Try the lowercase version that is suggested in the official documentation.

你有大写Yarn，这可能是它无法解决它的原因。尝试使用官方文档中建议的小写版本。

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
</configuration>

回答by neeraj

I was having similar issue, but yarn was not the issue. After adding following jars into my classpath issue got resolved:

我有类似的问题，但纱线不是问题。将以下 jars 添加到我的类路径后问题得到解决：

• hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-76
• hadoop-mapreduce-client-common-2.2.0.2.0.6.0-76
• hadoop-mapreduce-client-shuffle-2.2.0.2.0.6.0-76

• hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-76
• hadoop-mapreduce-client-common-2.2.0.2.0.6.0-76
• hadoop-mapreduce-client-shuffle-2.2.0.2.0.6.0-76

回答by oae

Looks like i had a lucky day and went with this exception through 'all' of those causes. Summary:

看起来我度过了幸运的一天，并且通过“所有”这些原因排除了这个例外。概括：

• wrong mapreduce.framework.name (see above)
• missing mapreduce job-client jars (see above)
• wrong version (see Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses-submiting job2remoteClustr)
• my configured 'yarn.ipc.client.factory.class' wasn't in the classpath of the yarn server's (just on the client)

• 错误的 mapreduce.framework.name （见上文）
• 缺少 mapreduce 作业客户端 jars（见上文）
• 错误版本（请参阅无法初始化集群。请检查您的 mapreduce.framework.name 和相应服务器地址的配置-提交 job2remoteClustr）
• 我配置的“yarn.ipc.client.factory.class”不在纱线服务器的类路径中（仅在客户端上）

回答by Pratik Khadloya

In my case i was trying to use sqoop and ran into this error. Turns out that i was pointing to the latest version of hadoop 2.0 available from CDH repo for which sqoop was not supported. The version of cloudera was 2.0.0-cdh4.4.0 which had yarn support build in.

在我的情况下，我试图使用 sqoop 并遇到了这个错误。原来我指的是 CDH 存储库提供的最新版本的 hadoop 2.0，但不支持 sqoop。cloudera 的版本是 2.0.0-cdh4.4.0，它内置了纱线支持。

When i used 2.0.0-cdh4.4.0 under hadoop-0.20 the problem went away.
Hope this helps.

当我在 hadoop-0.20 下使用 2.0.0-cdh4.4.0 时，问题就消失了。
希望这可以帮助。

回答by alex

changing mapreduce_shuffle to mapreduce.shuffle made it work in my case

将 mapreduce_shuffle 更改为 mapreduce.shuffle 使其在我的情况下起作用