java.io.IOException: 无法在 Hadoop 二进制文件中找到可执行文件 null\bin\winutils.exe。在 Windows 7 上触发 Eclipse

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. spark Eclipse on windows 7

提问by Elvish_Blade

I'm not able to run a simple sparkjob in Scala IDE(Maven spark project) installed on Windows 7

我无法spark在Scala IDE安装的（Maven spark项目）中运行一个简单的工作Windows 7

Spark core dependency has been added.

添加了 Spark 核心依赖项。

val conf = new SparkConf().setAppName("DemoDF").setMaster("local")
val sc = new SparkContext(conf)
val logData = sc.textFile("File.txt")
logData.count()

Error:

错误：

16/02/26 18:29:33 INFO SparkContext: Created broadcast 0 from textFile at FrameDemo.scala:13
16/02/26 18:29:34 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
    at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:362)
    at <br>org.apache.spark.SparkContext$$anonfun$hadoopFile$$anonfun.apply(SparkContext.scala:1015)
    at org.apache.spark.SparkContext$$anonfun$hadoopFile$$anonfun.apply(SparkContext.scala:1015)
    at <br>org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf.apply(HadoopRDD.scala:176)
    at <br>org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf.apply(HadoopRDD.scala:176)<br>
    at scala.Option.map(Option.scala:145)<br>
    at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176)<br>
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195)<br>
    at org.apache.spark.rdd.RDD$$anonfun$partitions.apply(RDD.scala:239)<br>
    at org.apache.spark.rdd.RDD$$anonfun$partitions.apply(RDD.scala:237)<br>
    at scala.Option.getOrElse(Option.scala:120)<br>
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)<br>
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)<br>
    at org.apache.spark.rdd.RDD$$anonfun$partitions.apply(RDD.scala:239)<br>
    at org.apache.spark.rdd.RDD$$anonfun$partitions.apply(RDD.scala:237)<br>
    at scala.Option.getOrElse(Option.scala:120)<br>
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)<br>
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)<br>
    at org.apache.spark.rdd.RDD.count(RDD.scala:1143)<br>
    at com.org.SparkDF.FrameDemo$.main(FrameDemo.scala:14)<br>
    at com.org.SparkDF.FrameDemo.main(FrameDemo.scala)<br>

回答by Taky

Hereis a good explanation of your problem with the solution.

这是对解决方案问题的一个很好的解释。

1. Download winutils.exe from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe.

2. SetUp your HADOOP_HOME environment variable on the OS level or programmatically:

   System.setProperty("hadoop.home.dir", "full path to the folder with winutils");

3. Enjoy

1. 从http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe下载 winutils.exe 。

2. 在操作系统级别或以编程方式设置您的 HADOOP_HOME 环境变量：

   System.setProperty("hadoop.home.dir", "winutils 所在文件夹的完整路径");

3. 享受

回答by Deokant Gupta

1. Download winutils.exe
2. Create folder, say C:\winutils\bin
3. Copy winutils.exeinside C:\winutils\bin
4. Set environment variable HADOOP_HOMEto C:\winutils

1. 下载 winutils.exe
2. 创建文件夹，说 C:\winutils\bin
3. 复制winutils.exe里面C:\winutils\bin
4. 将环境变量设置HADOOP_HOME为C:\winutils

回答by Ani Menon

Follow this:

按照这个：

1. Create a binfolder in any directory(to be used in step 3).

2. Download winutils.exeand place it in the bin directory.

3. Now add System.setProperty("hadoop.home.dir", "PATH/TO/THE/DIR");in your code.

1. bin在任何目录中创建一个文件夹（将在步骤 3 中使用）。

2. 下载winutils.exe并将其放在 bin 目录中。

3. 现在添加System.setProperty("hadoop.home.dir", "PATH/TO/THE/DIR");您的代码。

回答by Prem S

if we see below issue

如果我们看到下面的问题

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

错误外壳：无法在 hadoop 二进制路径中找到 winutils 二进制文件

java.io.IOException: 无法在 Hadoop 二进制文件中找到可执行文件 null\bin\winutils.exe。

then do following steps

然后执行以下步骤

1. download winutils.exe from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe.
2. and keep this under bin folder of any folder you created for.e.g. C:\Hadoop\bin
3. and in program add following line before creating SparkContext or SparkConf System.setProperty("hadoop.home.dir", "C:\Hadoop");

1. 从http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe下载 winutils.exe。
2. 并将其保存在您为其创建的任何文件夹的 bin 文件夹下。例如 C:\Hadoop\bin
3. 并在创建 SparkContext 或 SparkConf System.setProperty("hadoop.home.dir", "C:\Hadoop") 之前在程序中添加以下行；

回答by Sampat Kumar

1) Download winutils.exe from https://github.com/steveloughran/winutils 
2) Create a directory In windows "C:\winutils\bin
3) Copy the winutils.exe inside the above bib folder .
4) Set the environmental property in the code 
  System.setProperty("hadoop.home.dir", "file:///C:/winutils/");
5) Create a folder "file:///C:/temp" and give 777 permissions.
6) Add config property in spark Session ".config("spark.sql.warehouse.dir", "file:///C:/temp")"

回答by user1023627

On Windows 10 - you should add two different arguments.

在 Windows 10 上 - 您应该添加两个不同的参数。

(1) Add the new variable and value as - HADOOP_HOME and path (i.e. c:\Hadoop) under System Variables.

(1) 在 System Variables 下添加新的变量和值 - HADOOP_HOME 和路径（即 c:\Hadoop）。

(2) Add/append new entry to the "Path" variable as "C:\Hadoop\bin".

(2) 在“Path”变量中添加/追加新条目为“C:\Hadoop\bin”。

The above worked for me.

以上对我有用。

回答by Joabe Lucena

I got the same problem while running unit tests. I found this workaround solution:

我在运行单元测试时遇到了同样的问题。我找到了这个解决方法：

The following workaround allows to get rid of this message:

以下解决方法允许摆脱此消息：

    File workaround = new File(".");
    System.getProperties().put("hadoop.home.dir", workaround.getAbsolutePath());
    new File("./bin").mkdirs();
    new File("./bin/winutils.exe").createNewFile();

from: https://issues.cloudera.org/browse/DISTRO-544

来自：https: //issues.cloudera.org/browse/DISTRO-544

回答by Saurabh

You can alternatively download winutils.exefrom GITHub:

您也可以winutils.exe从 GITHub下载：

https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin

https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin

replace hadoop-2.7.1with the version you want and place the file in D:\hadoop\bin

替换hadoop-2.7.1为您想要的版本并将文件放入D:\hadoop\bin

If you do not have access rights to the environment variable settings on your machine, simply add the below line to your code:

如果您无权访问机器上的环境变量设置，只需将以下行添加到您的代码中：

System.setProperty("hadoop.home.dir", "D:\hadoop");

回答by Ramya

Setting the Hadoop_Home environment variable in system properties didn't work for me. But this did:

在系统属性中设置 Hadoop_Home 环境变量对我不起作用。但这确实：

• Set the Hadoop_Home in the Eclipse Run Configurations environment tab.
• Follow the 'Windows Environment Setup' from here

• 在 Eclipse 运行配置环境选项卡中设置 Hadoop_Home。
• 按照此处的“Windows 环境设置”进行操作

回答by Achilles

That's a tricky one... Your storage letter must be capical. For example "C:\..."

这是一个棘手的问题......你的存储信一定是大写的。例如“ C:\...”