Java Spark 1.6-未能在hadoop二进制路径中找到winutils二进制文件

Spark 1.6-Failed to locate the winutils binary in the hadoop binary path

提问by manuel mourato

I know that there is a very similar post to this one(Failed to locate the winutils binary in the hadoop binary path), however, I have tried every step that was suggested and the same error still appears.

我知道有一个非常相似的帖子（无法在 hadoop 二进制路径中找到 winutils 二进制文件），但是，我已经尝试了建议的每一步，但仍然出现相同的错误。

I'm trying to use the Apache Spark version 1.6.0 on Windows 7 to perform the tutorial on this page http://spark.apache.org/docs/latest/streaming-programming-guide.html, specifically using this code:

我正在尝试在 Windows 7 上使用 Apache Spark 1.6.0 版来执行此页面http://spark.apache.org/docs/latest/streaming-programming-guide.html上的教程，特别是使用以下代码：

./bin/run-example streaming.JavaNetworkWordCount localhost 9999

However, this error keeps appearing:

但是，此错误不断出现：

After reading this post Failed to locate the winutils binary in the hadoop binary path

阅读这篇文章后 Failed to locate the winutils binary in the hadoop binary path

I realized I needed the winutils.exe file, so I have downloaded a hadoop binary 2.6.0 with it, defined an Environment Variable called HADOOP_HOME:

我意识到我需要 winutils.exe 文件，所以我用它下载了一个 hadoop 二进制 2.6.0，定义了一个名为 HADOOP_HOME 的环境变量：

 with value C:\Users\GERAL\Desktop\hadoop-2.6.0\bin  

and placed it on Path like this: %HADOOP_HOME%

并把它放在这样的路径上：%HADOOP_HOME%

Yet the same error still appears when I try the code. Does anyone know how to solve this?

然而，当我尝试代码时仍然出现相同的错误。有谁知道如何解决这个问题？

采纳答案by Srini

If you are running Spark on Windows with Hadoop, then you need to ensure your windows hadoop installation is properly installed. to run spark you need to have winutils.exe and winutils.dll in your hadoop home directory bin folder.

如果您使用 Hadoop 在 Windows 上运行 Spark，那么您需要确保正确安装了 Windows hadoop 安装。要运行 spark，您需要在您的 hadoop 主目录 bin 文件夹中有 winutils.exe 和 winutils.dll。

I would ask you to try this first:

我会请你先试试这个：

1) You can download .dll and .exe fils from the bundle in below link.

1) 您可以从以下链接中的包中下载 .dll 和 .exe 文件。

https://codeload.github.com/sardetushar/hadooponwindows/zip/master

https://codeload.github.com/sardetushar/hadooponwindows/zip/master

2) Copy winutils.exe and winutils.dll from that folder to your $HADOOP_HOME/bin.

2) 将 winutils.exe 和 winutils.dll 从该文件夹复制到您的 $HADOOP_HOME/bin。

3) Set the HADOOP_HOMEeither in your spark-env.sh or at the command, and add HADOOP_HOME/binto PATH.

3)HADOOP_HOME在 spark-env.sh 或命令中设置 ，然后添加HADOOP_HOME/bin到PATH.

and then try running.

然后尝试运行。

If you need any assistance for hadoop installation help, there is a nice link, you can try it.

如果你需要hadoop安装帮助的任何帮助，有一个很好的链接，你可以试试。

http://toodey.com/2015/08/10/hadoop-installation-on-windows-without-cygwin-in-10-mints/

http://toodey.com/2015/08/10/hadoop-installation-on-windows-without-cygwin-in-10-mints/

But, that can wait. you can try the first few steps.

但是，那可以等。您可以尝试前几个步骤。

回答by MohamedAbbas

Download the bin file from here Hadoop Binthen System.setProperty("hadoop.home.dir", "Desktop\bin");

从这里下载 bin 文件Hadoop Bin然后System.setProperty("hadoop.home.dir", "Desktop\bin");

回答by teja

you can try set the HADOOP_HOME environment variable to:

您可以尝试将 HADOOP_HOME 环境变量设置为：

C:\Users\GERAL\Desktop\hadoop-2.6.0

instead of

代替

C:\Users\GERAL\Desktop\hadoop-2.6.0\bin  

回答by Mohasin

I too faced this issue when trying to launch spark-shell from my Windows laptop. I solved this and it worked for me, hope it would help. It's a very small mistake I made - I saved the winutils executable as "winutils.exe" instead of just winutils.

尝试从我的 Windows 笔记本电脑启动 spark-shell 时，我也遇到了这个问题。我解决了这个问题，它对我有用，希望它会有所帮助。这是我犯的一个很小的错误 - 我将 winutils 可执行文件保存为“winutils.exe”，而不仅仅是 winutils。

So when the variable gets resolved it's been resolving to winutils.exe.exe which is nowhere in the Hadoop binaries. I removed that ".exe" and triggered the shell, it worked. I suggest you to have a look at the name it is been saved.

因此，当变量得到解析时，它一直解析为 winutils.exe.exe，它在 Hadoop 二进制文件中无处可寻。我删除了那个“.exe”并触发了外壳，它起作用了。我建议你看看它被保存的名字。

回答by Akhil Jain

The following error is due to missing winutils binary in the classpath while running Spark application. Winutils is a part of Hadoop ecosystem and is not included in Spark. The actual functionality of your application may run correctly even after the exception is thrown. But it is better to have it in place to avoid unnecessary problems. In order to avoid error, download winutils.exebinary and add the same to the classpath.

以下错误是由于运行 Spark 应用程序时在类路径中缺少 winutils 二进制文件。Winutils 是 Hadoop 生态系统的一部分，不包含在 Spark 中。即使在抛出异常之后，您的应用程序的实际功能也可能正确运行。但最好将其安装到位，以避免出现不必要的问题。为了避免错误，下载winutils.exe二进制文件并将其添加到类路径中。

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;

public class SparkTestApp{

    public static void main(String[] args) {

            System.setProperty("hadoop.home.dir", "ANY_DIRECTORY");

    // Example
    // winutils.exe is copied to C:\winutil\bin\
    // System.setProperty("hadoop.home.dir", "C:\winutil\");
            String logFile = "C:\sample_log.log";
            SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local");
            JavaSparkContext sc = new JavaSparkContext(conf);
            JavaRDD logData = sc.textFile(logFile).cache();

            long numAs = logData.filter(new Function<String, Boolean>() {
                public Boolean call(String s) {
                        return s.contains("a");
                }
            }).count();

            System.out.println("Lines with a: " + numAs);

    }

}

}

If winutils.exeis copied to C:\winutil\bin\

如果winutils.exe复制到C:\winutil\bin\

then setProperty as below

然后setProperty如下

System.setProperty("hadoop.home.dir", "C:\winutil\");

回答by satish hiremath

Install JDK 1.8, Download Spark Binary from Apache Spark & Winutils from Git repo

安装 JDK 1.8，从 Apache Spark 和 Winutils 从 Git 存储库下载 Spark 二进制文件

Set the user variables path for JDK, Spark binary, Winutils

为 JDK、Spark 二进制文件、Winutils 设置用户变量路径

JAVA_HOME
C:\Program Files\Java\jdk1.8.0_73

HADOOP_HOME
C:\Hadoop

SPARK_HOME
C:\spark-2.3.1-bin-hadoop2.7

PATH
C:\Program Files\Java\jdk1.8.0_73\bin;%HADOOP_HOME%\bin;%SPARK_HOME%\bin;

JAVA_HOME
C:\Program Files\Java\jdk1.8.0_73

HADOOP_HOME
C:\Hadoop

SPARK_HOME
C:\spark-2.3.1-bin-hadoop2.7

PATH
C:\Program Files\Java\jdk1.8.0_73\bin;%HADOOP_HOME%\bin;%SPARK_HOME%\bin;

Open command prompt and run spark-shell

打开命令提示符并运行 spark-shell

Spark Shell

火花壳