Java 在 hadoop 中连接到 HDFS 时出现 EOFException
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/9710295/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
EOFException in connecting to HDFS in hadoop
提问by david
I am, in the included test program, attempting to copy a file from the local disk to HDFS. The code is as follows:
我在包含的测试程序中尝试将文件从本地磁盘复制到 HDFS。代码如下：
package foo.foo1.foo2.test;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class TestTestTest {
public static void main(String[] args) {
String srcLocation = "foo";
String destination = "hdfs:///tmp/";
FileSystem hdfs = null;
Configuration configuration = new Configuration();
configuration.set("fs.default.name", "hdfs://namenode:54310/");
try {
hdfs = FileSystem.get(configuration);
} catch (IOException e2) {
e2.printStackTrace();
return;
}
Path srcpath = new Path(srcLocation);
Path dstpath = new Path(destination);
try {
hdfs.copyFromLocalFile(srcpath, dstpath);
} catch (IOException e) {
e.printStackTrace();
}
}
}
This fails with the following exception:
这将失败，但有以下异常：
java.io.IOException: Call to namenode/10.1.1.1:54310 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access0(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at foo.foo1.foo2.test.TestTestTest.main(TestTestTest.java:22)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
My question is deceptively simple: What is causing this, and how can I make this program work? From what little information I've been able to find, I gather that there is a problem connecting to HDFS, and that this has something to do with the fs.default.name property in the Configuration. Below is the relevant section of my core-site.xml file:
我的问题看似简单：是什么导致了这种情况，我怎样才能使这个程序运行？从我所能找到的一些信息来看，我认为连接到 HDFS 存在问题，这与配置中的 fs.default.name 属性有关。下面是我的 core-site.xml 文件的相关部分：
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://namenode:54310</value>
</property>
</configuration>
Perhaps of special interest is the fact that if I bundle all the jars in my classpath into one mega-jar, and run this program via the hadoop command, it works just fine. So what am I doing wrong?
也许特别有趣的事实是，如果我将类路径中的所有 jar 捆绑到一个大型 jar 中，并通过 hadoop 命令运行该程序，它就可以正常工作。那么我做错了什么？
采纳答案by Matt D
Make sure you are compiling against the same Hadoop version that you are running on your cluster.
确保您编译的 Hadoop 版本与您在集群上运行的 Hadoop 版本相同。
回答by user1493288
Make sure the proper exception with serializable interface and going with a proper version of hadoop.
确保具有可序列化接口的正确异常并使用正确版本的 hadoop。
回答by can
Your code for problem perhaps "foo" directory path at fault
您的问题代码可能是“foo”目录路径有问题
回答by Karn_way
I had faced the similar issue in the past. But the problem was mine , i had two different version of hadoop. i had started the daemons from earlier version and bash_profile was pointing to new and this issue occurred. So make sure you are not playing with version mismatch``
我过去遇到过类似的问题。但问题是我的，我有两个不同版本的 hadoop。我已经从早期版本启动了守护进程，而 bash_profile 指向 new 并且发生了这个问题。所以确保你没有在玩版本不匹配``
