scala 尝试写入 hdfs 时出错：服务器 IPC 版本 9 无法与客户端版本 4 通信
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/23634985/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Error when trying to write to hdfs: Server IPC version 9 cannot communicate with client version 4
提问by Rahul
I am trying a write a file to hdfs using scala and I keep getting the following error
我正在尝试使用 scala 将文件写入 hdfs，但不断收到以下错误
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access0(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at bcomposes.twitter.Util$.<init>(TwitterStream.scala:39)
at bcomposes.twitter.Util$.<clinit>(TwitterStream.scala)
at bcomposes.twitter.StatusStreamer$.main(TwitterStream.scala:17)
at bcomposes.twitter.StatusStreamer.main(TwitterStream.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
I installed hadoop following this tutorial. The code below is what I use to insert a sample file to hdfs.
我按照本教程安装了 hadoop 。下面的代码是我用来将示例文件插入到 hdfs 的代码。
val configuration = new Configuration();
val hdfs = FileSystem.get( new URI( "hdfs://192.168.11.153:54310" ), configuration );
val file = new Path("hdfs://192.168.11.153:54310/s2013/batch/table.html");
if ( hdfs.exists( file )) { hdfs.delete( file, true ); }
val os = hdfs.create( file);
val br = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );
br.write("Hello World");
br.close();
hdfs.close();
The Hadoop version is 2.4.0and hadoop library version I use is 1.2.1. What change should I do to make this work?
Hadoop 版本是2.4.0，我使用的 hadoop 库版本是1.2.1。我应该做哪些改变才能使这项工作发挥作用？
采纳答案by prayagupd
hadoopand sparkversions should be in sync. (In my case, I am working with spark-1.2.0and hadoop 2.2.0)
hadoop和sparkversions 应该同步。（就我而言，我正在使用spark-1.2.0和hadoop 2.2.0）
STEP 1- goto $SPARK_HOME
第 1步 - 转到$SPARK_HOME
STEP 2- Simply mvn buildspark with the version of hadoopclient you want,
第 2步 - 只需mvn build触发hadoop您想要的客户端版本，
mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean package
STEP 3- Also spark project should have proper spark version,
第 3 步- 火花项目也应该有合适的火花版本，
name := "smartad-spark-songplaycount"
version := "1.0"
scalaVersion := "2.10.4"
//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.2.0"
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
References
参考
回答by eliasah
I had the same problem using Hadoop 2.3and I've solved it adding the following lines to my build.sbt file :
我在使用Hadoop 2.3 时遇到了同样的问题，我通过在 build.sbt 文件中添加以下几行解决了这个问题：
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.3.0"
So I think in your case you case use the 2.4.0 version.
所以我认为在你的情况下你使用 2.4.0 版本。
PS: It also worked on your code sample. I hope it will help
PS：它也适用于您的代码示例。我希望它会有所帮助
回答by om-nom-nom
As said in error message
Server IPC version 9 cannot communicate with client version 4your server has slighly newer version, than your client. You have to either downgrade your hadoop cluster (most likely not an option) or upgrade your client library from 1.2.1to 2.x version.
正如错误消息中所说，
Server IPC version 9 cannot communicate with client version 4您的服务器版本比您的客户端略新。您必须降级您的 hadoop 集群（很可能不是一个选项）或将您的客户端库从1.2.12.x 版本升级。
