如何使用 Scala 写入 HDFS
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/32380272/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to write to HDFS using Scala
提问by aironman
I am learning Scala and i need to write a custom file to HDFS. I have my own HDFS running on a Cloudera image using vmware fusion on my laptop.
我正在学习 Scala，我需要将自定义文件写入 HDFS。我在笔记本电脑上使用 vmware fusion 在 Cloudera 映像上运行自己的 HDFS。
This is my actual code:
这是我的实际代码：
package org.glassfish.samples
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.PrintWriter;
/**
* @author ${user.name}
*/
object App {
def main(args : Array[String]) {
println( "Trying to write to HDFS..." )
val conf = new Configuration()
val fs= FileSystem.get(conf)
val output = fs.create(new Path("hdfs://quickstart.cloudera:8020/tmp/mySample.txt"))
val writer = new PrintWriter(output)
try {
writer.write("this is a test")
writer.write("\n")
}
finally {
writer.close()
}
print("Done!")
}
}
And i am getting this exception:
我得到这个例外：
Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://quickstart.cloudera:8020/tmp, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:414)
at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:439)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:426)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
at org.glassfish.samples.App$.main(App.scala:19)
at org.glassfish.samples.App.main(App.scala)
... 6 more
I can access hdfs using the terminal and Hue
我可以使用终端和 Hue 访问 hdfs
[cloudera@quickstart ~]$ hdfs dfs -ls /tmp
Found 3 items
drwxr-xr-x - hdfs supergroup 0 2015-06-09 17:54 /tmp/hadoop-yarn
drwx-wx-wx - hive supergroup 0 2015-08-17 15:24 /tmp/hive
drwxr-xr-x - cloudera supergroup 0 2015-08-17 16:50 /tmp/labdata
this is my pom.xml
这是我的pom.xml
I ran the project using the command:
我使用以下命令运行了该项目：
mvn clean package scala:run
mvn 清洁包 scala:run
What do i am doing wrong? thank you in advance!
我做错了什么？先感谢您！
EDIT after @jeroenr advice
在@jeroenr 建议后编辑
This is actual code:
这是实际代码：
package org.glassfish.samples
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.PrintWriter;
/**
* @author ${user.name}
*/
object App {
//def foo(x : Array[String]) = x.foldLeft("")((a,b) => a + b)
def main(args : Array[String]) {
println( "Trying to write to HDFS..." )
val conf = new Configuration()
//conf.set("fs.defaultFS", "hdfs://quickstart.cloudera:8020")
conf.set("fs.defaultFS", "hdfs://192.168.30.147:8020")
val fs= FileSystem.get(conf)
val output = fs.create(new Path("/tmp/mySample.txt"))
val writer = new PrintWriter(output)
try {
writer.write("this is a test")
writer.write("\n")
}
finally {
writer.close()
println("Closed!")
}
println("Done!")
}
}
回答by Jeroen Rosenberg
Have a look at this this example here. I think the problem is that you don't configure the default file system using
conf.set("fs.defaultFS", "hdfs://quickstart.cloudera:8020")
and pass the relative path, like so:
并传递相对路径，如下所示：
fs.create(new Path("/tmp/mySample.txt"))
to write to the file, call 'write' directly on the output stream returned by fs.create, like so:
要写入文件，请直接在fs.create返回的输出流上调用“write” ，如下所示：
val os = fs.create(new Path("/tmp/mySample.txt"))
os.write("This is a test".getBytes)
