Java Spark 异常：写入行时任务失败
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/36034928/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Spark Exception : Task failed while writing rows
提问by Aditya Calangutkar
I am reading text files and converting them to parquet files. I am doing it using spark code. But when i try to run the code I get following exception
我正在阅读文本文件并将它们转换为镶木地板文件。我正在使用 spark 代码来做这件事。但是当我尝试运行代码时，我得到以下异常
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 9, XXXX.XXX.XXX.local): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows(commands.scala:191)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert.apply(commands.scala:160)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert.apply(commands.scala:160)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArithmeticException: / by zero
at parquet.hadoop.InternalParquetRecordWriter.initStore(InternalParquetRecordWriter.java:101)
at parquet.hadoop.InternalParquetRecordWriter.<init>(InternalParquetRecordWriter.java:94)
at parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:64)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:282)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:252)
at org.apache.spark.sql.parquet.ParquetOutputWriter.<init>(newParquet.scala:83)
at org.apache.spark.sql.parquet.ParquetRelation2$$anon.newInstance(newParquet.scala:229)
at org.apache.spark.sql.sources.DefaultWriterContainer.initWriters(commands.scala:470)
at org.apache.spark.sql.sources.BaseWriterContainer.executorSideSetup(commands.scala:360)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows(commands.scala:172)
... 8 more
I am trying to write the dataframe in following fashion :
我正在尝试以下列方式编写数据框：
dataframe.write().parquet(Path)
Any help is highly appreciated.
任何帮助都受到高度赞赏。
回答by leo9r
I found that disabling speculationprevents this error from happening. I'm not very sure why. It seems that speculative and non-speculative tasks are conflicting when writing parquet rows.
我发现禁用推测可以防止发生此错误。我不太确定为什么。在编写镶木地板行时，推测性和非推测性任务似乎是相互冲突的。
sparkConf.set("spark.speculation","false")
回答by Carlos Bribiescas
Another possible reason is that you're hitting s3 request rate limits. If you look closely at your logs you may see something like this
另一个可能的原因是您达到了 s3 请求速率限制。如果您仔细查看日志，您可能会看到类似这样的内容
AmazonS3Exception: Please reduce your request rate.
AmazonS3Exception: Please reduce your request rate.
While the Spark UI will say
虽然 Spark UI 会说
Task failed while writing rows
Task failed while writing rows
I doubt its the reason you're getting an issue, but its a possible reason if you're running a highly intensive job. So I included just for answer's completeness.
我怀疑这是您遇到问题的原因，但如果您正在执行一项高度密集的工作，这可能是一个原因。所以我包括只是为了答案的完整性。
回答by Steve Loughran
This is where having all the source to hand helps: paste the stack trace in an IDE that can go from stack trace to lines of code, and see what it says. It's probably just some init/config problem
这就是掌握所有源代码的帮助所在：将堆栈跟踪粘贴到可以从堆栈跟踪到代码行的 IDE 中，然后看看它说了什么。这可能只是一些初始化/配置问题
回答by Julien Nyambal
If ever it is still relavant, the experience I had with this issue was that I did not start hadoop. If you run spark on top of it, it might be worth starting hadoop and check again.
如果它仍然相关，我在这个问题上的经验是我没有启动 hadoop。如果你在它上面运行 spark，可能值得启动 hadoop 并再次检查。
回答by Michal Ficek
In my case, I saw this error when I tried to overwrite hdfs directory which belonged to a different user. Deleting the directory a letting my process write it from scratch solved it. So I guess, more digging is appropriate in direction of user permissions on hdfs.
就我而言，当我尝试覆盖属于不同用户的 hdfs 目录时，我看到了这个错误。删除目录让我的进程从头开始编写它解决了它。所以我想，在 hdfs 上的用户权限方向上进行更多的挖掘是合适的。
