scala 有没有更好的方法来显示整个 Spark SQL DataFrame？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/30264373/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Is there better way to display entire Spark SQL DataFrame?
提问by Yuri Brovman
I would like to display the entire Apache Spark SQL DataFrame with the Scala API. I can use the show()method:
我想用 Scala API 显示整个 Apache Spark SQL DataFrame。我可以使用以下show()方法：
myDataFrame.show(Int.MaxValue)
Is there a better way to display an entire DataFrame than using Int.MaxValue?
有没有比使用更好的方法来显示整个数据帧Int.MaxValue？
回答by Grega Ke?pret
It is generally not advisable to display an entire DataFrame to stdout, because that means you need to pull the entire DataFrame (all of its values) to the driver (unless DataFrameis already local, which you can check with df.isLocal).
通常不建议将整个 DataFrame 显示到 stdout，因为这意味着您需要将整个 DataFrame（其所有值）拉到驱动程序（除非DataFrame已经是本地的，您可以使用 进行检查df.isLocal）。
Unless you know ahead of time that the size of your dataset is sufficiently small so that driver JVM process has enough memory available to accommodate all values, it is not safe to do this. That's why DataFrame API's show()by default shows you only the first 20 rows.
除非您提前知道数据集的大小足够小，以便驱动程序 JVM 进程有足够的可用内存来容纳所有值，否则这样做是不安全的。这就是show()默认情况下DataFrame API只显示前 20 行的原因。
You could use the df.collectwhich returns Array[T]and then iterate over each line and print it:
您可以使用df.collectwhich 返回Array[T]，然后迭代每一行并打印它：
df.collect.foreach(println)
but you lose all formatting implemented in df.showString(numRows: Int)(that show()internally uses).
但是您丢失了df.showString(numRows: Int)（show()内部使用的）中实现的所有格式。
So no, I guess there is no better way.
所以不，我想没有更好的方法。
回答by AkshayK
One way is using count()function to get the total number of records and use show(rdd.count()).
一种方法是使用count()function 获取记录总数并使用show(rdd.count()).
回答by Suresh G
Try with,
试试看，
df.show(35, false)
df.show(35, 假)
It will display 35 rows and 35 column values with full values name.
它将显示具有完整值名称的 35 行和 35 列值。
回答by ayan guha
As others suggested, printing out entire DF is bad idea. However, you can use df.rdd.foreachPartition(f)to print out partition-by-partition without flooding driver JVM (y using collect)
正如其他人所建议的那样，打印出整个 DF 是个坏主意。但是，您可以使用df.rdd.foreachPartition(f)逐个分区打印出驱动程序 JVM（y 使用 collect）
回答by Justin Pihony
Nothing more succinct than that, but if you want to avoid the Int.MaxValue, then you could use a collectand process it, or foreach. But, for a tabular format without much manual code, showis the best you can do.
没有什么比这更简洁的了，但如果你想避免Int.MaxValue，那么你可以使用 acollect并处理它，或者foreach。但是，对于没有太多手动代码的表格格式，这show是您能做的最好的事情。
回答by Rajeev Rathor
In javaI have tried it with two ways.
This is working perfectly for me:
在java我尝试了两种方法。这对我来说非常有效：
1.
1.
data.show(SomeNo);
2.
2.
data.foreach(new ForeachFunction<Row>() {
public void call(Row arg0) throws Exception {
System.out.println(arg0);
}
});
回答by keypoint
I've tried show()and it seems working sometimes. But sometimes not working, just give it a try:
我试过show()，它有时似乎有效。但有时不工作，试一试：
println(df.show())
