Python 如何在pyspark中估计数据帧的实际大小？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/37077432/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to estimate dataframe real size in pyspark?
提问by TheSilence
How to determine a dataframe size?
如何确定数据帧大小？
Right now I estimate the real size of a dataframe as follows:
现在我估计数据帧的实际大小如下：
headers_size = key for key in df.first().asDict()
rows_size = df.map(lambda row: len(value for key, value in row.asDict()).sum()
total_size = headers_size + rows_size
It is too slow and I'm looking for a better way.
它太慢了，我正在寻找更好的方法。
回答by Ziggy Eunicien
nice post from Tamas Szuromi http://metricbrew.com/how-to-estimate-rdd-or-dataframe-real-size-in-pyspark/
来自 Tamas Szuromi 的好帖子http://metricbrew.com/how-to-estimate-rdd-or-dataframe-real-size-in-pyspark/
from pyspark.serializers import PickleSerializer, AutoBatchedSerializer
def _to_java_object_rdd(rdd):
""" Return a JavaRDD of Object by unpickling
It will convert each Python object into Java object by Pyrolite, whenever the
RDD is serialized in batch or not.
"""
rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer()))
return rdd.ctx._jvm.org.apache.spark.mllib.api.python.SerDe.pythonToJava(rdd._jrdd, True)
JavaObj = _to_java_object_rdd(df.rdd)
nbytes = sc._jvm.org.apache.spark.util.SizeEstimator.estimate(JavaObj)
回答by Kiran Thati
Currently I am using the below approach, not sure if this is the best way
目前我正在使用以下方法，不确定这是否是最好的方法
df.persist(StorageLevel.Memory)
df.count()
df.persist(StorageLevel.Memory)
df.count()
On the spark-web ui under the Storage tab you can check the size which is displayed in MB's and then I do unpersist to clear the memory.
在存储选项卡下的 spark-web ui 上，您可以检查以 MB 为单位显示的大小，然后我不坚持清除内存。
df.unpersist()
df.unpersist()
