Python Spark DataFrame TimestampType - 如何从字段中获取年、月、日值？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/30949202/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Spark DataFrame TimestampType - how to get Year, Month, Day values from field?
提问by curtisp
I have Spark DataFrame with take(5) top rows as follows:
我有带 take(5) 顶行的 Spark DataFrame，如下所示：
[Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=1, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=2, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=3, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=4, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=5, value=638.55)]
It's schema is defined as:
它的模式定义为：
elevDF.printSchema()
root
|-- date: timestamp (nullable = true)
|-- hour: long (nullable = true)
|-- value: double (nullable = true)
How do I get the Year, Month, Day values from the 'date' field?
如何从“日期”字段中获取年、月、日值？
采纳答案by zero323
Since Spark 1.5 you can use a number of date processing functions:
从 Spark 1.5 开始，您可以使用许多日期处理函数：
pyspark.sql.functions.yearpyspark.sql.functions.monthpyspark.sql.functions.dayofmonthpyspark.sql.functions.dayofweek()pyspark.sql.functions.dayofyearpyspark.sql.functions.weekofyear()
pyspark.sql.functions.yearpyspark.sql.functions.monthpyspark.sql.functions.dayofmonthpyspark.sql.functions.dayofweek()pyspark.sql.functions.dayofyearpyspark.sql.functions.weekofyear()
import datetime
from pyspark.sql.functions import year, month, dayofmonth
elevDF = sc.parallelize([
(datetime.datetime(1984, 1, 1, 0, 0), 1, 638.55),
(datetime.datetime(1984, 1, 1, 0, 0), 2, 638.55),
(datetime.datetime(1984, 1, 1, 0, 0), 3, 638.55),
(datetime.datetime(1984, 1, 1, 0, 0), 4, 638.55),
(datetime.datetime(1984, 1, 1, 0, 0), 5, 638.55)
]).toDF(["date", "hour", "value"])
elevDF.select(
year("date").alias('year'),
month("date").alias('month'),
dayofmonth("date").alias('day')
).show()
#?+----+-----+---+
#?|year|month|day|
#?+----+-----+---+
#?|1984| 1| 1|
#?|1984| 1| 1|
#?|1984| 1| 1|
#?|1984| 1| 1|
#?|1984| 1| 1|
#?+----+-----+---+
You can use simple mapas with any other RDD:
你可以map像任何其他 RDD 一样使用 simple ：
elevDF = sqlContext.createDataFrame(sc.parallelize([
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=1, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=2, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=3, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=4, value=638.55),
Row(date=datetime.datetime(1984, 1, 1, 0, 0), hour=5, value=638.55)]))
(elevDF
.map(lambda (date, hour, value): (date.year, date.month, date.day))
.collect())
and the result is:
结果是：
[(1984, 1, 1), (1984, 1, 1), (1984, 1, 1), (1984, 1, 1), (1984, 1, 1)]
Btw: datetime.datetimestores an hour anyway so keeping it separately seems to be a waste of memory.
顺便说一句：datetime.datetime无论如何都要存储一个小时，所以单独保存它似乎是在浪费内存。
回答by hamed
You can use functions in pyspark.sql.functions: functions like year, month, etc
您可以在pyspark.sql.functions以下函数中使用函数：year, month,etc
refer to here: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame
参考这里：https: //spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame
from pyspark.sql.functions import *
newdf = elevDF.select(year(elevDF.date).alias('dt_year'), month(elevDF.date).alias('dt_month'), dayofmonth(elevDF.date).alias('dt_day'), dayofyear(elevDF.date).alias('dt_dayofy'), hour(elevDF.date).alias('dt_hour'), minute(elevDF.date).alias('dt_min'), weekofyear(elevDF.date).alias('dt_week_no'), unix_timestamp(elevDF.date).alias('dt_int'))
newdf.show()
+-------+--------+------+---------+-------+------+----------+----------+
|dt_year|dt_month|dt_day|dt_dayofy|dt_hour|dt_min|dt_week_no| dt_int|
+-------+--------+------+---------+-------+------+----------+----------+
| 2015| 9| 6| 249| 0| 0| 36|1441497601|
| 2015| 9| 6| 249| 0| 0| 36|1441497601|
| 2015| 9| 6| 249| 0| 0| 36|1441497603|
| 2015| 9| 6| 249| 0| 1| 36|1441497694|
| 2015| 9| 6| 249| 0| 20| 36|1441498808|
| 2015| 9| 6| 249| 0| 20| 36|1441498811|
| 2015| 9| 6| 249| 0| 20| 36|1441498815|
