如何在列数据 Spark scala 上检查 isEmpty

How to check isEmpty on Column Data Spark scala

提问by Swetha

My data looks like :

我的数据看起来像：

[null,223433,WrappedArray(),null,460036382,0,home,home,home]

How do I check if the col3 is empty on query in spark sql ? I tried to explode but when I do that the empty array rows are disappearing. Can some suggest me a way to do this.

如何在 spark sql 中查询时检查 col3 是否为空？我试图爆炸，但是当我这样做时，空数组行正在消失。有人可以建议我这样做的方法。

I tried :

我试过 ：

val homeSet = result.withColumn("subscriptionProvider", explode($"subscriptionProvider"))

where subscriptionProvider(WrappedArray())is the column having array of values but some arrays can be empty. I need to get the subscriptionProvider with null values and subscriptionProvider array has "Comcast"

subscriptionProvider(WrappedArray())具有值数组的列在哪里，但某些数组可以为空。我需要使用空值获取 subscriptionProvider 并且 subscriptionProvider 数组具有“Comcast”

回答by

Try:

尝试：

import org.apache.spark.sql.functions._

val tmp = df.withColumn("subscriptionProvider", 
  when(size($"subscriptionProvider") !== 0, $"subscriptionProvider").otherwise(array(lit(null).cast("string"))))

tmp.withColumn("subscriptionProvider", explode($"subscriptionProvider"))

回答by Justin Pihony

LostInOverflow's answer is good for keeping in the dataframe mindset. However it depends on the size of your lists as to whether sizeis efficient. If you are going to have large lists, then dropping out and back into the dataframe might be best:

LostInOverflow 的答案有利于保持数据框的心态。但是，它是否size有效取决于列表的大小。如果您要拥有大型列表，那么退出并返回数据帧可能是最好的：

val dfSchema = df.schema
val filtered = df.rdd.filter(!_.getList[String](2).isEmpty)
sqlContext.createDataFrame(filtered, dfSchema)