scala 如何将 spark 数据框中的 WrappedArray 列转换为字符串？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/34539068/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How do I convert a WrappedArray column in spark dataframe to Strings?
提问by bdguy
I am trying to convert a column which contains Array[String] to String, but I consistently get this error
我正在尝试将包含 Array[String] 的列转换为 String，但我始终收到此错误
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 78.0 failed 4 times, most recent failure: Lost task 0.3 in stage 78.0 (TID 1691, ip-******): java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Ljava.lang.String;
Here's the piece of code
这是一段代码
val mkString = udf((arrayCol:Array[String])=>arrayCol.mkString(","))
val dfWithString=df.select($"arrayCol").withColumn("arrayString",
mkString($"arrayCol"))
回答by zero323
WrappedArrayis not an Array(which is plain old Java Arraynot a natve Scala collection). You can either change signature to:
WrappedArray不是Array（这是普通的旧 Java，Array而不是原生的 Scala 集合）。您可以将签名更改为：
import scala.collection.mutable.WrappedArray
(arrayCol: WrappedArray[String]) => arrayCol.mkString(",")
import scala.collection.mutable.WrappedArray
(arrayCol: WrappedArray[String]) => arrayCol.mkString(",")
or use one of the supertypes like Seq:
或使用超类型之一，如Seq：
(arrayCol: Seq[String]) => arrayCol.mkString(",")
In the recent Spark versions you can use concat_wsinstead:
在最近的 Spark 版本中，您可以concat_ws改用：
import org.apache.spark.sql.functions.concat_ws
df.select(concat_ws(",", $"arrayCol"))
回答by Burt
The code work for me:
代码对我有用：
df.select("wifi_ids").rdd.map(row =>row.get(0).asInstanceOf[WrappedArray[WrappedArray[String]]].toSeq.map(x=>x.toSeq.apply(0)))
In your case,I guess it is:
在你的情况下，我想是：
val mkString = udf(arrayCol=>arrayCol.asInstanceOf[WrappedArray[String]].toArray.mkString(","))
val dfWithString=df.select($"arrayCol").withColumn("arrayString",mkString($"arrayCol"))
