如何在 Java 的 Apache Spark 中将 DataFrame 转换为 Dataset？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/34654145/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to convert DataFrame to Dataset in Apache Spark in Java?
提问by Milad Khajavi
I can convert DataFrame to Dataset in Scala very easy:
我可以很容易地在 Scala 中将 DataFrame 转换为 Dataset：
case class Person(name:String, age:Long)
val df = ctx.read.json("/tmp/persons.json")
val ds = df.as[Person]
ds.printSchema
but in Java version I don't know how to convert Dataframe to Dataset? Any Idea?
但是在 Java 版本中我不知道如何将 Dataframe 转换为 Dataset？任何的想法？
my effort is:
我的努力是：
DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);
ds.printSchema();
but the compiler say:
但编译器说：
Error:(23, 27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated
Edited(Solution):
编辑（解决方案）：
solution based on @Leet-Falcon answers:
基于@Leet-Falcon 答案的解决方案：
DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx, df.logicalPlan(), encoder);
采纳答案by Leet-Falcon
Official Spark docs suggest in Dataset APIthe following:
官方 Spark 文档在Dataset API 中建议如下：
Java Encoders are specified by calling static methods on Encoders.
Java 编码器是通过在Encoders上调用静态方法来指定的。
List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());
Encoders can be composed into tuples:
编码器可以组合成元组：
Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);
Or constructed from Java Beans by Encoders#bean:
或者由Encoders#bean从 Java Bean 构建：
Encoders.bean(MyClass.class);
回答by nomad
If you want to convert a generic DF to a Dataset in Java, you can use RowEncoder class like below
如果要将通用 DF 转换为 Java 中的数据集，可以使用如下所示的 RowEncoder 类
DataFrame df = sql.read().json(sc.parallelize(ImmutableList.of(
"{\"id\": 0, \"phoneNumber\": 109, \"zip\": \"94102\"}"
)));
Dataset<Row> dataset = df.as(RowEncoder$.MODULE$.apply(df.schema()));
