为什么 apache spark 不适用于 java 10？我们得到非法反射然后 java.lang.IllegalArgumentException

Why apache spark does not work with java 10? We get illegal reflective then java.lang.IllegalArgumentException

提问by mehdi

Is there any technical reason why spark 2.3 does not work with java 1.10 (as of July 2018)?

spark 2.3 不适用于 java 1.10（截至 2018 年 7 月）是否有任何技术原因？

Here is the output when I run SparkPi example using spark-submit.

这是我使用spark-submit.

$ ./bin/spark-submit ./examples/src/main/python/pi.py 
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2018-07-13 14:31:30 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-07-13 14:31:31 INFO  SparkContext:54 - Running Spark version 2.3.1
2018-07-13 14:31:31 INFO  SparkContext:54 - Submitted application: PythonPi
2018-07-13 14:31:31 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 58681.
2018-07-13 14:31:31 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-07-13 14:31:31 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-07-13 14:31:31 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-07-13 14:31:31 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-07-13 14:31:31 INFO  DiskBlockManager:54 - Created local directory at /private/var/folders/mp/9hp4l4md4dqgmgyv7g58gbq0ks62rk/T/blockmgr-d24fab4c-c858-4cd8-9b6a-97b02aa630a5
2018-07-13 14:31:31 INFO  MemoryStore:54 - MemoryStore started with capacity 434.4 MB
2018-07-13 14:31:31 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
...
2018-07-13 14:31:32 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
Traceback (most recent call last):
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/./examples/src/main/python/pi.py", line 44, in <module>
    count = spark.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 862, in reduce
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 834, in collect
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "~/Documents/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.IllegalArgumentException
    at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
    at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
    at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source)
    at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:46)
    at org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:449)
    at org.apache.spark.util.FieldAccessFinder$$anon$$anonfun$visitMethodInsn.apply(ClosureCleaner.scala:432)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach.apply(TraversableLike.scala:733)
    at scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103)
    at scala.collection.mutable.HashMap$$anon$$anonfun$foreach.apply(HashMap.scala:103)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap$$anon.foreach(HashMap.scala:103)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.apache.spark.util.FieldAccessFinder$$anon.visitMethodInsn(ClosureCleaner.scala:432)
    at org.apache.xbean.asm5.ClassReader.a(Unknown Source)
    at org.apache.xbean.asm5.ClassReader.b(Unknown Source)
    at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
    at org.apache.xbean.asm5.ClassReader.accept(Unknown Source)
    at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:262)
    at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean.apply(ClosureCleaner.scala:261)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:261)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2299)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2073)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
    at org.apache.spark.rdd.RDD$$anonfun$collect.apply(RDD.scala:939)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:162)
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:844)

2018-07-13 14:31:33 INFO  SparkContext:54 - Invoking stop() from shutdown hook
...

I resolved the issue by switching to Java8 instead of Java10 as mentioned here.

我通过切换到 Java8 而不是这里提到的 Java10 解决了这个问题。

回答by user10077548

Primary technical reason is that Spark depends heavily on direct access to native memory with sun.misc.Unsafe, which has been made private in Java 9.

主要的技术原因是 Spark 严重依赖于使用 直接访问本机内存sun.misc.Unsafe，这在 Java 9 中已成为私有。

• https://issues.apache.org/jira/browse/SPARK-24421
• http://apache-spark-developers-list.1001551.n3.nabble.com/Java-9-td20875.html

• https://issues.apache.org/jira/browse/SPARK-24421
• http://apache-spark-developers-list.1001551.n3.nabble.com/Java-9-td20875.html

回答by Sean Owen

Committer here. It's actually a fair bit of work to support Java 9+: SPARK-24417

提交在这里。支持 Java 9+ 实际上是相当多的工作：SPARK-24417

It's also almost done and should be ready for Spark 3.0, which should run on Java 8 through 11 and beyond.

它也快完成了，应该可以用于 Spark 3.0，它应该可以在 Java 8 到 11 及更高版本上运行。

The goal (well, mine) is to make it work without opening up module access. The key issues include:

目标（嗯，我的）是在不开放模块访问的情况下使其工作。关键问题包括：

• sun.misc.Unsafeusage has to be removed or worked around
• Changes to the structure of boot classloader
• Scala support for Java 9+
• A bunch of dependency updates to work with Java 9+
• JAXB no longer automatically available

• sun.misc.Unsafe必须删除或解决使用情况
• 对引导类加载器结构的更改
• Scala 对 Java 9+ 的支持
• 一堆依赖更新以使用 Java 9+
• JAXB 不再自动可用

回答by KayV

Spark depends on the memory API's which has been changed in JDK 9 so it is not available starting JDK 9.

Spark 依赖于 JDK 9 中已更改的内存 API，因此从 JDK 9 开始它不可用。

And that is the reason for this.

这就是原因。

Please check the issue:

请检查问题：

https://issues.apache.org/jira/browse/SPARK-24421

https://issues.apache.org/jira/browse/SPARK-24421