java spark中的这个异常是什么意思
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/36325773/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
What is means by this exception in spark
提问by Farman Ali
16/03/30 23:23:20 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, 10.208.91.144): java.lang.NoClassDefFoundError: edu/stanford/nlp/trees/TreebankLanguagePack at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2499) at java.lang.Class.getDeclaredField(Class.java:1951) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.(ObjectStreamClass.java:468) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: edu.stanford.nlp.trees.TreebankLanguagePack at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 more
16/03/30 23:23:20 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 2, 10.208.91.144, PROCESS_LOCAL, 5942 bytes) 16/03/30 23:23:20 INFO TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) on executor 10.208.91.144: java.lang.NoClassDefFoundError (edu/stanford/nlp/trees/TreebankLanguagePack) [duplicate 1] 16/03/30 23:23:20 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 3, 10.208.91.144, PROCESS_LOCAL, 1435 bytes) 16/03/30 23:23:20 WARN TransportChannelHandler: Exception in connection from /10.208.91.144:61788 java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) 16/03/30 23:23:20 ERROR TaskSchedulerImpl: Lost executor 0 on 10.208.91.144: remote Rpc client disassociated 16/03/30 23:23:20 INFO TaskSetManager: Re-queueing tasks for 0 from TaskSet 0.0 16/03/30 23:23:20 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://[email protected]:61767] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 16/03/30 23:23:20 WARN TaskSetManager: Lost task 1.1 in stage 0.0 (TID 2, 10.208.91.144): ExecutorLostFailure (executor 0 lost) 16/03/30 23:23:20 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 3, 10.208.91.144): ExecutorLostFailure (executor 0 lost) 16/03/30 23:23:20 INFO DAGScheduler: Executor lost: 0 (epoch 0) 16/03/30 23:23:20 INFO BlockManagerMasterEndpoint: Trying to remove executor 0 from BlockManagerMaster. 16/03/30 23:23:20 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(0, 10.208.91.144, 61786) 16/03/30 23:23:20 INFO BlockManagerMaster: Removed 0 successfully in removeExecutor 16/03/30 23:23:20 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/0 is now EXITED (Command exited with code 50) 16/03/30 23:23:20 INFO SparkDeploySchedulerBackend: Executor app-20160330232314-0002/0 removed: Command exited with code 50 16/03/30 23:23:20 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0 16/03/30 23:23:20 INFO AppClient$ClientActor: Executor added: app-20160330232314-0002/1 on worker-20160330231130-10.208.91.144-61218 (10.208.91.144:61218) with 4 cores 16/03/30 23:23:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160330232314-0002/1 on hostPort 10.208.91.144:61218 with 4 cores, 512.0 MB RAM 16/03/30 23:23:20 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/1 is now RUNNING 16/03/30 23:23:20 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/1 is now LOADING 16/03/30 23:23:23 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://[email protected]:61815/user/Executor#-238863041]) with ID 1 16/03/30 23:23:23 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, 10.208.91.144, PROCESS_LOCAL, 1435 bytes) 16/03/30 23:23:23 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, 10.208.91.144, PROCESS_LOCAL, 5942 bytes) 16/03/30 23:23:24 INFO BlockManagerMasterEndpoint: Registering block manager 10.208.91.144:61834 with 265.4 MB RAM, BlockManagerId(1, 10.208.91.144, 61834) 16/03/30 23:23:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.208.91.144:61834 (size: 30.4 KB, free: 265.4 MB) 16/03/30 23:23:24 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5) on executor 10.208.91.144: java.lang.NoClassDefFoundError (edu/stanford/nlp/trees/TreebankLanguagePack) [duplicate 2] 16/03/30 23:23:24 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID 6, 10.208.91.144, PROCESS_LOCAL, 5942 bytes) 16/03/30 23:23:24 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4) on executor 10.208.91.144: java.lang.NoClassDefFoundError (edu/stanford/nlp/trees/TreebankLanguagePack) [duplicate 3] 16/03/30 23:23:24 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 7, 10.208.91.144, PROCESS_LOCAL, 1435 bytes) 16/03/30 23:23:25 WARN TransportChannelHandler: Exception in connection from /10.208.91.144:61835 java.io.IOException: An existing connection was forcibly closed by the remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) 16/03/30 23:23:25 ERROR TaskSchedulerImpl: Lost executor 1 on 10.208.91.144: remote Rpc client disassociated 16/03/30 23:23:25 INFO TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0 16/03/30 23:23:25 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 7, 10.208.91.144): ExecutorLostFailure (executor 1 lost) 16/03/30 23:23:25 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://[email protected]:61815] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 16/03/30 23:23:25 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 16/03/30 23:23:25 WARN TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6, 10.208.91.144): ExecutorLostFailure (executor 1 lost) 16/03/30 23:23:25 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 16/03/30 23:23:25 INFO TaskSchedulerImpl: Cancelling stage 0 16/03/30 23:23:25 INFO DAGScheduler: ResultStage 0 (saveAsTextFile at Polarity.java:62) failed in 8.085 s 16/03/30 23:23:25 INFO DAGScheduler: Job 0 failed: saveAsTextFile at Polarity.java:62, took 8.447334 s 16/03/30 23:23:25 INFO DAGScheduler: Executor lost: 1 (epoch 1) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 10.208.91.144): ExecutorLostFailure (executor 1 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Exception in thread "main" java.io.FileNotFoundException: G:\Workspace\DSS\TextFile\part-00000 (The system cannot find the file specified) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileInputStream.(FileInputStream.java:93) at java.io.FileReader.(FileReader.java:58) at com.Polarity.read(Polarity.java:94) at com.Polarity.main(Polarity.java:288) 16/03/30 23:23:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster. 16/03/30 23:23:25 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, 10.208.91.144, 61834) 16/03/30 23:23:25 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor 16/03/30 23:23:25 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/1 is now EXITED (Command exited with code 50) 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend: Executor app-20160330232314-0002/1 removed: Command exited with code 50 16/03/30 23:23:25 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1 16/03/30 23:23:25 INFO AppClient$ClientActor: Executor added: app-20160330232314-0002/2 on worker-20160330231130-10.208.91.144-61218 (10.208.91.144:61218) with 4 cores 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend: Granted executor ID app-20160330232314-0002/2 on hostPort 10.208.91.144:61218 with 4 cores, 512.0 MB RAM 16/03/30 23:23:25 INFO SparkContext: Invoking stop() from shutdown hook 16/03/30 23:23:25 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/2 is now RUNNING 16/03/30 23:23:25 INFO AppClient$ClientActor: Executor updated: app-20160330232314-0002/2 is now LOADING 16/03/30 23:23:25 INFO SparkUI: Stopped Spark web UI at http://10.208.91.144:404016/03/30 23:23:25 INFO DAGScheduler: Stopping DAGScheduler 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend: Shutting down all executors 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 16/03/30 23:23:25 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/03/30 23:23:25 INFO Utils: path = C:\Users\Developer\AppData\Local\Temp\spark-618a9039-a9d4-4fb2-bdc8-408d1e7f3c0e\blockmgr-4436a6d6-fca4-4190-ac2a-48c8ebd4e7db, already present as root for deletion. 16/03/30 23:23:25 INFO MemoryStore: MemoryStore cleared 16/03/30 23:23:25 INFO BlockManager: BlockManager stopped 16/03/30 23:23:25 INFO BlockManagerMaster: BlockManagerMaster stopped 16/03/30 23:23:25 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/03/30 23:23:25 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/03/30 23:23:25 INFO SparkContext: Successfully stopped SparkContext 16/03/30 23:23:25 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/03/30 23:23:25 INFO Utils: Shutdown hook called 16/03/30 23:23:25 INFO Utils: Deleting directory C:\Users\Developer\AppData\Local\Temp\spark-618a9039-a9d4-4fb2-bdc8-408d1e7f3c0e
16/03/30 23:23:20 警告 TaskSetManager：在 0.0 阶段丢失任务 1.0（TID 1，10.208.91.144）：java.lang.NoClassDefFoundError：edu/stanford/nlp/trees/TreebankLanguagePack at java.lang.Class getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2499) at java.lang.Class.getDeclaredField(Class.java:1951) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)在 java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72) 在 java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480) 在 java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468) 在 java .security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.(ObjectStreamClass.java:468) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365) at java.io。ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream) .java:1771) 在 java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 在 java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 在 java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)在 java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 在 java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 在 java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 在 java.io。 ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 在 java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 在 java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 在 java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 在 java。 io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java: 1350) 在 java.io.ObjectInputStream。readObject(ObjectInputStream.java:370) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl) .java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java: 1017) 在 java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) 在 java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 在 java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 在 java。 io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 在 java.io.ObjectInputStream。readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java) :1990) 在 java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 在 java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 在 java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 在 java .io.ObjectInputStream.readObject(ObjectInputStream.java:370) 在 org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69) 在 org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95) ) 在 org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) 在 org.apache.spark.scheduler.Task.run(Task.scala:70) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) 在 java.util.concurrent.ThreadPoolExecutor .runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 引起：java.lang。 ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController。 doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多58 更多58 更多58 更多58 更多apache.spark.scheduler.Task.run(Task.scala:70) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford .nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method)在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多apache.spark.scheduler.Task.run(Task.scala:70) 在 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford .nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method)在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java .util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader .findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java .util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader .findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多spark.executor.Executor$TaskRunner.run(Executor.scala:213) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366 ) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader .loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多spark.executor.Executor$TaskRunner.run(Executor.scala:213) 在 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) 在 java.lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366 ) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader .loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 引起：java .lang.ClassNotFoundException: edu.stanford.nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security .AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java) :358) ... 58 更多concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 引起：java .lang.ClassNotFoundException: edu.stanford.nlp.trees.TreebankLanguagePack 在 java.net.URLClassLoader$1.run(URLClassLoader.java:366) 在 java.net.URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security .AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java) :358) ... 58 更多lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net .URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader. java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多lang.Thread.run(Thread.java:745) 引起：java.lang.ClassNotFoundException：edu.stanford.nlp.trees.TreebankLanguagePack at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net .URLClassLoader$1.run(URLClassLoader.java:355) 在 java.security.AccessController.doPrivileged(Native Method) 在 java.net.URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader. java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多URLClassLoader.findClass(URLClassLoader.java:354) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:425) 在 java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 58 更多
run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) 16/03/30 23:23:20 错误 TaskSchedulerImpl：10.208.91.144 上丢失执行程序 0：远程 Rpc 客户端断开关联 16/ 03/30 23:23:20 INFO TaskSetManager：从 TaskSet 0.0 重新排队 0 的任务 16/03/30 23:23:20 警告 ReliableDeliverySupervisor：与远程系统关联 [akka.tcp://[email protected] 61767] 失败，地址现在门控 [5000] 毫秒。原因是：[已解除关联]。16/03/30 23:23:20 WARN TaskSetManager：在阶段 0.0 中丢失任务 1.1（TID 2，10.208.91.144）：ExecutorLostFailure（执行器 0 丢失）16/03/30 23:23:20 WARN TaskSetManager.1 WARN taskSet在阶段 0.0 (TID 3, 10.208.91.144): ExecutorLostFailure (executor 0 lost) 16/03/30 23:23:20 INFO DAGScheduler: Executor lost: 0 (epoch 0) 16/03/230:2 20 INFO BlockManagerMasterEndpoint：试图从 BlockManagerMaster 中删除 executor 0。16/03/30 23:23:20 INFO BlockManagerMasterEndpoint：删除块管理器 BlockManagerId(0, 10.208.91.144, 61786) 16/03/30 23:23:20 INFO BlockManagerMaster：在 removeExecutor3/16/16/16/16/302003 中成功删除 0 :23:20 INFO AppClient$ClientActor: Executor 更新: app-20160330232314-0002/0 现在退出 (命令退出代码 50) 16/03/30 23:23:20 INFO SparkDeploySchedulerBackend: Executor app-310203203删除：命令退出代码 50 16/03/30 23:23:20 错误 SparkDeploySchedulerBackend：要求删除不存在的执行程序 0 16/03/30 23:23:20 INFO AppClient$ClientActor：执行程序添加：app-201603134-232 worker-20160330231130-10.208.91.144-61218 (10.208.91.144:61218) 上的 0002/1 4 核 16/03/30 23:23：23:25 警告 ReliableDeliverySupervisor：与远程系统 [akka.tcp://[email protected]:61815] 的关联失败，地址现在被限制为 [5000] 毫秒。原因是：[已解除关联]。16/03/30 23:23:25 错误 TaskSetManager：阶段 0.0 中的任务 0 失败了 4 次；中止作业 16/03/30 23:23:25 WARN TaskSetManager：在阶段 0.0 中丢失任务 1.3（TID 6，10.208.91.144）：ExecutorLostFailure（执行器 1 丢失）16/03/30 23:23:23:23:25 TaskSet 0.0，其任务已全部完成，来自池 16/03/30 23:23:25 INFO TaskSchedulerImpl: Canceling stage 0 16/03/30 23:23:25 INFO DAGScheduler: ResultStage 0 (saveAsTextFile at Polarity.java:62 ) 在 8.085 s 16/03/30 23:23:25 失败 INFO DAGScheduler: Job 0 failed: saveAsTextFile at Polarity.java:62, 花了 8.447334 s 16/03/30 23:23:25 INFO DAGScheduler: FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileInputStream.(FileInputStream.java:93) at java.io.FileReader.(FileReader.java: 58) at com.Polarity.read(Polarity.java:94) at com.Polarity.main(Polarity.java:288) 16/03/30 23:23:25 INFO BlockManagerMasterEndpoint：试图从 BlockManagerMaster 中删除 executor 1。16/03/30 23:23:25 INFO BlockManagerMasterEndpoint：删除块管理器 BlockManagerId(1, 10.208.91.144, 61834) 16/03/30 23:23:25 INFO BlockManagerMaster：在 removeExecutor3/30203 中成功删除 1 :23:25 INFO AppClient$ClientActor: Executor 更新: app-20160330232314-0002/1 现在退出 (命令退出代码 50) 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend: Executor app-31203-2032移除：http://10.208.91.144:404016/03/30 23:23:25 INFO DAGScheduler：停止 DAGScheduler 16/03/30 23:23:25 INFO SparkDeploySchedulerBackend：关闭所有执行程序 16/03/30 23:23:25 INFO SparkScheduler 每个执行程序关闭执行程序down 16/03/30 23:23:25 INFO MapOutputTrackerMasterEndpoint：MapOutputTrackerMasterEndpoint 停止了！16/03/30 23:23:25 信息实用程序：路径 = C:\Users\Developer\AppData\Local\Temp\spark-618a9039-a9d4-4fb2-bdc8-408d1e7f3c0e\blockmgr-4436a6d419-fca2 48c8ebd4e7db，已作为 root 存在以进行删除。16/03/30 23:23:25 INFO MemoryStore：MemoryStore 清除 16/03/30 23:23:25 INFO BlockManager：BlockManager 停止 16/03/30 23:23:25 INFO BlockManagerMaster：BlockManagerMaster 停止 16/03/30 23:23:25 信息 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint：OutputCommitCoordinator 停止了！16/03/30 23:23：25 信息 RemoteActorRefProvider$RemotingTerminator：关闭远程守护进程。16/03/30 23:23:25 INFO SparkContext：成功停止 SparkContext 16/03/30 23:23:25 INFO RemoteActorRefProvider$RemotingTerminator：远程守护程序关闭；继续冲洗远程传输。16/03/30 23:23:25 INFO Utils：关闭钩子调用 16/03/30 23:23:25 INFO Utils：删除目录 C:\Users\Developer\AppData\Local\Temp\spark-618a9039-a9d4- 4fb2-bdc8-408d1e7f3c0e
this is my pom.xml
这是我的 pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Fyp</groupId>
<artifactId>DSS</artifactId>
<version>0.0.1-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.Polarity</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<!-- Import Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>jaws</groupId>
<artifactId>jaws</artifactId>
<version>1.2</version>
<type>jar</type>
<scope>system</scope>
<systemPath>G:/Workspace/DSS/lib/jaws-bin.jar</systemPath>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.3</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.5.2</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-parser</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.5.0</version>
<classifier>models</classifier>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
</project>
回答by ???v?т?
The first line of the stack trace tells you what the problem is.
堆栈跟踪的第一行告诉您问题所在。
java.lang.NoClassDefFoundError: edu/stanford/nlp/trees/TreebankLanguagePack at
If you search Google or StackOverflow for NoClassDefFoundError, you'll find articles describing how to fix this problem. It's not specific to Apache Spark. It's a general Java problem. In all likelihood your Java classpath is incorrectly set up---you're missing a "jar" file on the classpath, have the wrong version of a jar or have not included all the class files on the classpath.
如果您在 Google 或 StackOverflow 上搜索NoClassDefFoundError，您会找到描述如何解决此问题的文章。它不是特定于 Apache Spark。这是一个普遍的 Java 问题。很可能您的 Java 类路径设置不正确——类路径上缺少“jar”文件、jar 版本错误或未包含类路径中的所有类文件。
You probably need to work out which "jar" contains the class edu.stanford.nlp.trees.TreebankLanguagePack. Searching Google for that class name, suggests it's the stanford-parser.jarthat you are missing. You may be missing other jars too.
您可能需要确定哪个“jar”包含 class edu.stanford.nlp.trees.TreebankLanguagePack。在谷歌上搜索那个班级名称，表明它stanford-parser.jar是你所缺少的。您也可能缺少其他罐子。
UPDATE: Now you've posted your maven config. I think you have specified an old version of stanford-parser.jar that does not contain the TreebankLanguagePackclass. Try this instead:
更新：现在您已经发布了您的 Maven 配置。我认为您指定了不包含TreebankLanguagePack该类的旧版本 stanford-parser.jar 。试试这个：
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-parser</artifactId>
<version>3.6.0</version>
</dependency>
回答by Farman Ali
I have solved this issue. The error is due to missing jar files that are not attached in SparkContext.
我已经解决了这个问题。该错误是由于缺少 SparkContext 中未附加的 jar 文件。
Here is the jars that I added :
这是我添加的罐子：
String jars[]={"lib/stanford-corenlp-1.3.5.jar","lib/stanford-parse-models-1.3.2.jar","lib/stanford-parser.jar","lib/stanford-parser-3.5.2-javadoc.jar","lib/stanford-postagger.jar","lib/stanford-postagger-3.5.2.jar","lib/stanford-postagger-3.5.2-javadoc.jar","lib/org.springframework.core-3.0.3.RELEASE.jar"};
SparkConf sparkConf = new SparkConf().setAppName("DSS").setMaster("spark://192.168.1.100:7077").setJars(jars);
回答by Manas
There is a better way to solve problems like this. Using sbt assembly (Or onejar) you can create single fat jar for your spark application.
有更好的方法来解决此类问题。使用 sbt assembly（或 onejar），您可以为 Spark 应用程序创建单个 fat jar。
AFAIK this is the default choice everywhere.
AFAIK 这是任何地方的默认选择。
