Spark从未停止过对第一批产品的处理

问题描述 投票:0回答:1

我正在尝试运行一个我在github上找到的应用程序,这个。https:/github.comCSIRT-MUAIDA-Framework。

我在Ubuntu 18.04.1虚拟机中运行它。在它的数据处理管道中的某个点上,它使用了spark,似乎在这个点上卡住了。我可以从Web用户界面上看到,我在那里发送的一些数据被作为一个批次接收。然而,它似乎永远不会完成处理第一个批次(即使它有0条记录)。不幸的是,我没有使用火花的经验,不知道到底是什么故障。在搜索修复方法的时候,我遇到了一些建议,认为可能是所有执行者的核心数不够。我试着将核心增加到3个,但这并没有帮助。

我已经提供了所有的Web用户界面的屏幕,我希望它们能足够清晰地显示问题。有谁知道我在这里做错了什么?

截图。Spark 1 火花2 火花3 火花4 火花5 火花6

排队和未完成的批处理作业的输出为

callForeachRDD at NativeMethodAccessorImpl.java:0
org.apache.spark.streaming.api.python.PythonDStream.callForeachRDD(PythonDStream.scala)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    py4j.Gateway.invoke(Gateway.java:282)
    py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    py4j.commands.CallCommand.execute(CallCommand.java:79)
    py4j.GatewayConnection.run(GatewayConnection.java:238)
    java.lang.Thread.run(Thread.java:748)

EDIT:我注意到,当进程启动时,会记录错误。我现在才意识到,因为进程没有停止。这些错误是。

May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

谁能帮我解决这些错误?

apache-spark dstream
1个回答
0
投票

Kafka与spark jars有冲突的依赖关系......所以,要么不使用lz压缩,而使用snappy压缩,就可以工作了

或者按照答案 此处 用于解决冲突的罐子。

© www.soinside.com 2019 - 2024. All rights reserved.