java.lang.NoSuchMethodError在flink进程中使用Java反射的异常

问题描述 投票:1回答:1

我正在尝试使用谷歌反射库扩展flink过程的业务逻辑。我在运行时遇到了这个异常。有人试验过同样的问题。我怀疑是maven冲突,但我不知道。我粘贴了异常和pom文件。

pom文件:

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>it.almaviva.wtf.mms</groupId>
    <artifactId>mms-integretedmobilitystatusevent-flink-process</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <name>Passing Event Process</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <flink.version>1.4.0</flink.version>
        <!--<slf4j.version>1.7.7</slf4j.version>-->
        <log4j.version>1.2.17</log4j.version>
        <scala.binary.version>2.11</scala.binary.version>
        <junit.version>4.12</junit.version>
        <cassandra.version>3.2.0</cassandra.version>
        <flink.connector.elastic.version>1.4.0</flink.connector.elastic.version>
        <thyco.compiler.version>0.21.0</thyco.compiler.version>
        <jackson.version>2.9.4</jackson.version>
        <cassandra.unit.version>3.3.0.2</cassandra.unit.version>
    </properties>

    <repositories>
        <repository>
            <id>apache.snapshots</id>
            <name>Apache Development Snapshot Repository</name>
            <url>https://repository.apache.org/content/repositories/snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
            </snapshots>
        </repository>
    </repositories>

    <!-- Execute "mvn clean package -Pbuild-jar" to build a jar file out of
              this project! How to use the Flink Quickstart pom: a) Adding new dependencies:
              You can add dependencies to the list below. Please check if the maven-shade-plugin
              below is filtering out your dependency and remove the exclude from there.
              b) Build a jar for running on the cluster: There are two options for creating
              a jar from this project b.1) "mvn clean package" -> this will create a fat
              jar which contains all dependencies necessary for running the jar created
              by this pom in a cluster. The "maven-shade-plugin" excludes everything that
              is provided on a running Flink cluster. b.2) "mvn clean package -Pbuild-jar"
              -> This will also create a fat-jar, but with much nicer dependency exclusion
              handling. This approach is preferred and leads to much cleaner jar files. -->

    <dependencies>
        <!-- Apache Flink dependencies -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-core</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- This dependency is required to actually execute jobs. It is currently pulled in by
flink-streaming-java, but we explicitly depend on it to safeguard against future changes. -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <!-- explicitly add a standard loggin framework, as Flink does not have
                      a hard dependency on one specific framework by default -->

        <!-- https://mvnrepository.com/artifact/ch.qos.logback/logback-classic -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <!-- TEST PURPOSE -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>
        <dependency>
            <groupId>org.cassandraunit</groupId>
            <artifactId>cassandra-unit-shaded</artifactId>
            <version>3.3.0.2</version>
        </dependency>

        <!-- DATASTAX -->

        <dependency>
            <groupId>com.datastax.cassandra</groupId>
            <artifactId>cassandra-driver-extras</artifactId>
            <version>${cassandra.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
                <exclusion>
                    <artifactId>netty-handler</artifactId>
                    <groupId>io.netty</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.datastax.cassandra</groupId>
            <artifactId>cassandra-driver-mapping</artifactId>
            <version>${cassandra.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
            <version>${flink.connector.elastic.version}</version>
            <scope>compile</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-core -->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>5.1.2</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.1.2</version>
        </dependency>

        <dependency>
            <groupId>com.carrotsearch</groupId>
            <artifactId>hppc</artifactId>
            <version>0.7.0</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-cep_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-statebackend-rocksdb_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-cassandra_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.6.1</version>
        </dependency>
        <dependency>
            <groupId>io.flinkspector</groupId>
            <artifactId>flinkspector-datastream_2.11</artifactId>
            <version>0.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.reflections</groupId>
            <artifactId>reflections</artifactId>
            <version>0.9.11</version>
            <exclusions>
                <exclusion>
                    <artifactId>guava</artifactId>
                    <groupId>com.google.guava</groupId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <profiles>
        <profile>
            <!-- Profile for packaging correct JAR files -->
            <id>build-jar</id>

            <activation>
                <activeByDefault>false</activeByDefault>
            </activation>

            <dependencies>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-core</artifactId>
                    <version>${flink.version}</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-java</artifactId>
                    <version>${flink.version}</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-clients_${scala.binary.version}</artifactId>
                    <version>${flink.version}</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>org.apache.flink</groupId>
                    <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
                    <version>${flink.version}</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                    <version>${log4j.version}</version>
                    <scope>provided</scope>
                </dependency>
                <!-- DTO -->
                <dependency>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                    <version>${jackson.version}</version>
                </dependency>
                <!-- APACHE CASSANDRA -->
                <dependency>
                    <groupId>com.datastax.cassandra</groupId>
                    <artifactId>cassandra-driver-extras</artifactId>
                    <version>${cassandra.version}</version>
                    <exclusions>
                        <exclusion>
                            <groupId>io.netty</groupId>
                            <artifactId>*</artifactId>
                        </exclusion>
                    </exclusions>
                </dependency>
                <dependency>
                    <groupId>com.datastax.cassandra</groupId>
                    <artifactId>cassandra-driver-mapping</artifactId>
                    <version>${cassandra.version}</version>
                    <exclusions>
                        <exclusion>
                            <groupId>io.netty</groupId>
                            <artifactId>*</artifactId>
                        </exclusion>
                    </exclusions>
                </dependency>
                <dependency>
                    <groupId>org.reflections</groupId>
                    <artifactId>reflections</artifactId>
                    <version>0.9.11</version>
                </dependency>

            </dependencies>

            <build>
                <plugins>
                    <!-- disable the exclusion rules -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-shade-plugin</artifactId>
                        <version>2.4.1</version>
                        <executions>
                            <execution>
                                <phase>package</phase>
                                <goals>
                                    <goal>shade</goal>
                                </goals>
                                <configuration>
                                    <artifactSet>
                                        <excludes combine.self="override"/>
                                    </artifactSet>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>

    <build>
        <plugins>
            <!-- We use the maven-shade plugin to create a fat jar that contains all
          dependencies except flink and it's transitive dependencies. The resulting
          fat-jar can be executed on a cluster. Change the value of Program-Class if
          your program entry point changes. -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <!-- Run shade goal on package phase -->
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <!-- This list contains all dependencies of flink-dist Everything
                      else will be packaged into the fat-jar -->
                                    <exclude>org.apache.flink:flink-annotations</exclude>
                                    <exclude>org.apache.flink:flink-shaded-hadoop1_2.10</exclude>
                                    <exclude>org.apache.flink:flink-shaded-hadoop2</exclude>
                                    <exclude>org.apache.flink:flink-shaded-curator-recipes</exclude>
                                    <exclude>org.apache.flink:flink-core</exclude>
                                    <exclude>org.apache.flink:flink-java</exclude>
                                    <!--                                                                    <exclude>org.apache.flink:flink-scala_2.10</exclude> -->
                                    <exclude>org.apache.flink:flink-runtime_2.10</exclude>
                                    <exclude>org.apache.flink:flink-optimizer_2.10</exclude>
                                    <exclude>org.apache.flink:flink-clients_2.10</exclude>
                                    <exclude>org.apache.flink:flink-avro_2.10</exclude>
                                    <exclude>org.apache.flink:flink-examples-batch_2.10</exclude>
                                    <exclude>org.apache.flink:flink-examples-streaming_2.10</exclude>
                                    <exclude>org.apache.flink:flink-streaming-java_2.10</exclude>

                                    <!-- Also exclude very big transitive dependencies of Flink WARNING:
                      You have to remove these excludes if your code relies on other versions of
                      these dependencies. -->
                                    <exclude>org.scala-lang:scala-library</exclude>
                                    <exclude>org.scala-lang:scala-compiler</exclude>
                                    <exclude>org.scala-lang:scala-reflect</exclude>
                                    <exclude>com.typesafe.akka:akka-actor_*</exclude>
                                    <exclude>com.typesafe.akka:akka-remote_*</exclude>
                                    <!--<exclude>io.netty:netty-all</exclude>-->
                                    <exclude>io.netty:netty</exclude>
                                    <exclude>commons-fileupload:commons-fileupload</exclude>
                                    <exclude>org.apache.avro:avro</exclude>
                                    <exclude>commons-collections:commons-collections</exclude>
                                    <exclude>org.codehaus.jackson:jackson-core-asl</exclude>
                                    <exclude>org.codehaus.jackson:jackson-mapper-asl</exclude>
                                    <exclude>com.thoughtworks.paranamer:paranamer</exclude>
                                    <exclude>org.xerial.snappy:snappy-java</exclude>
                                    <exclude>org.apache.commons:commons-compress</exclude>
                                    <exclude>org.tukaani:xz</exclude>
                                    <exclude>com.esotericsoftware.kryo:kryo</exclude>
                                    <exclude>com.esotericsoftware.minlog:minlog</exclude>
                                    <exclude>org.objenesis:objenesis</exclude>
                                    <exclude>com.twitter:chill_*</exclude>
                                    <exclude>com.twitter:chill-java</exclude>
                                    <exclude>commons-lang:commons-lang</exclude>
                                    <exclude>junit:junit</exclude>
                                    <exclude>org.apache.commons:commons-lang3</exclude>
                                    <exclude>log4j:log4j</exclude>
                                    <exclude>org.apache.commons:commons-math</exclude>
                                    <exclude>org.apache.sling:org.apache.sling.commons.json</exclude>
                                    <exclude>commons-logging:commons-logging</exclude>
                                    <exclude>commons-codec:commons-codec</exclude>
                                    <exclude>stax:stax-api</exclude>
                                    <exclude>com.typesafe:config</exclude>
                                    <exclude>org.uncommons.maths:uncommons-maths</exclude>
                                    <exclude>com.github.scopt:scopt_*</exclude>
                                    <exclude>commons-io:commons-io</exclude>
                                    <exclude>commons-cli:commons-cli</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>org.apache.flink:*</artifact>
                                    <excludes>
                                        <!-- exclude shaded google but include shaded curator -->
                                        <exclude>org/apache/flink/shaded/com/**</exclude>
                                        <exclude>web-docs/**</exclude>
                                    </excludes>
                                </filter>
                                <filter>
                                    <!-- Do not copy the signatures in the META-INF folder. Otherwise,
                      this might cause SecurityExceptions when using the JAR. -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <!-- If you want to use ./bin/flink run <quickstart jar> uncomment
                  the following lines. This will add a Main-Class entry to the manifest file -->
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>it.almaviva.wtf.mms.integratemobilitystatusevent.PassingEventProcess
                                    </mainClass>
                                </transformer>
                            </transformers>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>

        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.7.0</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                        <compilerId>jdt</compilerId>
                    </configuration>
                    <dependencies>
                        <dependency>
                            <groupId>org.eclipse.tycho</groupId>
                            <artifactId>tycho-compiler-jdt</artifactId>
                            <version>0.21.0</version>
                        </dependency>
                    </dependencies>
                </plugin>

                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>org.apache.maven.plugins</groupId>
                                        <artifactId>maven-assembly-plugin</artifactId>
                                        <versionRange>[2.4,)</versionRange>
                                        <goals>
                                            <goal>single</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore/>
                                    </action>
                                </pluginExecution>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>org.apache.maven.plugins</groupId>
                                        <artifactId>maven-compiler-plugin</artifactId>
                                        <versionRange>[3.1,)</versionRange>
                                        <goals>
                                            <goal>testCompile</goal>
                                            <goal>compile</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore/>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

错误:

22:24:56.552 [main] INFO org.reflections.Reflections - 反射需要370毫秒来扫描1个网址,在线程“main”java.lang.NoSuchMethodError:com.google.common.collect中生成16个键和52个值Exception。设置$ SetView.iterator()LCOM /谷歌/普通/收集/ UnmodifiableIterator; at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)atg.reflections.Reflections。(Reflections.java:126)atg.reflections.Reflections。(Reflections.java:168)atg.reflections.Reflections。 (Reflections.java:141)at it.almaviva.wtf.mms.integratemobilitystatusevent.PassingEventProcess.preferredCustomLogicMethod(PassingEventProcess.java:213)it ital.jpg,。

java maven reflection apache-flink
1个回答
0
投票

我很确定这是非Flink相关的问题。由反射0.9.11引起的问题方法SetsView#iterator来自Guava 20.0,你在你的pom中排除,而cassandra-driver-extras拉入Guava 19(doesn't have this method)。你可以通过执行以下方式检查:

mvn dependency:tree

你必须以某种方式解决cassandra和反射依赖之间的这种冲突。您可以尝试将guava版本强制为20.0,删除cassandra-driver-extras依赖项或查找没有guava依赖项的cassandra-driver-extras版本(Flink隐藏它的Guava依赖性,因此Flink的用户不必处理此问题)。

© www.soinside.com 2019 - 2024. All rights reserved.