我试图使用以下方法读取文件:
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");
我收到以下错误:
java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found
我使用的是flink 1.3.2版,java版“1.8.0_91”
与依赖项存在冲突。 Apache Flink默认将许多类加载到其类路径中。
请阅读本文https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html的最后一节
使用maven-shade-plugin解决与Flink的依赖冲突
Apache Flink默认将许多类加载到其类路径中。如果用户经常使用Flink正在使用的库的不同版本
IllegalAccessExceptions
要么
NoSuchMethodError
是结果。
所以,我建议你使用你的pom.xml并使用maven-shade-plugin并添加正确的重定位,就像我们在示例中所做的那样
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
您是否在IntelliJ或Dashboard中收到此错误,如果您在IntelliJ中收到此错误,请确保在pom.xml中使用相同的Flink版本,并在构建中添加依赖关系着色,如下所示
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>>
确保在进行更改后在终端中运行maven clean install。另一方面,如果你只在仪表板中有这个问题而不是在intelliJ中,那么看看here
您需要检查构建路径,确保libs存在并正确导入它们
错误“java.lang.NoSuchMethodError”的一个可能原因是当我们使用不同版本的flink时,我们在系统上安装了什么。对我来说,我有Flink 1.3.2,而我使用的版本是1.1.1。所以我更新了我的pom文件以获得相同的版本。
对于那些正在努力解决上述问题的人,在使用qazxsw poi时,这里是我能够成功构建的整个pom:
Flink 1.3.2
请在<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>FlinktoLambda</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<!--added newly-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerId>jdt</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-jdt</artifactId>
<version>0.21.0</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>FlinktoLambda</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
<relocations>
<relocation>
<pattern>org.codehaus.plexus.util</pattern>
<shadedPattern>org.shaded.plexus.util</shadedPattern>
<excludes>
<exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
<exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<name>my-app</name>
<url>http://maven.apache.org</url>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>FlinktoLambda</groupId>
<artifactId>my-app</artifactId>
<exclusions>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-remote_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.10</artifactId>
</exclusion>
<exclusion>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-slf4j_2.10</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.10</artifactId>
<version>1.3.2</version>
</dependency>
</dependencies>
插件中相应更改您的主类。
此错误的另一个原因可能是Flink与您的应用程序代码(或使用scala的应用程序依赖项)之间的scala版本不匹配。
例如,在我的情况下,我使用的是Flink 1.7.1,我必须将我的scala依赖项从2.11更新为2.12;我更新了相关依赖项的artifcatId如下:从shade
到flink-scala_2.11
,flink-scala_2.12
到flink-table_2.11
等。
有关更多信息,请参阅flink-table_2.12
。