StorageUtils$ 无法访问类 sun.nio.ch.DirectBuffer

Question

在 Spring Maven 中运行集成测试。 创建 SparkSession 时出现以下错误

• 我有

   <additionalClasspathElement>${project.build.outputDirectory}

• 并且确实有

   <argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>

ERROR!
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x74235045) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x74235045
        at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala:213)
        at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
        at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
        at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
        at scala.Option.getOrElse(Option.scala:201)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)

Maven 配置文件看起来像这样

  <profile>
            <id>integration-test</id>
            <build>
                <resources>
                    <resource>
                        <directory>src/main/resources</directory>
                    </resource>
                    <resource>
                        <directory>src/test/resources</directory>
                    </resource>
                </resources>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-failsafe-plugin</artifactId>
                        <version>3.0.0</version>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>integration-test</goal>
                                    <goal>verify</goal>
                                </goals>
                            </execution>
                        </executions>
                        <configuration>
                            <!--  Setting additionalClasspathElement to solve NoClassDefFoundError error.
                           See here for more info: https://stackoverflow.com/a/50804520/9914653 -->
                            <additionalClasspathElements>
                                <additionalClasspathElement>${project.build.outputDirectory}
                                </additionalClasspathElement>
                            </additionalClasspathElements>
                            <includes>
                                <include>**/*IntegrationTest.java</include>
                                <include>**/*IntegrationTest.kt</include>
                            </includes>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

SureFire 插件看起来像这样，

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.0</version>
                <configuration>
                    <!-- custom value for `skipTests` allows to skip surefire tests (unit tests)
                     when running integration tests (failsafe tests) https://stackoverflow.com/a/17932772/9914653
                     In order to run integration tests: `mvn verify -Pintegration-test -Dskip.surefire.tests`
                     In order to skip unit tests use -DskipTests as usual: `mvn clean install -DskipTests`.
                      -->
                    <skipTests>${skip.surefire.tests}</skipTests>
                    <excludes>
                        <exclude>**/*IntegrationTest.java</exclude>
                        <exclude>**/*IntegrationTest.kt</exclude>
                    </excludes>
                    <!-- The arg is needed because of Spark and JDK 17 issue https://stackoverflow.com/a/72724817/9914653 -->
                    <argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>
                </configuration>
            </plugin>

Answer 1

所以问题是配置文件部分中缺少一行，与插件

maven-surefire-plugin

具有相同的行。

<argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>