问：在spark-submit之后，找不到logger(org.apache.spark.deploy.SparkSubmit$$anon$2)的附属机构。

Question

用Apache Spark一步步快速启动，但最后显示出这个警告信息。

20/05/25 09:43:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$$anon$2).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

我的代码是

package firstmaven;

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
/**
 * Hello world!
 *
 */
public class App 
{
    public static void main( String[] args )
    {
        String logFile = "/data/spark/README.md";
        SparkSession spark = SparkSession.builder().appName("Simple Application")
                            .config("spark.master","local").getOrCreate();
        Dataset<String> logData = spark.read().textFile(logFile).cache();
        long numAs = logData.filter(s -> s.contains("a")).count();
        long numBs = logData.filter(s -> s.contains("b")).count();
        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

        System.out.println("Hello world");

        spark.stop();
    }
}

我应该怎么做才能让它工作 ? 谢谢。

Answer 1

你需要指定你想过滤的列的值。查看下面这个片段。

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.SparkSession;

import java.util.List;

public class App {
    public static void main(String[] args) {
        String logFile = "/Users/ajay/input.txt";
        SparkSession spark = SparkSession.builder().appName("Simple Application")
                .config("spark.master", "local").getOrCreate();
        Dataset<String> logData = spark.read().textFile(logFile).cache();
        List<String> rowList = logData.collectAsList();
        System.out.println("rowList is = " + rowList);
        Dataset<String> rowDatasetWithA = logData.filter((logData.col("value").contains("a")));
        List<String> rowWithA = rowDatasetWithA.collectAsList();
        System.out.println("rowWithA is = " + rowWithA);

        Dataset<String> rowDatasetWithB = logData.filter((logData.col("value").contains("b")));
        List<String> rowWithB = rowDatasetWithB.collectAsList();
        System.out.println("rowWithB is = " + rowWithB);

        long numAs = rowWithA.size();
        long numBs = rowWithB.size();
        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

        spark.stop();
    }
}

假设列的内容是 input.txt 是这样

a
b
c
a
aa 

上述代码段的输出将如下所示

rowList is = [a, b, c, a, aa]
rowWithA is = [a, a, aa]
rowWithB is = [b]
Lines with a: 3, lines with b: 1
enter code here

希望对你有所帮助。