无法从Eclipse将Spark与Hortonworks沙箱连接

问题描述 投票:1回答:1
I am not able to connect my spark Code which is written on eclipse.
 Below is the code please guid me how I can do the same. 
Anything would be helpful  

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;

 public class SparkTest {

public static void main(String [] args){

  SparkConf conf = new SparkConf().setAppName("JD Word

Counter“)。setMaster(” local“);

  JavaSparkContext sc = new JavaSparkContext(conf);
          //hdfs://localhost:8020/user/root/textfile/test.txt         JavaRDD<String> inputFile =

sc.textFile(“ hdfs:// localhost:8020 / user / root / textfile / test.txt”);System.out.println(“ H​​ello start”);System.out.println(inputFile.collect()); JavaRDDwordsFromFile = inputFile.flatMap(content->Arrays.asList(content.split(“”))。iterator());System.out.println(“ hello end”);

//JavaPairRDD countData = wordsFromFile.mapToPair(t -> new Tuple2(t, 1)).reduceByKey((x, y) -> (int) x + (int) y);

  //wordsFromFile.saveAsTextFile("hdfs://localhost:8020/user/root/fileTest/");

  System.out.println(" This java program is complete");   }

}


    Error
    I/O error constructing remote block reader.
    org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for 
    channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending 
     remote=/172.18.0.2:50010]
   at org.apache.hadoop.net.NetUtils.c
eclipse apache-spark hadoop hortonworks-data-platform spark-java
1个回答
0
投票

将localhost更改为hdp沙箱的ip地址,或将hdfs-site.xml文件放入类路径中。

© www.soinside.com 2019 - 2024. All rights reserved.