Hadoop MapReduce输出最大化

Question

我目前正在使用Eclipse和Hadoop创建一个mapper和reducer来查找航空公司数据集的最大总成本。 所以总成本是十进制值，航空公司承运人是文本。

我使用的数据集可以在以下网站链接中找到：https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/236265/dft-flights-data-2011.csv

当我在Hadoop中导出jar文件时，我收到以下消息：ls：“output”：没有这样的文件或目录。 任何人都可以帮我纠正代码吗？ 我的代码如下。

制图员：

package org.myorg;

import java.io.IOException;

import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MaxTotalCostMapper extends Mapper<LongWritable, Text, Text, DoubleWritable>
{
    private final static DoubleWritable totalcostWritable = new DoubleWritable(0);
    private Text AirCarrier = new Text();

    @Override
    public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException
    {
        String[] line = value.toString().split(",");
        AirCarrier.set(line[8]);
        double totalcost = Double.parseDouble(line[2].trim());
        totalcostWritable.set(totalcost);
        context.write(AirCarrier, totalcostWritable);
    }
}

减速器：

package org.myorg;

import java.io.IOException;
import java.util.ArrayList;

import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxTotalCostReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable>
{
    ArrayList<Double> totalcostList = new ArrayList<Double>();

    @Override
    public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
            throws IOException, InterruptedException
    {
        double maxValue=0.0;
        for (DoubleWritable value : values) 
        {
            maxValue = Math.max(maxValue, value.get());
        }
        context.write(key, new DoubleWritable(maxValue));
    }
}

主要：

package org.myorg;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;


public class MaxTotalCost
{
    public static void main(String[] args) throws Exception
    {
        Configuration conf = new Configuration();
        if (args.length != 2)
        {
            System.err.println("Usage: MaxTotalCost<input path><output path>");
            System.exit(-1);
        }

        Job job;
        job=Job.getInstance(conf, "Max Total Cost");
        job.setJarByClass(MaxTotalCost.class);

        FileInputFormat.addInputPath(job, new Path(args[1]));
        FileOutputFormat.setOutputPath(job, new Path(args[2]));

        job.setMapperClass(MaxTotalCostMapper.class);
        job.setReducerClass(MaxTotalCostReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(DoubleWritable.class);

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Answer 1

ls：“output”：没有这样的文件或目录

您没有HDFS用户目录。您的代码没有进入Mapper或Reducer。这个错误通常出现在Job

  FileOutputFormat.setOutputPath(job, new Path(args[2]));

运行hdfs dfs -ls，看看你是否有任何错误。如果是这样，请在/user下创建一个与当前用户匹配的目录。

否则，将输出目录更改为/tmp/max