Java

Hadoop中的Word Count例子到底是怎么回事？

1. 不管是国外还是国内，一提到Hadoop的入门程序，第一个一定会是WordCount，就是给你一堆文件，让你计算出文件中出现的单词的个数；

2. 他们一般给出的程序如下：

Map： 

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordCountMapper extends MapReduceBase
    implements Mapper<LongWritable, Text, Text, IntWritable> {

  private final IntWritable one = new IntWritable(1);
  private Text word = new Text();

  public void map(WritableComparable key, Writable value,
      OutputCollector output, Reporter reporter) throws IOException {

    String line = value.toString();
    StringTokenizer itr = new StringTokenizer(line.toLowerCase());
    while(itr.hasMoreTokens()) {
      word.set(itr.nextToken());
      output.collect(word, one);
    }
  }
}

Reduce：

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class WordCountReducer extends MapReduceBase
    implements Reducer<Text, IntWritable, Text, IntWritable> {

  public void reduce(Text key, Iterator values,
      OutputCollector output, Reporter reporter) throws IOException {

    int sum = 0;
    while (values.hasNext()) {
      IntWritable value = (IntWritable) values.next();
      sum += value.get(); // process value
    }

    output.collect(key, new IntWritable(sum));
  }
}

3. 但是没有解释为什么在Map中使用的是值为1的IntWritable，我google了一段时间，找到下面两个图，应该可以很好的解释了这个问题。

The MapReduce workflow

The MapReduce workflow

The MapReduce workflow

The word count flow:

refs:

http://blog.gopivotal.com/pivotal/products/hadoop-101-programming-mapreduce-
with-native-libraries-hive-pig-and-cascading

http://kickstarthadoop.blogspot.de/2011/04/word-count-hadoop-map-reduce-
example.html

https://developer.yahoo.com/hadoop/tutorial/