org.apache.nutch.scoring.webgraph

Class WebGraph.OutlinkDb

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.scoring.webgraph.WebGraph.OutlinkDb

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,NutchWritable,org.apache.hadoop.io.Text,LinkDatum>

Enclosing class:

WebGraph

public static class WebGraph.OutlinkDb
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,NutchWritable,org.apache.hadoop.io.Text,LinkDatum>

The OutlinkDb creates a database of all outlinks. Outlinks to internal urls by domain and host can be ignored. The number of Outlinks out to a given page or domain can also be limited.

Field Summary

Fields

Modifier and Type	Field and Description
static String	URL_FILTERING
static String	URL_NORMALIZING

Constructor Summary

Constructors

Constructor and Description
WebGraph.OutlinkDb() Default constructor.
WebGraph.OutlinkDb(org.apache.hadoop.conf.Configuration conf) Configurable constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	configure(org.apache.hadoop.mapred.JobConf conf) Configures the OutlinkDb job.
void	map(org.apache.hadoop.io.Text key, org.apache.hadoop.io.Writable value, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output, org.apache.hadoop.mapred.Reporter reporter) Passes through existing LinkDatum objects from an existing OutlinkDb and maps out new LinkDatum objects from new crawls ParseData.
void	reduce(org.apache.hadoop.io.Text key, Iterator<NutchWritable> values, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,LinkDatum> output, org.apache.hadoop.mapred.Reporter reporter)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

URL_NORMALIZING

public static final String URL_NORMALIZING

See Also:

Constant Field Values

URL_FILTERING

public static final String URL_FILTERING

See Also:

Constant Field Values

Constructor Detail

WebGraph.OutlinkDb

public WebGraph.OutlinkDb()

Default constructor.

WebGraph.OutlinkDb

public WebGraph.OutlinkDb(org.apache.hadoop.conf.Configuration conf)

Configurable constructor.

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Configures the OutlinkDb job. Sets up internal links and link limiting.

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

map

public void map(org.apache.hadoop.io.Text key,
       org.apache.hadoop.io.Writable value,
       org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
       org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Passes through existing LinkDatum objects from an existing OutlinkDb and maps out new LinkDatum objects from new crawls ParseData.

Specified by:

map in interface org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>

Throws:

IOException

reduce

public void reduce(org.apache.hadoop.io.Text key,
          Iterator<NutchWritable> values,
          org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,LinkDatum> output,
          org.apache.hadoop.mapred.Reporter reporter)
            throws IOException

Specified by:

reduce in interface org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.Text,NutchWritable,org.apache.hadoop.io.Text,LinkDatum>

Throws:

IOException

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable