org.apache.nutch.fetcher

Class OldFetcher

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.fetcher.OldFetcher

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.util.Tool

public class OldFetcher
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>

The fetcher. Most of the work is done by plugins.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OldFetcher.InputFormat

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTENT_REDIR
static org.slf4j.Logger	LOG
static int	PERM_REFRESH_TIME
static String	PROTOCOL_REDIR

Constructor Summary

Constructors

Constructor and Description
OldFetcher()
OldFetcher(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	configure(org.apache.hadoop.mapred.JobConf job)
void	fetch(org.apache.hadoop.fs.Path segment, int threads)
static boolean	isParsing(org.apache.hadoop.conf.Configuration conf)
static boolean	isStoringContent(org.apache.hadoop.conf.Configuration conf)
static void	main(String[] args) Run the fetcher.
void	run(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable> input, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output, org.apache.hadoop.mapred.Reporter reporter)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

public static final org.slf4j.Logger LOG

PERM_REFRESH_TIME

public static final int PERM_REFRESH_TIME

See Also:

Constant Field Values

CONTENT_REDIR

public static final String CONTENT_REDIR

See Also:

Constant Field Values

PROTOCOL_REDIR

public static final String PROTOCOL_REDIR

See Also:

Constant Field Values

Constructor Detail

OldFetcher

public OldFetcher()

OldFetcher

public OldFetcher(org.apache.hadoop.conf.Configuration conf)

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf job)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

close

public void close()

isParsing

public static boolean isParsing(org.apache.hadoop.conf.Configuration conf)

isStoringContent

public static boolean isStoringContent(org.apache.hadoop.conf.Configuration conf)

run

public void run(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable> input,
       org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
       org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Specified by:

run in interface org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>

Throws:

IOException

fetch

public void fetch(org.apache.hadoop.fs.Path segment,
         int threads)
           throws IOException

Throws:

IOException

main

public static void main(String[] args)
                 throws Exception

Run the fetcher.

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception