public class ArcSegmentCreator extends Configured implements Tool, Mapper<Text,BytesWritable,Text,NutchWritable>
The ArcSegmentCreator
is a replacement for fetcher that will
take arc files as input and produce a nutch segment as output.
Arc files are tars of compressed gzips which are produced by both the internet archive project and the grub distributed crawler project.
Modifier and Type | Field and Description |
---|---|
static org.slf4j.Logger |
LOG |
static String |
URL_VERSION |
Constructor and Description |
---|
ArcSegmentCreator() |
ArcSegmentCreator(Configuration conf)
Constructor that sets the job configuration.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
configure(JobConf job)
Configures the job.
|
void |
createSegments(Path arcFiles,
Path segmentsOutDir)
Creates the arc files to segments job.
|
static String |
generateSegmentName()
Generates a random name for the segments.
|
static void |
main(String[] args) |
void |
map(Text key,
BytesWritable bytes,
OutputCollector<Text,NutchWritable> output,
Reporter reporter)
Runs the Map job to translate an arc record into output for Nutch segments.
|
int |
run(String[] args) |
getConf, setConf
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getConf, setConf
public static final org.slf4j.Logger LOG
public static final String URL_VERSION
public ArcSegmentCreator()
public ArcSegmentCreator(Configuration conf)
Constructor that sets the job configuration.
conf
- public static String generateSegmentName()
public void configure(JobConf job)
Configures the job. Sets the url filters, scoring filters, url normalizers and other relevant data.
configure
in interface JobConfigurable
job
- The job configuration.public void close()
close
in interface Closeable
close
in interface AutoCloseable
public void map(Text key, BytesWritable bytes, OutputCollector<Text,NutchWritable> output, Reporter reporter) throws IOException
Runs the Map job to translate an arc record into output for Nutch segments.
map
in interface Mapper<Text,BytesWritable,Text,NutchWritable>
key
- The arc record header.bytes
- The arc record raw content bytes.output
- The output collecter.reporter
- The progress reporter.IOException
public void createSegments(Path arcFiles, Path segmentsOutDir) throws IOException
Creates the arc files to segments job.
arcFiles
- The path to the directory holding the arc filessegmentsOutDir
- The output directory for writing the segmentsIOException
- If an IO error occurs while running the job.Copyright © 2015 The Apache Software Foundation