public class WebGraph
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool
Modifier and Type | Class and Description |
---|---|
static class |
WebGraph.OutlinkDb
The OutlinkDb creates a database of all outlinks.
|
Modifier and Type | Field and Description |
---|---|
static String |
INLINK_DIR |
static String |
LOCK_NAME |
static org.slf4j.Logger |
LOG |
static String |
NODE_DIR |
static String |
OLD_OUTLINK_DIR |
static String |
OUTLINK_DIR |
Constructor and Description |
---|
WebGraph() |
Modifier and Type | Method and Description |
---|---|
void |
createWebGraph(org.apache.hadoop.fs.Path webGraphDb,
org.apache.hadoop.fs.Path[] segments,
boolean normalize,
boolean filter)
Creates the three different WebGraph databases, Outlinks, Inlinks, and
Node.
|
static void |
main(String[] args) |
int |
run(String[] args)
Parses command link arguments and runs the WebGraph jobs.
|
public static final org.slf4j.Logger LOG
public static final String LOCK_NAME
public static final String INLINK_DIR
public static final String OUTLINK_DIR
public static final String OLD_OUTLINK_DIR
public static final String NODE_DIR
public void createWebGraph(org.apache.hadoop.fs.Path webGraphDb, org.apache.hadoop.fs.Path[] segments, boolean normalize, boolean filter) throws IOException
webGraphDb
- The WebGraph to create or update.segments
- The array of segments used to update the WebGraph. Newer
segments and fetch times will overwrite older segments.normalize
- whether to use URLNormalizers on URL's in the segmentfilter
- whether to use URLFilters on URL's in the segmentIOException
- If an error occurs while processing the WebGraph.Copyright © 2014 The Apache Software Foundation