org.apache.nutch.net

Class URLNormalizers

java.lang.Object

org.apache.nutch.net.URLNormalizers

public final class URLNormalizers
extends Object

This class uses a "chained filter" pattern to run defined normalizers. Different lists of normalizers may be defined for different "scopes", or contexts where they are used (note however that they need to be activated first through plugin.include property).

There is one global scope defined by default, which consists of all active normalizers. The order in which these normalizers are executed may be defined in "urlnormalizer.order" property, which lists space-separated implementation classes (if this property is missing normalizers will be run in random order). If there are more normalizers activated than explicitly named on this list, the remaining ones will be run in random order after the ones specified on the list are executed.

You can define a set of contexts (or scopes) in which normalizers may be called. Each scope can have its own list of normalizers (defined in "urlnormalizer.scope." property) and its own order (defined in "urlnormalizer.order." property). If any of these properties are missing, default settings are used for the global scope.

In case no normalizers are required for any given scope, a org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer should be used.

Each normalizer may further select among many configurations, depending on the scope in which it is called, because the scope name is passed as a parameter to each normalizer. You can also use the same normalizer for many scopes.

Several scopes have been defined, and various Nutch tools will attempt using scope-specific normalizers first (and fall back to default config if scope-specific configuration is missing).

Normalizers may be run several times, to ensure that modifications introduced by normalizers at the end of the list can be further reduced by normalizers executed at the beginning. By default this loop is executed just once - if you want to ensure that all possible combinations have been applied you may want to run this loop up to the number of activated normalizers. This loop count can be configured through urlnormalizer.loop.count property. As soon as the url is unchanged the loop will stop and return the result.

Author:

Andrzej Bialecki

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
static String	SCOPE_CRAWLDB Scope used when updating the CrawlDb with new URLs.
static String	SCOPE_DEFAULT Default scope.
static String	SCOPE_FETCHER Scope used by Fetcher when processing redirect URLs.
static String	SCOPE_GENERATE_HOST_COUNT Scope used by Generator.
static String	SCOPE_INDEXER Scope used when indexing URLs.
static String	SCOPE_INJECT Scope used by Injector.
static String	SCOPE_LINKDB Scope used when updating the LinkDb with new URLs.
static String	SCOPE_OUTLINK Scope used when constructing new Outlink instances.
static String	SCOPE_PARTITION Scope used by URLPartitioner.

Constructor Summary

Constructors

Constructor and Description
URLNormalizers(org.apache.hadoop.conf.Configuration conf, String scope)

Method Summary

Methods

Modifier and Type	Method and Description
String	normalize(String urlString, String scope) Normalize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCOPE_DEFAULT

public static final String SCOPE_DEFAULT

Default scope. If no scope properties are defined then the configuration for this scope will be used.

See Also:

Constant Field Values

SCOPE_PARTITION

public static final String SCOPE_PARTITION

Scope used by URLPartitioner.

See Also:

Constant Field Values

SCOPE_GENERATE_HOST_COUNT

public static final String SCOPE_GENERATE_HOST_COUNT

Scope used by Generator.

See Also:

Constant Field Values

SCOPE_FETCHER

public static final String SCOPE_FETCHER

Scope used by Fetcher when processing redirect URLs.

See Also:

Constant Field Values

SCOPE_CRAWLDB

public static final String SCOPE_CRAWLDB

Scope used when updating the CrawlDb with new URLs.

See Also:

Constant Field Values

SCOPE_LINKDB

public static final String SCOPE_LINKDB

Scope used when updating the LinkDb with new URLs.

See Also:

Constant Field Values

SCOPE_INJECT

public static final String SCOPE_INJECT

Scope used by Injector.

See Also:

Constant Field Values

SCOPE_OUTLINK

public static final String SCOPE_OUTLINK

Scope used when constructing new Outlink instances.

See Also:

Constant Field Values

SCOPE_INDEXER

public static final String SCOPE_INDEXER

Scope used when indexing URLs.

See Also:

Constant Field Values

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

URLNormalizers

public URLNormalizers(org.apache.hadoop.conf.Configuration conf,
              String scope)

Method Detail

normalize

public String normalize(String urlString,
               String scope)
                 throws MalformedURLException

Normalize

Parameters:

urlString - The URL string to normalize.

scope - The given scope.

Returns:

A normalized String, using the given scope

Throws:

MalformedURLException - If the given URL string is malformed.