org.apache.nutch.urlfilter.domain

Class DomainURLFilter

java.lang.Object

org.apache.nutch.urlfilter.domain.DomainURLFilter

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, URLFilter, Pluggable

public class DomainURLFilter
extends Object
implements URLFilter

Filters URLs based on a file containing domain suffixes, domain names, and hostnames. Only a url that matches one of the suffixes, domains, or hosts present in the file is allowed.

Urls are checked in order of domain suffix, domain name, and hostname against entries in the domain file. The domain file would be setup as follows with one entry per line:

 com apache.org www.apache.org 

The first line is an example of a filter that would allow all .com domains. The second line allows all urls from apache.org and all of its subdomains such as lucene.apache.org and hadoop.apache.org. The third line would allow only urls from www.apache.org. There is no specific ordering to entries. The entries are from more general to more specific with the more general overridding the more specific.

The domain file defaults to domain-urlfilter.txt in the classpath but can be overridden using the:

property "urlfilter.domain.file" in ./conf/nutch-*.xml, and
attribute "file" in plugin.xml of this plugin

the attribute "file" has higher precedence if defined.

Field Summary

Fields inherited from interface org.apache.nutch.net.URLFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
DomainURLFilter() Default constructor.
DomainURLFilter(String domainFile) Constructor that specifies the domain file to use.

Method Summary

Methods

Modifier and Type	Method and Description
String	filter(String url)
org.apache.hadoop.conf.Configuration	getConf()
void	setConf(org.apache.hadoop.conf.Configuration conf) Sets the configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DomainURLFilter

public DomainURLFilter()

Default constructor.

DomainURLFilter

public DomainURLFilter(String domainFile)

Constructor that specifies the domain file to use.

Parameters:

domainFile - The domain file, overrides domain-urlfilter.text default.

Throws:

IOException

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Sets the configuration.

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

filter

public String filter(String url)

Specified by:

filter in interface URLFilter