org.apache.nutch.urlfilter.domainblacklist

Class DomainBlacklistURLFilter

java.lang.Object

org.apache.nutch.urlfilter.domainblacklist.DomainBlacklistURLFilter

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, URLFilter, Pluggable

public class DomainBlacklistURLFilter
extends Object
implements URLFilter

Filters URLs based on a file containing domain suffixes, domain names, and hostnames. A url that matches one of the suffixes, domains, or hosts present in the file is filtered out.

Urls are checked in order of domain suffix, domain name, and hostname against entries in the domain file. The domain file would be setup as follows with one entry per line:

 com apache.org www.apache.org 

The first line is an example of a filter that would allow all .com domains. The second line allows all urls from apache.org and all of its subdomains such as lucene.apache.org and hadoop.apache.org. The third line would allow only urls from www.apache.org. There is no specific ordering to entries. The entries are from more general to more specific with the more general overridding the more specific.

The domain file defaults to domainblacklist-urlfilter.txt in the classpath but can be overridden using the:

property "urlfilter.domainblacklist.file" in ./conf/nutch-*.xml, and
attribute "file" in plugin.xml of this plugin

the attribute "file" has higher precedence if defined.

Field Summary

Fields inherited from interface org.apache.nutch.net.URLFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
DomainBlacklistURLFilter() Default constructor.
DomainBlacklistURLFilter(String domainFile) Constructor that specifies the domain file to use.

Method Summary

Methods

Modifier and Type	Method and Description
String	filter(String url)
org.apache.hadoop.conf.Configuration	getConf()
void	setConf(org.apache.hadoop.conf.Configuration conf) Sets the configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DomainBlacklistURLFilter

public DomainBlacklistURLFilter()

Default constructor.

DomainBlacklistURLFilter

public DomainBlacklistURLFilter(String domainFile)

Constructor that specifies the domain file to use.

Parameters:

domainFile - The domain file, overrides domainblacklist-urlfilter.text default.

Throws:

IOException

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Sets the configuration.

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

filter

public String filter(String url)

Specified by:

filter in interface URLFilter