org.apache.nutch.collection

Class Subcollection

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.collection.Subcollection

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, URLFilter, Pluggable

public class Subcollection
extends org.apache.hadoop.conf.Configured
implements URLFilter

SubCollection represents a subset of index, you can define url patterns that will indicate that particular page (url) is part of SubCollection.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TAG_BLACKLIST
static String	TAG_COLLECTION
static String	TAG_COLLECTIONS
static String	TAG_ID
static String	TAG_KEY
static String	TAG_NAME
static String	TAG_WHITELIST

Fields inherited from interface org.apache.nutch.net.URLFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
Subcollection(org.apache.hadoop.conf.Configuration conf)
Subcollection(String id, String name, org.apache.hadoop.conf.Configuration conf) public Constructor
Subcollection(String id, String name, String key, org.apache.hadoop.conf.Configuration conf) public Constructor

Method Summary

Methods

Modifier and Type	Method and Description
String	filter(String urlString) Simple "indexOf" currentFilter for matching patterns.
String	getBlackListString() Returns blacklist String
String	getId()
String	getKey()
String	getName()
List<String>	getWhiteList() Returns whitelist
String	getWhiteListString() Returns whitelist String
void	initialize(Element collection) Initialize Subcollection from dom element
protected void	parseList(List<String> list, String text) Create a list of patterns from chunk of text, patterns are separated with newline
void	setBlackList(String list) Set contents of blacklist from String
void	setWhiteList(ArrayList<String> whiteList)
void	setWhiteList(String list) Set contents of whitelist from String

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

TAG_COLLECTIONS

public static final String TAG_COLLECTIONS

See Also:

Constant Field Values

TAG_COLLECTION

public static final String TAG_COLLECTION

See Also:

Constant Field Values

TAG_WHITELIST

public static final String TAG_WHITELIST

See Also:

Constant Field Values

TAG_BLACKLIST

public static final String TAG_BLACKLIST

See Also:

Constant Field Values

TAG_NAME

public static final String TAG_NAME

See Also:

Constant Field Values

TAG_KEY

public static final String TAG_KEY

See Also:

Constant Field Values

TAG_ID

public static final String TAG_ID

See Also:

Constant Field Values

Constructor Detail

Subcollection

public Subcollection(String id,
             String name,
             org.apache.hadoop.conf.Configuration conf)

public Constructor

Parameters:

id - id of SubCollection

name - name of SubCollection

Subcollection

public Subcollection(String id,
             String name,
             String key,
             org.apache.hadoop.conf.Configuration conf)

public Constructor

Parameters:

id - id of SubCollection

name - name of SubCollection

Subcollection

public Subcollection(org.apache.hadoop.conf.Configuration conf)

Method Detail

getName

public String getName()

Returns:

Returns the name

getKey

public String getKey()

Returns:

Returns the key

getId

public String getId()

Returns:

Returns the id

getWhiteList

public List<String> getWhiteList()

Returns whitelist

Returns:

Whitelist entries

getWhiteListString

public String getWhiteListString()

Returns whitelist String

Returns:

Whitelist String

getBlackListString

public String getBlackListString()

Returns blacklist String

Returns:

Blacklist String

setWhiteList

public void setWhiteList(ArrayList<String> whiteList)

Parameters:

whiteList - The whiteList to set.

filter

public String filter(String urlString)

Simple "indexOf" currentFilter for matching patterns.

  rules for evaluation are as follows:
  1. if pattern matches in blacklist then url is rejected
  2. if pattern matches in whitelist then url is allowed
  3. url is rejected
 

Specified by:

filter in interface URLFilter

See Also:

URLFilter.filter(java.lang.String)

initialize

public void initialize(Element collection)

Initialize Subcollection from dom element

Parameters:

collection -

parseList

protected void parseList(List<String> list,
             String text)

Create a list of patterns from chunk of text, patterns are separated with newline

Parameters:

list -

text -

setBlackList

public void setBlackList(String list)

Set contents of blacklist from String

Parameters:

list - the blacklist contents

setWhiteList

public void setWhiteList(String list)

Set contents of whitelist from String

Parameters:

list - the whitelist contents