org.apache.nutch.scoring

Interface ScoringFilter

All Superinterfaces:

Configurable, Pluggable

All Known Implementing Classes:

AbstractScoringFilter, DepthScoringFilter, LinkAnalysisScoringFilter, OPICScoringFilter, ScoringFilters, TLDScoringFilter, URLMetaScoringFilter

public interface ScoringFilter
extends Configurable, Pluggable

A contract defining behavior of scoring plugins. A scoring filter will manipulate scoring variables in CrawlDatum and in resulting search indexes. Filters can be chained in a specific order, to provide multi-stage scoring adjustments.

Author:

Andrzej Bialecki

Field Summary

Fields

Modifier and Type	Field and Description
static String	X_POINT_ID The name of the extension point.

Method Summary

Methods

Modifier and Type	Method and Description
CrawlDatum	distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount) Distribute score value from the current page to all its outlinked pages.
float	generatorSortValue(Text url, CrawlDatum datum, float initSort) This method prepares a sort value for the purpose of sorting and selecting top N scoring pages during fetchlist generation.
float	indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) This method calculates a Lucene document boost.
void	initialScore(Text url, CrawlDatum datum) Set an initial score for newly discovered pages.
void	injectedScore(Text url, CrawlDatum datum) Set an initial score for newly injected pages.
void	passScoreAfterParsing(Text url, Content content, Parse parse) Currently a part of score distribution is performed using only data coming from the parsing process.
void	passScoreBeforeParsing(Text url, CrawlDatum datum, Content content) This method takes all relevant score information from the current datum (coming from a generated fetchlist) and stores it into Content metadata.
void	updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked) This method calculates a new score of CrawlDatum during CrawlDb update, based on the initial value of the original CrawlDatum, and also score values contributed by inlinked pages.

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

X_POINT_ID

static final String X_POINT_ID

The name of the extension point.

Method Detail

injectedScore

void injectedScore(Text url,
                 CrawlDatum datum)
                   throws ScoringFilterException

Set an initial score for newly injected pages. Note: newly injected pages may have no inlinks, so filter implementations may wish to set this score to a non-zero value, to give newly injected pages some initial credit.

Parameters:

url - url of the page

datum - new datum. Filters will modify it in-place.

Throws:

ScoringFilterException

initialScore

void initialScore(Text url,
                CrawlDatum datum)
                  throws ScoringFilterException

Set an initial score for newly discovered pages. Note: newly discovered pages have at least one inlink with its score contribution, so filter implementations may choose to set initial score to zero (unknown value), and then the inlink score contribution will set the "real" value of the new page.

Parameters:

url - url of the page

datum - new datum. Filters will modify it in-place.

Throws:

ScoringFilterException

generatorSortValue

float generatorSortValue(Text url,
                       CrawlDatum datum,
                       float initSort)
                         throws ScoringFilterException

This method prepares a sort value for the purpose of sorting and selecting top N scoring pages during fetchlist generation.

Parameters:

url - url of the page

datum - page's datum, should not be modified

initSort - initial sort value, or a value from previous filters in chain

Throws:

ScoringFilterException

passScoreBeforeParsing

void passScoreBeforeParsing(Text url,
                          CrawlDatum datum,
                          Content content)
                            throws ScoringFilterException

This method takes all relevant score information from the current datum (coming from a generated fetchlist) and stores it into Content metadata. This is needed in order to pass this value(s) to the mechanism that distributes it to outlinked pages.

Parameters:

url - url of the page

datum - source datum. NOTE: modifications to this value are not persisted.

content - instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.

Throws:

ScoringFilterException

passScoreAfterParsing

void passScoreAfterParsing(Text url,
                         Content content,
                         Parse parse)
                           throws ScoringFilterException

Currently a part of score distribution is performed using only data coming from the parsing process. We need this method in order to ensure the presence of score data in these steps.

Parameters:

url - page url

content - original content. NOTE: modifications to this value are not persisted.

parse - target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.

Throws:

ScoringFilterException

distributeScoreToOutlinks

CrawlDatum distributeScoreToOutlinks(Text fromUrl,
                                   ParseData parseData,
                                   Collection<Map.Entry<Text,CrawlDatum>> targets,
                                   CrawlDatum adjust,
                                   int allCount)
                                     throws ScoringFilterException

Distribute score value from the current page to all its outlinked pages.

Parameters:

fromUrl - url of the source page

parseData - ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.

targets - <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.

adjust - a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status to CrawlDatum.STATUS_LINKED.

allCount - number of all collected outlinks from the source page

Returns:

if needed, implementations may return an instance of CrawlDatum, with status CrawlDatum.STATUS_LINKED, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed.

Throws:

ScoringFilterException

updateDbScore

void updateDbScore(Text url,
                 CrawlDatum old,
                 CrawlDatum datum,
                 List<CrawlDatum> inlinked)
                   throws ScoringFilterException

This method calculates a new score of CrawlDatum during CrawlDb update, based on the initial value of the original CrawlDatum, and also score values contributed by inlinked pages.

Parameters:

url - url of the page

old - original datum, with original score. May be null if this is a newly discovered page. If not null, filters should use score values from this parameter as the starting values - the datum parameter may contain values that are no longer valid, if other updates occured between generation and this update.

datum - the new datum, with the original score saved at the time when fetchlist was generated. Filters should update this in-place, and it will be saved in the crawldb.

inlinked - (partial) list of CrawlDatum-s (with their scores) from links pointing to this page, found in the current update batch.

Throws:

ScoringFilterException

indexerScore

float indexerScore(Text url,
                 NutchDocument doc,
                 CrawlDatum dbDatum,
                 CrawlDatum fetchDatum,
                 Parse parse,
                 Inlinks inlinks,
                 float initScore)
                   throws ScoringFilterException

This method calculates a Lucene document boost.

Parameters:

url - url of the page

doc - Lucene document. NOTE: this already contains all information collected by indexing filters. Implementations may modify this instance, in order to store/remove some information.

dbDatum - current page from CrawlDb. NOTE: changes made to this instance are not persisted.

fetchDatum - datum from FetcherOutput (containing among others the fetching status)

parse - parsing result. NOTE: changes made to this instance are not persisted.

inlinks - current inlinks from LinkDb. NOTE: changes made to this instance are not persisted.

initScore - initial boost value for the Lucene document.

Returns:

boost value for the Lucene document. This value is passed as an argument to the next scoring filter in chain. NOTE: implementations may also express other scoring strategies by modifying Lucene document directly.

Throws:

ScoringFilterException