org.apache.nutch.analysis.lang
Class LanguageIndexingFilter
java.lang.Object
org.apache.nutch.analysis.lang.LanguageIndexingFilter
- All Implemented Interfaces:
- Configurable, IndexingFilter, Pluggable
public class LanguageIndexingFilter
- extends Object
- implements IndexingFilter
An IndexingFilter
that
add a lang
(language) field to the document.
It tries to find the language of the document by:
- First, checking if
HTMLLanguageParser
add some language
information
- Then, checking if a
Content-Language
HTTP header can be
found
- Finaly by analyzing the document content
- Author:
- Sami Siren, Jerome Charron
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
LanguageIndexingFilter
public LanguageIndexingFilter()
- Constructs a new Language Indexing Filter.
filter
public NutchDocument filter(NutchDocument doc,
Parse parse,
Text url,
CrawlDatum datum,
Inlinks inlinks)
throws IndexingException
- Description copied from interface:
IndexingFilter
- Adds fields or otherwise modifies the document that will be indexed for a
parse. Unwanted documents can be removed from indexing by returning a null value.
- Specified by:
filter
in interface IndexingFilter
- Parameters:
doc
- document instance for collecting fieldsparse
- parse data instanceurl
- page urldatum
- crawl datum for the pageinlinks
- page inlinks
- Returns:
- modified (or a new) document instance, or null (meaning the document
should be discarded)
- Throws:
IndexingException
addIndexBackendOptions
public void addIndexBackendOptions(Configuration conf)
- Description copied from interface:
IndexingFilter
- Adds index-level configuraition options.
Implementations can update given configuration to pass document-independent
information to indexing backends. As a rule of thumb, prefix meta keys
with the name of the backend intended. For example, when
passing information to lucene backend, prefix keys with "lucene.".
- Specified by:
addIndexBackendOptions
in interface IndexingFilter
- Parameters:
conf
- Configuration instance.
setConf
public void setConf(Configuration conf)
- Specified by:
setConf
in interface Configurable
getConf
public Configuration getConf()
- Specified by:
getConf
in interface Configurable
Copyright © 2006 The Apache Software Foundation