org.apache.nutch.crawl
Class MimeAdaptiveFetchSchedule
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.crawl.AbstractFetchSchedule
org.apache.nutch.crawl.AdaptiveFetchSchedule
org.apache.nutch.crawl.MimeAdaptiveFetchSchedule
- All Implemented Interfaces:
- Configurable, FetchSchedule
public class MimeAdaptiveFetchSchedule
- extends AdaptiveFetchSchedule
Extension of @see AdaptiveFetchSchedule that allows for more flexible configuration
of DEC and INC factors for various MIME-types.
This class can be typically used in cases where a recrawl consists of many different
MIME-types. It's not very common for MIME-types other than text/html to change frequently.
Using this class you can configure different factors per MIME-type so to prefer frequently
changing MIME-types over others.
For it to work this class relies on the Content-Type MetaData key being present in the CrawlDB.
This can either be done when injecting new URL's or by adding "Content-Type" to the
db.parsemeta.to.crawldb configuration setting to force MIME-types of newly discovered URL's to
be added to the CrawlDB.
- Author:
- markus
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
LOG
public static final org.slf4j.Logger LOG
SCHEDULE_INC_RATE
public static final String SCHEDULE_INC_RATE
- See Also:
- Constant Field Values
SCHEDULE_DEC_RATE
public static final String SCHEDULE_DEC_RATE
- See Also:
- Constant Field Values
SCHEDULE_MIME_FILE
public static final String SCHEDULE_MIME_FILE
- See Also:
- Constant Field Values
MimeAdaptiveFetchSchedule
public MimeAdaptiveFetchSchedule()
setConf
public void setConf(Configuration conf)
- Specified by:
setConf
in interface Configurable
- Overrides:
setConf
in class AdaptiveFetchSchedule
setFetchSchedule
public CrawlDatum setFetchSchedule(Text url,
CrawlDatum datum,
long prevFetchTime,
long prevModifiedTime,
long fetchTime,
long modifiedTime,
int state)
- Description copied from class:
AbstractFetchSchedule
- Sets the
fetchInterval
and fetchTime
on a
successfully fetched page. NOTE: this implementation resets the
retry counter - extending classes should call super.setFetchSchedule() to
preserve this behavior.
- Specified by:
setFetchSchedule
in interface FetchSchedule
- Overrides:
setFetchSchedule
in class AdaptiveFetchSchedule
- Parameters:
url
- url of the pagedatum
- page description to be adjusted. NOTE: this instance, passed by reference,
may be modified inside the method.prevFetchTime
- previous value of fetch time, or 0 if not available.prevModifiedTime
- previous value of modifiedTime, or 0 if not available.fetchTime
- the latest time, when the page was recently re-fetched. Most FetchSchedule
implementations should update the value in @see CrawlDatum to something greater than this value.modifiedTime
- last time the content was modified. This information comes from
the protocol implementations, or is set to < 0 if not available. Most FetchSchedule
implementations should update the value in @see CrawlDatum to this value.state
- if FetchSchedule.STATUS_MODIFIED
, then the content is considered to be "changed" before the
fetchTime
, if FetchSchedule.STATUS_NOTMODIFIED
then the content is known to be unchanged.
This information may be obtained by comparing page signatures before and after fetching. If this
is set to FetchSchedule.STATUS_UNKNOWN
, then it is unknown whether the page was changed; implementations
are free to follow a sensible default behavior.
- Returns:
- adjusted page information, including all original information. NOTE: this may
be a different instance than @see CrawlDatum, but implementations should make sure that
it contains at least all information from @see CrawlDatum}.
main
public static void main(String[] args)
throws Exception
- Throws:
Exception
Copyright © 2012 The Apache Software Foundation