setFetchSchedule
public CrawlDatum setFetchSchedule(Text url,
CrawlDatum datum,
long prevFetchTime,
long prevModifiedTime,
long fetchTime,
long modifiedTime,
int state)
Sets the fetchInterval
and fetchTime
on a
successfully fetched page. NOTE: this implementation resets the retry
counter - extending classes should call super.setFetchSchedule() to
preserve this behavior.
- Specified by:
setFetchSchedule
in interface FetchSchedule
- Overrides:
setFetchSchedule
in class AdaptiveFetchSchedule
- Parameters:
url
- url of the pagedatum
- page description to be adjusted. NOTE: this instance, passed by
reference, may be modified inside the method.prevFetchTime
- previous value of fetch time, or 0 if not available.prevModifiedTime
- previous value of modifiedTime, or 0 if not available.fetchTime
- the latest time, when the page was recently re-fetched. Most
FetchSchedule implementations should update the value in @see
CrawlDatum to something greater than this value.modifiedTime
- last time the content was modified. This information comes from
the protocol implementations, or is set to < 0 if not available.
Most FetchSchedule implementations should update the value in @see
CrawlDatum to this value.state
- if FetchSchedule.STATUS_MODIFIED
, then the content is considered to be
"changed" before the fetchTime
, if
FetchSchedule.STATUS_NOTMODIFIED
then the content is known to be
unchanged. This information may be obtained by comparing page
signatures before and after fetching. If this is set to
FetchSchedule.STATUS_UNKNOWN
, then it is unknown whether the page was
changed; implementations are free to follow a sensible default
behavior.
- Returns:
- adjusted page information, including all original information.
NOTE: this may be a different instance than @see CrawlDatum, but
implementations should make sure that it contains at least all
information from @see CrawlDatum}.