A B C D E F G H I L M N O P Q R S T U V W

A

AbstractDroid<T extends Task> - Class in org.apache.droids

Manage common tasks in standard Droids

AbstractDroid(TaskQueue<T>, TaskMaster<T>) - Constructor for class org.apache.droids.AbstractDroid

accept(String) - Method in class org.apache.droids.helper.factories.URLFiltersFactory

Run all defined filters.

accept(String, String) - Method in class org.apache.droids.helper.factories.URLFiltersFactory

Run a specific filter class.

accept() - Method in class org.apache.droids.net.RegexRule

Return if this rule is used for filtering-in or out.

addOutlinkURI(String) - Method in class org.apache.droids.parse.html.LinkExtractor

Add the outlink to the links list if the value is a valid URI.

afterExecute(T, Worker<T>, Exception) - Method in interface org.apache.droids.api.WorkMonitor

afterExecute(T, Worker<T>, Exception) - Method in class org.apache.droids.monitor.SimpleWorkMonitor

awaitTermination(long, TimeUnit) - Method in interface org.apache.droids.api.TaskMaster

awaitTermination(long, TimeUnit) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

awaitTermination(long, TimeUnit) - Method in class org.apache.droids.impl.SequentialTaskMaster

B

beforeExecute(T, Worker<T>) - Method in interface org.apache.droids.api.WorkMonitor

beforeExecute(T, Worker<T>) - Method in class org.apache.droids.monitor.SimpleWorkMonitor

bufferSize - Variable in class org.apache.droids.handle.Save

C

clear() - Method in class org.apache.droids.impl.SimpleTaskQueue

clear() - Method in class org.apache.droids.impl.SimpleTaskQueueWithHistory

completedCount - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

ContentEntity - Interface in org.apache.droids.api

Abstract interface representing a body of content with a particular MIME type and an optional charset.

ContentFormatViolationException - Exception in org.apache.droids.exception

Signals content format violation.

ContentFormatViolationException(String, Throwable) - Constructor for exception org.apache.droids.exception.ContentFormatViolationException

CrawlingDroid - Class in org.apache.droids.robot.crawler

CrawlingDroid(TaskQueue<Link>, TaskMaster<Link>) - Constructor for class org.apache.droids.robot.crawler.CrawlingDroid

CrawlingWorker - Class in org.apache.droids.robot.crawler

CrawlingWorker(CrawlingDroid) - Constructor for class org.apache.droids.robot.crawler.CrawlingWorker

createAuthSchemeRegistry() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createClientConnectionManager() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createConnectionKeepAliveStrategy() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createConnectionReuseStrategy() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createCookieSpecRegistry() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createCookieStore() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createCredentialsProvider() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createFile(File) - Static method in class org.apache.droids.handle.Save

createHttpContext() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createHttpParams() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createHttpProcessor() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createHttpRequestRetryHandler() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createHttpRoutePlanner() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createProxyAuthenticationHandler() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createRedirectHandler() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createRequestExecutor() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createTargetAuthenticationHandler() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

createUserTokenHandler() - Method in class org.apache.droids.protocol.http.DroidsHttpClient

D

data - Variable in class org.apache.droids.parse.ParseImpl

DefaultTaskExceptionHandler - Class in org.apache.droids.impl

DefaultTaskExceptionHandler() - Constructor for class org.apache.droids.impl.DefaultTaskExceptionHandler

delaySpread - Variable in class org.apache.droids.delay.RandomDelayTimer

DelayTimer - Interface in org.apache.droids.api

Define the timer delay interface.

delayTimer - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

Droid<T extends Task> - Interface in org.apache.droids.api

Interface for a droid.

droid - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

DroidFactory<T extends Task> - Class in org.apache.droids.helper.factories

Factory that will lookup a droid by its name and returns it.

DroidFactory() - Constructor for class org.apache.droids.helper.factories.DroidFactory

DroidsException - Exception in org.apache.droids.exception

Wrapper object to limit the number of different Exception we can throw.

DroidsException(String) - Constructor for exception org.apache.droids.exception.DroidsException

Constructs a new exception with the specified detail message.

DroidsException(String, Throwable) - Constructor for exception org.apache.droids.exception.DroidsException

For more information Exception

DroidsException(Throwable) - Constructor for exception org.apache.droids.exception.DroidsException

For more information Exception

DroidsHttpClient - Class in org.apache.droids.protocol.http

HttpClient implementation optimized specifically for web crawling.

DroidsHttpClient() - Constructor for class org.apache.droids.protocol.http.DroidsHttpClient

DroidsHttpClient(HttpParams) - Constructor for class org.apache.droids.protocol.http.DroidsHttpClient

E

endDocument() - Method in class org.apache.droids.parse.html.LinkExtractor

execute(T) - Method in interface org.apache.droids.api.Worker

execute(Link) - Method in class org.apache.droids.robot.crawler.CrawlingWorker

execute(FileTask) - Method in class org.apache.droids.robot.walker.FileWorker

exHandler - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

exists(URI) - Method in class org.apache.droids.protocol.http.HttpClientContentLoader

F

FileProtocol - Class in org.apache.droids.protocol.file

FileProtocol() - Constructor for class org.apache.droids.protocol.file.FileProtocol

FileTask - Class in org.apache.droids.robot.walker

FileTask(File, int) - Constructor for class org.apache.droids.robot.walker.FileTask

FileWorker - Class in org.apache.droids.robot.walker

FileWorker(TaskQueue<FileTask>) - Constructor for class org.apache.droids.robot.walker.FileWorker

filter(String) - Method in interface org.apache.droids.api.URLFilter

Transforms the URL: can pass the original URL through or "delete" the URL by returning null

filter(String) - Method in class org.apache.droids.net.RegexURLFilter

finish() - Method in interface org.apache.droids.api.ManagedContentEntity

Releases all underlying resources held by the entity.

finish() - Method in class org.apache.droids.protocol.http.HttpContentEntity

finished() - Method in interface org.apache.droids.api.Droid

Invoke when the droid has completed

finished() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

finished() - Method in class org.apache.droids.robot.walker.SimpleWalkingDroid

finishedWorking - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

G

GaussianRandomDelayTimer - Class in org.apache.droids.delay

GaussianRandomDelayTimer() - Constructor for class org.apache.droids.delay.GaussianRandomDelayTimer

GaussianRandomDelayTimer(int, int) - Constructor for class org.apache.droids.delay.GaussianRandomDelayTimer

GenericFactory<T> - Class in org.apache.droids.helper.factories

Basically all factories till now extend this generic factory.

GenericFactory() - Constructor for class org.apache.droids.helper.factories.GenericFactory

getAnchorText() - Method in interface org.apache.droids.api.Link

getAnchorText() - Method in class org.apache.droids.LinkTask

getCharset() - Method in interface org.apache.droids.api.ContentEntity

Returns charset of the entity if known.

getCharset() - Method in class org.apache.droids.protocol.http.HttpContentEntity

getCompletedTasks() - Method in interface org.apache.droids.api.TaskMaster

getCompletedTasks() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getCompletedTasks() - Method in class org.apache.droids.impl.SequentialTaskMaster

getContentType() - Method in exception org.apache.droids.exception.ParserNotFoundException

If not constructed via message only it will return the content typee which has caused the problem

getData() - Method in interface org.apache.droids.api.Parse

Other data extracted from the page.

getData() - Method in class org.apache.droids.parse.ParseImpl

getDelayMillis() - Method in interface org.apache.droids.api.DelayTimer

Returns the value of the delay between request.

getDelayMillis() - Method in class org.apache.droids.delay.GaussianRandomDelayTimer

getDelayMillis() - Method in class org.apache.droids.delay.RandomDelayTimer

getDelayMillis() - Method in class org.apache.droids.delay.SimpleDelayTimer

Gets the delay time.

getDelaySpread() - Method in class org.apache.droids.delay.RandomDelayTimer

getDelayTimer() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getDepth() - Method in interface org.apache.droids.api.Task

getDepth() - Method in class org.apache.droids.LinkTask

getDepth() - Method in class org.apache.droids.robot.walker.FileTask

getDroid(String) - Method in class org.apache.droids.helper.factories.DroidFactory

Lookup a droid by its name and return it.

getElements() - Method in class org.apache.droids.parse.html.HtmlParser

getElements() - Method in class org.apache.droids.parse.html.LinkExtractor

getEventFactory() - Method in class org.apache.droids.helper.StAX

Get the ready to used EventFactory

getEventParser(InputStream) - Method in class org.apache.droids.helper.StAX

Get an event Parser based on the incoming stream

getException() - Method in class org.apache.droids.monitor.WorkBean

getExecutionState() - Method in interface org.apache.droids.api.TaskMaster

getExecutionState() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getExecutionState() - Method in class org.apache.droids.impl.SequentialTaskMaster

getFile() - Method in class org.apache.droids.robot.walker.FileTask

getFilteredOutlinks(Parse) - Method in class org.apache.droids.robot.crawler.CrawlingWorker

getFiltersFactory() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

getFinishedWorking() - Method in interface org.apache.droids.api.TaskMaster

getFinishedWorking() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getFinishedWorking() - Method in class org.apache.droids.impl.SequentialTaskMaster

getFrom() - Method in interface org.apache.droids.api.Link

From where the link was created

getFrom() - Method in class org.apache.droids.LinkTask

getHandlerFactory() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

getId() - Method in interface org.apache.droids.api.Task

The id of the task.

getId() - Method in class org.apache.droids.LinkTask

getId() - Method in class org.apache.droids.robot.walker.FileTask

getLastCompletedTask() - Method in interface org.apache.droids.api.TaskMaster

getLastCompletedTask() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getLastCompletedTask() - Method in class org.apache.droids.impl.SequentialTaskMaster

getLastModifedDate() - Method in class org.apache.droids.LinkTask

getLastModifiedDate() - Method in interface org.apache.droids.api.Link

last modified date

getLastModifiedDate() - Method in class org.apache.droids.LinkTask

getLinks() - Method in class org.apache.droids.parse.html.LinkExtractor

getMap() - Method in class org.apache.droids.helper.factories.GenericFactory

Get the register which contains all components.

getMaxDepth() - Method in class org.apache.droids.impl.SimpleTaskValidator

getMaxSize() - Method in class org.apache.droids.impl.SimpleTaskQueue

getMaxThreads() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

Get number of maximum allowed threads

getMimeType() - Method in interface org.apache.droids.api.ContentEntity

Returns MIME type of the entity.

getMimeType() - Method in class org.apache.droids.protocol.http.HttpContentEntity

getMinimumDelay() - Method in class org.apache.droids.delay.RandomDelayTimer

getMonitor() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getName() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getNewWorker() - Method in interface org.apache.droids.api.Droid

Ask the droid for a new worker

getNewWorker() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

getNewWorker() - Method in class org.apache.droids.robot.walker.SimpleWalkingDroid

getNewWorker() - Method in interface org.apache.droids.robot.walker.WalkingDroid

getOutlinks() - Method in interface org.apache.droids.api.Parse

Outlinks extracted from the Content Entity

getOutlinks() - Method in class org.apache.droids.parse.ParseImpl

getOutputDir() - Method in class org.apache.droids.handle.Save

Get the directory where we want to save the stream.

getParse() - Method in interface org.apache.droids.api.ContentEntity

Returns the parse object from a former processing step May be null if not available.

getParse() - Method in class org.apache.droids.protocol.http.HttpContentEntity

getParser(String) - Method in class org.apache.droids.helper.factories.ParserFactory

Lookup a parser by its identifier (content type) and return it.

getParser(InputStream) - Method in class org.apache.droids.helper.StAX

Get a stream Parser based on the incoming stream

getParserFactory() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

getProtocol(URI) - Method in class org.apache.droids.helper.factories.ProtocolFactory

Will lookup a protocol based on the underlying uri

getProtocolFactory() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

getQueue() - Method in class org.apache.droids.AbstractDroid

getRunningTasks() - Method in class org.apache.droids.monitor.SimpleWorkMonitor

getSize() - Method in interface org.apache.droids.api.TaskQueue

How many task do we have left in the queue.

getSize() - Method in class org.apache.droids.impl.SimpleTaskQueue

getStartTime() - Method in interface org.apache.droids.api.TaskMaster

getStartTime() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

getStartTime() - Method in class org.apache.droids.impl.SequentialTaskMaster

getStartTime() - Method in class org.apache.droids.monitor.WorkBean

getStreamWriter(OutputStream) - Method in class org.apache.droids.helper.StAX

Get a stream writer based on the incoming stream

getTask(String) - Method in interface org.apache.droids.api.TaskQueue

Return the task that is identified with the given id

getTask(String) - Method in class org.apache.droids.impl.SimpleTaskQueue

getTask() - Method in class org.apache.droids.monitor.WorkBean

getTaskDate() - Method in interface org.apache.droids.api.Task

When was the task created

getTaskDate() - Method in class org.apache.droids.LinkTask

getTaskDate() - Method in class org.apache.droids.robot.walker.FileTask

getTaskMaster() - Method in class org.apache.droids.AbstractDroid

getTaskMaster() - Method in interface org.apache.droids.api.Droid

Get the task master

getTaskValidator() - Method in class org.apache.droids.impl.SimpleTaskQueue

getText() - Method in interface org.apache.droids.api.Parse

The textual content of the page.

getText() - Method in class org.apache.droids.parse.ParseImpl

getTo() - Method in interface org.apache.droids.api.Link

To where the link is pointing to

getTo() - Method in class org.apache.droids.LinkTask

getURI() - Method in interface org.apache.droids.api.Link

getUri() - Method in exception org.apache.droids.exception.ProtocolNotFoundException

Will return the url which has caused the problem

getURI() - Method in class org.apache.droids.LinkTask

getUrl() - Method in exception org.apache.droids.exception.ParserNotFoundException

If not constructed via message only it will return the url which has caused the problem

getUserAgent() - Method in class org.apache.droids.protocol.http.HttpProtocol

getWeight() - Method in class org.apache.droids.LinkTask

getWorkBean(T) - Method in class org.apache.droids.monitor.SimpleWorkMonitor

getWorker() - Method in class org.apache.droids.monitor.WorkBean

getWriter(OutputStream) - Method in class org.apache.droids.helper.StAX

Get an event writer based on the incoming stream

H

handle(URI, ContentEntity) - Method in interface org.apache.droids.api.Handler

handle(URI, ContentEntity) - Method in class org.apache.droids.handle.Save

handle(URI, ContentEntity) - Method in class org.apache.droids.handle.Sysout

handle(URI, ContentEntity) - Method in class org.apache.droids.helper.factories.HandlerFactory

Will traverse all registered handler and execute them.

handle(ContentEntity, Link) - Method in class org.apache.droids.robot.crawler.CrawlingWorker

handleException(Exception) - Method in interface org.apache.droids.api.TaskExceptionHandler

handleException(Exception) - Method in class org.apache.droids.impl.DefaultTaskExceptionHandler

Handler - Interface in org.apache.droids.api

A handler is a component that uses the stream, the parse and url to invoke arbitrary business logic on the objects.

HandlerFactory - Class in org.apache.droids.helper.factories

Factory that will traverse all registered handler and execute them.

HandlerFactory() - Constructor for class org.apache.droids.helper.factories.HandlerFactory

hasNext() - Method in interface org.apache.droids.api.TaskQueue

Do we have more task waiting for service

hasNext() - Method in class org.apache.droids.impl.SimpleTaskQueue

HtmlParser - Class in org.apache.droids.parse.html

HtmlParser() - Constructor for class org.apache.droids.parse.html.HtmlParser

HttpClientContentLoader - Class in org.apache.droids.protocol.http

ContentLoader based on HttpClient 4.0.

HttpClientContentLoader(HttpClient) - Constructor for class org.apache.droids.protocol.http.HttpClientContentLoader

HttpContentEntity - Class in org.apache.droids.protocol.http

HttpContentEntity(HttpEntity) - Constructor for class org.apache.droids.protocol.http.HttpContentEntity

HttpProtocol - Class in org.apache.droids.protocol.http

Protocol handler based on HttpClient 4.0.

HttpProtocol(HttpClient) - Constructor for class org.apache.droids.protocol.http.HttpProtocol

HttpProtocol() - Constructor for class org.apache.droids.protocol.http.HttpProtocol

I

init() - Method in interface org.apache.droids.api.Droid

Initialize the queue.

init() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

init() - Method in class org.apache.droids.robot.walker.SimpleWalkingDroid

InvalidLinkException - Exception in org.apache.droids.exception

Signals Link format violation.

InvalidLinkException(String) - Constructor for exception org.apache.droids.exception.InvalidLinkException

InvalidLinkException(String, Throwable) - Constructor for exception org.apache.droids.exception.InvalidLinkException

InvalidTaskException - Exception in org.apache.droids.exception

InvalidTaskException(String) - Constructor for exception org.apache.droids.exception.InvalidTaskException

isAllowed(URI) - Method in interface org.apache.droids.api.Protocol

Some protocols (like http) offer a mechanism to evaluate whether the client can request a given url (in http this is the robots.txt configuration)

isAllowed(URI) - Method in class org.apache.droids.protocol.file.FileProtocol

isAllowed(URI) - Method in class org.apache.droids.protocol.http.HttpProtocol

isForceAllow() - Method in class org.apache.droids.protocol.http.HttpProtocol

You can force that a site is allowed (ignoring the robots.txt).

isIncludeHost() - Method in class org.apache.droids.handle.Save

Do we want to prefix the export dir with the host name.

isWorking() - Method in class org.apache.droids.impl.SequentialTaskMaster

L

lastCompletedTask - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

Link - Interface in org.apache.droids.api

Simple extension of a Task.

LinkExtractor - Class in org.apache.droids.parse.html

LinkExtractor(Link, Map<String, String>) - Constructor for class org.apache.droids.parse.html.LinkExtractor

LinkTask - Class in org.apache.droids

LinkTask(Link, URI, int) - Constructor for class org.apache.droids.LinkTask

LinkTask(Link, URI, int, int) - Constructor for class org.apache.droids.LinkTask

load(URI) - Method in interface org.apache.droids.api.Protocol

Return the content entity represent of the url

load(URI) - Method in class org.apache.droids.protocol.file.FileProtocol

load(URI) - Method in class org.apache.droids.protocol.http.HttpClientContentLoader

load(URI) - Method in class org.apache.droids.protocol.http.HttpProtocol

log - Variable in class org.apache.droids.helper.Loggable

log - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

log - Variable in class org.apache.droids.parse.html.LinkExtractor

Loggable - Class in org.apache.droids.helper

Simple wrapper class to easier debug/log.

Loggable() - Constructor for class org.apache.droids.helper.Loggable

M

main(String[]) - Static method in class org.apache.droids.robot.crawler.CrawlingDroid

main(String[]) - Static method in class org.apache.droids.robot.walker.SimpleWalkingDroid

ManagedContentEntity - Interface in org.apache.droids.api

Abstract interface representing a body of content managed by a Droid.

match(String) - Method in class org.apache.droids.net.RegexRule

Checks if a url matches this rule.

maxThreads - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

merge(T) - Method in interface org.apache.droids.api.TaskQueue

Add a Task to the queue...

merge(Collection<? extends T>) - Method in interface org.apache.droids.api.TaskQueue

Merge a given list of tasks with the current queue.

merge(T) - Method in class org.apache.droids.impl.SimpleTaskQueue

merge(Collection<? extends T>) - Method in class org.apache.droids.impl.SimpleTaskQueue

Tasks with duplicate keys will be overwritten

merge(T) - Method in class org.apache.droids.impl.SimpleTaskQueueWithHistory

minimumDelay - Variable in class org.apache.droids.delay.RandomDelayTimer

monitor - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

MultiThreadedTaskMaster<T extends Task> - Class in org.apache.droids.impl

This task master provides a base implementation that support multithreaded task processing powered by a ThreadPoolExecutor.

MultiThreadedTaskMaster() - Constructor for class org.apache.droids.impl.MultiThreadedTaskMaster

N

name - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

next() - Method in interface org.apache.droids.api.TaskQueue

Return the next task that is waiting for service

next() - Method in class org.apache.droids.impl.SimpleTaskQueue

O

obtainContent() - Method in interface org.apache.droids.api.ContentEntity

Returns content of the entity as an input stream.

obtainContent() - Method in class org.apache.droids.protocol.http.HttpContentEntity

org.apache.droids - package org.apache.droids

This package is the principal package for Apache Droids.

org.apache.droids.api - package org.apache.droids.api

This package defines all interfaces that we are using for droids.

org.apache.droids.delay - package org.apache.droids.delay

This package is the principal package for Apache Droids Delay Timers.

org.apache.droids.exception - package org.apache.droids.exception

This package defines some custom exceptions that we are using in droids.

org.apache.droids.handle - package org.apache.droids.handle

This package contains some basic implementations of various handlers.

org.apache.droids.helper - package org.apache.droids.helper

This package contains various helper.

org.apache.droids.helper.factories - package org.apache.droids.helper.factories

This package contains all core factories that we use in Droids.

org.apache.droids.impl - package org.apache.droids.impl

org.apache.droids.monitor - package org.apache.droids.monitor

org.apache.droids.net - package org.apache.droids.net

This package contains various helper for the work with protocols and network communication.

org.apache.droids.parse - package org.apache.droids.parse

This package contains various helper and implementations around parsing.

org.apache.droids.parse.html - package org.apache.droids.parse.html

This package contains various parser.

org.apache.droids.protocol - package org.apache.droids.protocol

This package contains various class around the support of protocol specific classes.

org.apache.droids.protocol.file - package org.apache.droids.protocol.file

This package contains various file protocol implementations.

org.apache.droids.protocol.http - package org.apache.droids.protocol.http

This package contains various http protocol implementations.

org.apache.droids.robot.crawler - package org.apache.droids.robot.crawler

org.apache.droids.robot.walker - package org.apache.droids.robot.walker

outlinks - Variable in class org.apache.droids.parse.ParseImpl

P

Parse - Interface in org.apache.droids.api

Wrapper object that encapsulate the result of the parsing of the underlying document.

parse(ContentEntity, Link) - Method in interface org.apache.droids.api.Parser

Creates the parse for some content.

parse(ContentEntity, Link) - Method in class org.apache.droids.parse.html.HtmlParser

ParseImpl - Class in org.apache.droids.parse

Default implementation of Parse

ParseImpl() - Constructor for class org.apache.droids.parse.ParseImpl

ParseImpl(String, Collection<Link>) - Constructor for class org.apache.droids.parse.ParseImpl

ParseImpl(String, Object, Collection<Link>) - Constructor for class org.apache.droids.parse.ParseImpl

Parser - Interface in org.apache.droids.api

Simple parser that is only forcing to return a parse object.

ParserFactory - Class in org.apache.droids.helper.factories

Factory that will lookup a parser by its identifier and return it.

ParserFactory() - Constructor for class org.apache.droids.helper.factories.ParserFactory

ParserNotFoundException - Exception in org.apache.droids.exception

ParserNotFoundException gives a detailed exception for problems that can occur while parsing a task.

ParserNotFoundException(String, String) - Constructor for exception org.apache.droids.exception.ParserNotFoundException

Create an exception for the given url and content type

ParserNotFoundException(String, String, String) - Constructor for exception org.apache.droids.exception.ParserNotFoundException

Create an exception for the given url and content type

ParserNotFoundException(String) - Constructor for exception org.apache.droids.exception.ParserNotFoundException

Constructs a new exception with the specified detail message.

peek(int) - Method in interface org.apache.droids.api.TaskQueue

Get the top 'size' elements in the queue

peek(int) - Method in class org.apache.droids.impl.SimpleTaskQueue

pipe(Reader, Writer) - Static method in class org.apache.droids.handle.WriterHandler

Pipes everything from the reader to the writer via a buffer

pool - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

processAllTasks(TaskQueue<T>, Droid<T>) - Method in interface org.apache.droids.api.TaskMaster

processAllTasks(TaskQueue<T>, Droid<T>) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

The queue has been initialized

processAllTasks(TaskQueue<T>, Droid<T>) - Method in class org.apache.droids.impl.SequentialTaskMaster

The queue has been initialized

Protocol - Interface in org.apache.droids.api

The protocol interface is a wrapper to hide the underlying implementation of the communication at protocol level.

ProtocolFactory - Class in org.apache.droids.helper.factories

Factory that will lookup a protocol plugin and return it.

ProtocolFactory() - Constructor for class org.apache.droids.helper.factories.ProtocolFactory

ProtocolNotFoundException - Exception in org.apache.droids.exception

If we do not have any instance of a protocol registered for the iven url.

ProtocolNotFoundException(URI) - Constructor for exception org.apache.droids.exception.ProtocolNotFoundException

Create an exception for the given url

ProtocolNotFoundException(URI, String) - Constructor for exception org.apache.droids.exception.ProtocolNotFoundException

Create an exception for the given url and detailed message

Q

queue - Variable in class org.apache.droids.AbstractDroid

queue - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

R

random - Variable in class org.apache.droids.delay.RandomDelayTimer

RandomDelayTimer - Class in org.apache.droids.delay

RandomDelayTimer() - Constructor for class org.apache.droids.delay.RandomDelayTimer

RandomDelayTimer(int, int) - Constructor for class org.apache.droids.delay.RandomDelayTimer

RegexRule - Class in org.apache.droids.net

A generic regular expression rule.

RegexRule(boolean) - Constructor for class org.apache.droids.net.RegexRule

Constructs a new regular expression rule.

RegexURLFilter - Class in org.apache.droids.net

Regular expression implementation of an UrlFilter.

RegexURLFilter() - Constructor for class org.apache.droids.net.RegexURLFilter

resolve(String) - Method in class org.apache.droids.helper.factories.GenericFactory

Will lookup which component is linked to the name and will return it.

S

Save - Class in org.apache.droids.handle

Handler which is writing the stream to the file system.

Save() - Constructor for class org.apache.droids.handle.Save

SequentialTaskMaster<T extends Task> - Class in org.apache.droids.impl

SequentialTaskMaster() - Constructor for class org.apache.droids.impl.SequentialTaskMaster

setAnchorText(String) - Method in class org.apache.droids.LinkTask

setData(Object) - Method in class org.apache.droids.parse.ParseImpl

setDelayMillis(long) - Method in class org.apache.droids.delay.SimpleDelayTimer

Sets the delay time.

setDelaySpread(int) - Method in class org.apache.droids.delay.RandomDelayTimer

setDelayTimer(DelayTimer) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

setDelayTimer(DelayTimer) - Method in class org.apache.droids.impl.SequentialTaskMaster

setElements(Map<String, String>) - Method in class org.apache.droids.parse.html.HtmlParser

setExceptionHandler(TaskExceptionHandler) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

setExceptionHandler(TaskExceptionHandler) - Method in class org.apache.droids.impl.SequentialTaskMaster

setFile(String) - Method in class org.apache.droids.net.RegexURLFilter

setFiltersFactory(URLFiltersFactory) - Method in class org.apache.droids.robot.crawler.CrawlingDroid

setForceAllow(boolean) - Method in class org.apache.droids.protocol.http.HttpProtocol

You can force that a site is allowed (ignoring the robot.txt).

setHandlerFactory(HandlerFactory) - Method in class org.apache.droids.robot.crawler.CrawlingDroid

setIncludeHost(boolean) - Method in class org.apache.droids.handle.Save

Do we want to prefix the export dir with the host name.

setInitialFiles(Collection<File>) - Method in class org.apache.droids.robot.walker.SimpleWalkingDroid

setInitialFiles(Collection<File>) - Method in interface org.apache.droids.robot.walker.WalkingDroid

setInitialLocations(Collection<String>) - Method in class org.apache.droids.robot.crawler.CrawlingDroid

setLastModifedDate(Date) - Method in class org.apache.droids.LinkTask

setLinksTo(Collection<URI>) - Method in class org.apache.droids.LinkTask

setMap(Map) - Method in class org.apache.droids.helper.factories.GenericFactory

Set the register which contains all components.

setMaxDepth(int) - Method in class org.apache.droids.impl.SimpleTaskValidator

setMaxSize(int) - Method in class org.apache.droids.impl.SimpleTaskQueue

setMaxThreads(int) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

Set the maximum allowed thread count.

setMinimumDelay(int) - Method in class org.apache.droids.delay.RandomDelayTimer

setMonitor(WorkMonitor<T>) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

setName(String) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

setOutlinks(Collection<Link>) - Method in class org.apache.droids.parse.ParseImpl

setOutputDir(String) - Method in class org.apache.droids.handle.Save

Set the directory where we want to save the stream.

setParse(Parse) - Method in interface org.apache.droids.api.ManagedContentEntity

Sets the parse object.

setParse(Parse) - Method in class org.apache.droids.protocol.http.HttpContentEntity

setParserFactory(ParserFactory) - Method in class org.apache.droids.robot.crawler.CrawlingDroid

setProtocolFactory(ProtocolFactory) - Method in class org.apache.droids.robot.crawler.CrawlingDroid

setTaskDate(Date) - Method in class org.apache.droids.LinkTask

setTaskValidator(TaskValidator<T>) - Method in class org.apache.droids.impl.SimpleTaskQueue

setText(String) - Method in class org.apache.droids.parse.ParseImpl

setUserAgent(String) - Method in class org.apache.droids.protocol.http.HttpProtocol

setWeight(int) - Method in class org.apache.droids.LinkTask

shutdownAndAwaitTermination() - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

Shutdown all threads, close the pools and leave.

SimpleDelayTimer - Class in org.apache.droids.delay

SimpleDelayTimer() - Constructor for class org.apache.droids.delay.SimpleDelayTimer

SimpleDelayTimer(long) - Constructor for class org.apache.droids.delay.SimpleDelayTimer

SimpleTaskQueue<T extends Task> - Class in org.apache.droids.impl

A simple

SimpleTaskQueue() - Constructor for class org.apache.droids.impl.SimpleTaskQueue

Simple queue constructor.

SimpleTaskQueue(Queue) - Constructor for class org.apache.droids.impl.SimpleTaskQueue

SimpleTaskQueueWithHistory<T extends Task> - Class in org.apache.droids.impl

Extend the task queue to ignore any tasks we have already seen

SimpleTaskQueueWithHistory() - Constructor for class org.apache.droids.impl.SimpleTaskQueueWithHistory

Simple queue constructor.

SimpleTaskValidator<T extends Task> - Class in org.apache.droids.impl

A simple

SimpleTaskValidator() - Constructor for class org.apache.droids.impl.SimpleTaskValidator

SimpleTaskValidator(int) - Constructor for class org.apache.droids.impl.SimpleTaskValidator

SimpleWalkingDroid - Class in org.apache.droids.robot.walker

SimpleWalkingDroid(TaskQueue<FileTask>, TaskMaster<FileTask>) - Constructor for class org.apache.droids.robot.walker.SimpleWalkingDroid

SimpleWorkMonitor<T extends Task> - Class in org.apache.droids.monitor

A simple

SimpleWorkMonitor() - Constructor for class org.apache.droids.monitor.SimpleWorkMonitor

start() - Method in class org.apache.droids.AbstractDroid

start() - Method in interface org.apache.droids.api.Droid

Invoke an instance of the worker used in the droid

start() - Method in class org.apache.droids.robot.crawler.CrawlingDroid

startDocument() - Method in class org.apache.droids.parse.html.LinkExtractor

startedWorking - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

startElement(String, String, String, Attributes) - Method in class org.apache.droids.parse.html.LinkExtractor

state - Variable in class org.apache.droids.impl.MultiThreadedTaskMaster

StAX - Class in org.apache.droids.helper

Helper class that eases the usage of StAX in your plugins.

StAX() - Constructor for class org.apache.droids.helper.StAX

Easy helper to get StAX based parser and writer.

submitNewTaskRunner(ExecutorService, TaskQueue<? extends T>) - Method in class org.apache.droids.impl.MultiThreadedTaskMaster

This method is designed for sub-classing

TODO: refactor and create a specialized thread that - construct with the delayTimer and transparently apply the delay - construct with the Task and transparent set the lastCompleteTask

TODO: consider to provide all variables as argument so that the method overrider needs not to call the parent variables

TODO: caller of this method could use the Future to track the task completion and perform the termination and other checking.

Sysout - Class in org.apache.droids.handle

Handler that write the stream to the sysout.

Sysout() - Constructor for class org.apache.droids.handle.Sysout

T

Task - Interface in org.apache.droids.api

A task is a working instruction for a droid.

TaskExceptionHandler - Interface in org.apache.droids.api

TaskExceptionResult - Enum in org.apache.droids.api

taskMaster - Variable in class org.apache.droids.AbstractDroid

TaskMaster<T extends Task> - Interface in org.apache.droids.api

Responsible for running all the tasks

TaskMaster.ExecutionState - Enum in org.apache.droids.api

TaskQueue<T extends Task> - Interface in org.apache.droids.api

A queue is the data structure where the different tasks are waiting for service.

TaskValidator<T extends Task> - Interface in org.apache.droids.api

text - Variable in class org.apache.droids.parse.ParseImpl

toString() - Method in class org.apache.droids.robot.walker.FileTask

U

URLFilter - Interface in org.apache.droids.api

Filter to limit the urls that we want to allow in our queue.

URLFiltersFactory - Class in org.apache.droids.helper.factories

Factory that will traverse all registered filter and execute them.

URLFiltersFactory() - Constructor for class org.apache.droids.helper.factories.URLFiltersFactory

V

validateTask(T) - Method in interface org.apache.droids.api.TaskValidator

This will take a task and make sure it is valid.

validateTask(T) - Method in class org.apache.droids.impl.SimpleTaskQueue

validateTask(T) - Method in class org.apache.droids.impl.SimpleTaskValidator

valueOf(String) - Static method in enum org.apache.droids.api.TaskExceptionResult

Returns the enum constant of this type with the specified name.

valueOf(String) - Static method in enum org.apache.droids.api.TaskMaster.ExecutionState

Returns the enum constant of this type with the specified name.

values() - Static method in enum org.apache.droids.api.TaskExceptionResult

Returns an array containing the constants of this enum type, in the order they are declared.

values() - Static method in enum org.apache.droids.api.TaskMaster.ExecutionState

Returns an array containing the constants of this enum type, in the order they are declared.

W

WalkingDroid - Interface in org.apache.droids.robot.walker

WorkBean<T extends Task> - Class in org.apache.droids.monitor

A simple

WorkBean(T, Worker<T>) - Constructor for class org.apache.droids.monitor.WorkBean

Worker<T extends Task> - Interface in org.apache.droids.api

A worker is the unit that is doing the actual work.

WorkMonitor<T extends Task> - Interface in org.apache.droids.api

WriterHandler - Class in org.apache.droids.handle

Wrapper that allows you to pipe a stream from a reader to a writer via a buffer

WriterHandler() - Constructor for class org.apache.droids.handle.WriterHandler

A B C D E F G H I L M N O P Q R S T U V W