------
            Features
            ------
            ------
             2011-02-09
            ------

Feature list

  * <<Customizable>>

    Completely controlled by its default.properties which can be easily be overridden by creating a file build.properties and overriding the default properties that are needed.

  * <<Multi-threaded>>

    The architecture is that a robot (e.g. HelloCrawler controls various worker (threads) that are doing the actual work.

  * <<Honor robots.txt>>

    By default droids honors the robot.txt. However you can turn on the hostile mode of a droid (droids.protocol.http.force=true).

  * <<Crawl throttling>>

    You can configure the amount of concurrent threads that a droid can distribute to their workers (droids.maxThreads=5) and the delay time between the requests (droids.delay.request=500). You can use one of the different delay components:

  * <<Spring based - dynamics>>

    The properties mentioned above get picked up by the build process which inject them in the spring configuration.

  * <<Extensible - dynamics>>

    The spring configuration makes usage of the cocoon-configurator and its dynamic registry support (making extending droids a pleasure).