Package org.apache.http.norobots

Using norobots-rfc

See:
Description

Interface Summary

Rule	A robots.txt rule.

Class Summary

NoRobotClient

A Client which may be used to decide which urls on a website may be looked at, according to the norobots specification located at: http://www.robotstxt.org/wc/norobots-rfc.html

Exception Summary

NoRobotException

Application exception for anything that might go wrong in the checking of a robots.txt file.

Package org.apache.http.norobots Description

Using norobots-rfc

1. Import the class import org.apache.http.norobots.NoRobotClient;
2. Create an instance for your user-agent NoRobotClient nrc = NoRobotClient("googlebot");
3. Parse a robots.txt at a site nrc.parse( new URL( "http://www.apache.org/" ) );
4. Ask if a url is allowed boolean test = nrc.isUrlAllowed( new URL( "http://www.apache.org/index.html" ) );