org.apache.droids.net
Class UrlHelper

java.lang.Object
  org.apache.droids.net.UrlHelper

public class UrlHelper
extends Object

Helper class that offers a couple of method to work with urls

Method Summary

static boolean	existUrl(URL url) Does the url exist?
static URL	findRobotsUrl(URL base, String prefix) Search the url of the robots.txt that is responsible for the given base url.
static String	getUrlPrefix(URL base) Creating a valid protocol prefix.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

findRobotsUrl

public static URL findRobotsUrl(URL base,
                                String prefix)
                         throws MalformedURLException

Search the url of the robots.txt that is responsible for the given base url. Crawl down the directory tree if needed.

Parameters:

base - base url to evaluate

prefix - the protocol prefix we want to use

Returns:

null if no robots.txt can be found otherwise the url of this file.

Throws:

MalformedURLException

getUrlPrefix

public static String getUrlPrefix(URL base)

Creating a valid protocol prefix.

Parameters:

base - the base we want to evaluate

Returns:

the prefix that is constructed from the protocol + host + port (if set)

existUrl

public static boolean existUrl(URL url)

Does the url exist?

Parameters:

url - the url to test

Returns:

true if te url exist; false otherwise