org.apache.nutch.net.urlnormalizer.basic

Class BasicURLNormalizer

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer

All Implemented Interfaces:

Configurable, URLNormalizer

public class BasicURLNormalizer
extends Configured
implements URLNormalizer

Converts URLs to a normal form:

• remove dot segments in path: /./ or /../
• remove default ports, e.g. 80 for protocol http://

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from interface org.apache.nutch.net.URLNormalizer

X_POINT_ID

Constructor Summary

Constructors

Constructor and Description
BasicURLNormalizer()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(String[] args)
String	normalize(String urlString, String scope)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

BasicURLNormalizer

public BasicURLNormalizer()

Method Detail

normalize

public String normalize(String urlString,
               String scope)
                 throws MalformedURLException

Specified by:

normalize in interface URLNormalizer

Throws:

MalformedURLException

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException