org.apache.nutch.util

Class EncodingDetector

java.lang.Object

org.apache.nutch.util.EncodingDetector

public class EncodingDetector
extends Object

A simple class for detecting character encodings.

Broadly this encompasses two functions, which are distinctly separate:

1. Auto detecting a set of "clues" from input text.
2. Taking a set of clues and making a "best guess" as to the "real" encoding.

A caller will often have some extra information about what the encoding might be (e.g. from the HTTP header or HTML meta-tags, often wrong but still potentially useful clues). The types of clues may differ from caller to caller. Thus a typical calling sequence is:

• Run step (1) to generate a set of auto-detected clues;
• Combine these clues with the caller-dependent "extra clues" available;
• Run step (2) to guess what the most probable answer is.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
static String	MIN_CONFIDENCE_KEY
static int	NO_THRESHOLD

Constructor Summary

Constructors

Constructor and Description
EncodingDetector(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	addClue(String value, String source)
void	addClue(String value, String source, int confidence)
void	autoDetectClues(Content content, boolean filter)
void	clearClues() Clears all clues.
String	guessEncoding(Content content, String defaultValue) Guess the encoding with the previously specified list of clues.
static void	main(String[] args)
static String	parseCharacterEncoding(String contentType) Parse the character encoding from the specified content type header.
static String	resolveEncodingAlias(String encoding)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

NO_THRESHOLD

public static final int NO_THRESHOLD

See Also:

Constant Field Values

MIN_CONFIDENCE_KEY

public static final String MIN_CONFIDENCE_KEY

See Also:

Constant Field Values

Constructor Detail

EncodingDetector

public EncodingDetector(org.apache.hadoop.conf.Configuration conf)

Method Detail

autoDetectClues

public void autoDetectClues(Content content,
                   boolean filter)

addClue

public void addClue(String value,
           String source,
           int confidence)

addClue

public void addClue(String value,
           String source)

guessEncoding

public String guessEncoding(Content content,
                   String defaultValue)

Guess the encoding with the previously specified list of clues.

Parameters:

content - Content instance

defaultValue - Default encoding to return if no encoding can be detected with enough confidence. Note that this will not be normalized with resolveEncodingAlias(java.lang.String)

Returns:

Guessed encoding or defaultValue

clearClues

public void clearClues()

Clears all clues.

resolveEncodingAlias

public static String resolveEncodingAlias(String encoding)

parseCharacterEncoding

public static String parseCharacterEncoding(String contentType)

Parse the character encoding from the specified content type header. If the content type is null, or there is no explicit character encoding, null is returned.
This method was copied from org.apache.catalina.util.RequestUtil, which is licensed under the Apache License, Version 2.0 (the "License").

Parameters:

contentType - a content type header

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException