org.apache.nutch.util
Class EncodingDetector

java.lang.Object
  org.apache.nutch.util.EncodingDetector

public class EncodingDetector
extends Object

A simple class for detecting character encodings.

Broadly this encompasses two functions, which are distinctly separate:

1. Auto detecting a set of "clues" from input text.
2. Taking a set of clues and making a "best guess" as to the "real" encoding.

A caller will often have some extra information about what the encoding might be (e.g. from the HTTP header or HTML meta-tags, often wrong but still potentially useful clues). The types of clues may differ from caller to caller. Thus a typical calling sequence is:

• Run step (1) to generate a set of auto-detected clues;
• Combine these clues with the caller-dependent "extra clues" available;
• Run step (2) to guess what the most probable answer is.

Field Summary

static org.apache.avro.util.Utf8	CONTENT_TYPE_UTF8
static org.slf4j.Logger	LOG
static String	MIN_CONFIDENCE_KEY
static int	NO_THRESHOLD

Constructor Summary

EncodingDetector(org.apache.hadoop.conf.Configuration conf)


Method Summary

void	addClue(String value, String source)
void	addClue(String value, String source, int confidence)
void	autoDetectClues(WebPage page, boolean filter)
void	clearClues() Clears all clues.
String	guessEncoding(WebPage page, String defaultValue) Guess the encoding with the previously specified list of clues.
static String	parseCharacterEncoding(org.apache.avro.util.Utf8 contentTypeUtf8) Parse the character encoding from the specified content type header.
static String	resolveEncodingAlias(String encoding)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONTENT_TYPE_UTF8

public static final org.apache.avro.util.Utf8 CONTENT_TYPE_UTF8

LOG

public static final org.slf4j.Logger LOG

NO_THRESHOLD

public static final int NO_THRESHOLD

See Also:

Constant Field Values

MIN_CONFIDENCE_KEY

public static final String MIN_CONFIDENCE_KEY

See Also:

Constant Field Values

Constructor Detail

EncodingDetector

public EncodingDetector(org.apache.hadoop.conf.Configuration conf)

Method Detail

autoDetectClues

public void autoDetectClues(WebPage page,
                            boolean filter)

addClue

public void addClue(String value,
                    String source,
                    int confidence)

addClue

public void addClue(String value,
                    String source)

guessEncoding

public String guessEncoding(WebPage page,
                            String defaultValue)

Guess the encoding with the previously specified list of clues.

Parameters:

row - URL's row

defaultValue - Default encoding to return if no encoding can be detected with enough confidence. Note that this will not be normalized with resolveEncodingAlias(java.lang.String)

Returns:

Guessed encoding or defaultValue

clearClues

public void clearClues()

Clears all clues.

resolveEncodingAlias

public static String resolveEncodingAlias(String encoding)

parseCharacterEncoding

public static String parseCharacterEncoding(org.apache.avro.util.Utf8 contentTypeUtf8)

Parse the character encoding from the specified content type header. If the content type is null, or there is no explicit character encoding, null is returned.
This method was copied from org.apache.catalina.util.RequestUtil, which is licensed under the Apache License, Version 2.0 (the "License").

Parameters:

contentType - a content type header