org.apache.nutch.analysis
Class NutchAnalysisTokenManager

java.lang.Object
  org.apache.nutch.analysis.NutchAnalysisTokenManager

All Implemented Interfaces:

NutchAnalysisConstants

public class NutchAnalysisTokenManager
extends Object
implements NutchAnalysisConstants

Token Manager.

Field Summary

protected char	curChar
PrintStream	debugStream Debug output.
protected org.apache.nutch.analysis.CharStream	input_stream
static String[]	jjstrLiteralImages Token literal values.
static String[]	lexStateNames Lexer state names.

Fields inherited from interface org.apache.nutch.analysis.NutchAnalysisConstants

ACRONYM, APOSTROPHE, ATSIGN, C_PLUS_PLUS, C_SHARP, CJK, COLON, DEFAULT, DIGIT, DOT, EOF, IRREGULAR_WORD, LETTER, MINUS, PLUS, QUOTE, SIGRAM, SLASH, tokenImage, WHITE, WORD, WORD_PUNCT

Constructor Summary

NutchAnalysisTokenManager(org.apache.nutch.analysis.CharStream stream)
Constructor.

NutchAnalysisTokenManager(org.apache.nutch.analysis.CharStream stream, int lexState)
Constructor.

NutchAnalysisTokenManager(Reader reader)
Constructs a token manager for the provided Reader.

Method Summary

org.apache.nutch.analysis.Token	getNextToken() Get the next Token.
protected org.apache.nutch.analysis.Token	jjFillToken()
void	ReInit(org.apache.nutch.analysis.CharStream stream) Reinitialise parser.
void	ReInit(org.apache.nutch.analysis.CharStream stream, int lexState) Reinitialise parser.
void	setDebugStream(PrintStream ds) Set debug output.
void	SwitchTo(int lexState) Switch to specified lex state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

debugStream

public PrintStream debugStream

Debug output.

jjstrLiteralImages

public static final String[] jjstrLiteralImages

Token literal values.

lexStateNames

public static final String[] lexStateNames

Lexer state names.

input_stream

protected org.apache.nutch.analysis.CharStream input_stream

curChar

protected char curChar

Constructor Detail

NutchAnalysisTokenManager

public NutchAnalysisTokenManager(Reader reader)

Constructs a token manager for the provided Reader.

NutchAnalysisTokenManager

public NutchAnalysisTokenManager(org.apache.nutch.analysis.CharStream stream)

Constructor.

NutchAnalysisTokenManager

public NutchAnalysisTokenManager(org.apache.nutch.analysis.CharStream stream,
                                 int lexState)

Constructor.

Method Detail

setDebugStream

public void setDebugStream(PrintStream ds)

Set debug output.

ReInit

public void ReInit(org.apache.nutch.analysis.CharStream stream)

Reinitialise parser.

ReInit

public void ReInit(org.apache.nutch.analysis.CharStream stream,
                   int lexState)

Reinitialise parser.

SwitchTo

public void SwitchTo(int lexState)

Switch to specified lex state.

jjFillToken

protected org.apache.nutch.analysis.Token jjFillToken()

getNextToken

public org.apache.nutch.analysis.Token getNextToken()

Get the next Token.