The methods of the LetterTokenizer class are listed below. For a complete list of LetterTokenizer class members, see the LetterTokenizer Members topic.
Overloaded. Returns the next token in the stream, or null at EOS. @deprecated The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling {@link #Next(Token)} instead..
Overloaded. Returns the next token in the stream, or null at EOS. @deprecated The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling {@link #Next(Token)} instead..
Overloaded. Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. Reset() is not needed for the standard indexing process. However, if the Tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).
Overloaded. Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. Reset() is not needed for the standard indexing process. However, if the Tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).
Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.