Overloaded. Returns the next token in the stream, or null at EOS. @deprecated The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling {@link #Next(Token)} instead..
Overloaded. Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. Reset() is not needed for the standard indexing process. However, if the Tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).
Returns true iff a character should be included in a token. This tokenizer generates as tokens adjacent sequences of characters which satisfy this predicate. Characters for which this is false are used to define token boundaries and are not included in tokens.
Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.
Close (inherited from Tokenizer)
Finalize (inherited from Object)
IsTokenChar