Public Instance Constructors
WhitespaceTokenizer Constructor
|
Construct a new WhitespaceTokenizer. |
Public Instance Methods
| Close (inherited from Tokenizer) | By default, closes the input Reader. |
| Equals (inherited from Object) | Determines whether the specified Object is equal to the current Object. |
| GetHashCode (inherited from Object) | Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. |
| GetType (inherited from Object) | Gets the Type of the current instance. |
| Next (inherited from CharTokenizer) | Returns the next token in the stream, or null at EOS. |
| ToString (inherited from Object) | Returns a String that represents the current Object. |
Protected Instance Methods
 Finalize (inherited from Object) | Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. |
| MemberwiseClone (inherited from Object) | Creates a shallow copy of the current Object. |
Protected Internal Instance Fields
 input (inherited from Tokenizer) | The text source for this Tokenizer. |
Protected Internal Instance Methods
 IsTokenChar | Collects only characters which do not satisfy {@link Character#isWhitespace(char)}. |
| Normalize (inherited from CharTokenizer) | Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens. |