The methods of the CharTokenizer class are listed below. For a complete list of CharTokenizer class members, see the CharTokenizer Members topic.
Public Instance Methods
 Close (inherited from Tokenizer) | By default, closes the input Reader. |
| Equals (inherited from Object) | Determines whether the specified Object is equal to the current Object. |
| GetHashCode (inherited from Object) | Serves as a hash function for a particular type, suitable for use in hashing algorithms and data structures like a hash table. |
| GetType (inherited from Object) | Gets the Type of the current instance. |
| Next | Returns the next token in the stream, or null at EOS. |
| ToString (inherited from Object) | Returns a String that represents the current Object. |
Protected Instance Methods
 Finalize (inherited from Object) | Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection. |
| MemberwiseClone (inherited from Object) | Creates a shallow copy of the current Object. |
Protected Internal Instance Methods
 IsTokenChar | Returns true iff a character should be included in a token. This tokenizer generates as tokens adjacent sequences of characters which satisfy this predicate. Characters for which this is false are used to define token boundaries and are not included in tokens. |
| Normalize | Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens. |