Returns the next token in the stream, or null at EOS

Returns the next token in the stream, or null at EOS. When possible, the input Token should be used as the returned Token (this gives fastest tokenization performance), but this is not required and a new Token may be returned. Callers may re-use a single Token instance for successive calls to this method. This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):

• A consumer must fully consume the previously returned {@link Token} before calling this method again.
• A producer must call {@link Token#Clear()} before setting the fields in it and returning it

Also, the producer must make no assumptions about a {@link Token} after it has been returned: the caller may arbitrarily change it. If the producer needs to hold onto the {@link Token} for subsequent calls, it must clone() it before storing it. Note that a {@link TokenFilter} is considered a consumer.