public class MockTokenizer
extends org.apache.lucene.analysis.Tokenizer
This tokenizer is a replacement for WHITESPACE
, SIMPLE
, and KEYWORD
tokenizers. If you are writing a component such as a TokenFilter, its a great idea to test
it wrapping this tokenizer instead for extra checks. This tokenizer has the following behavior:
setEnableChecks(boolean)
.
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_TOKEN_LENGTH |
static int |
KEYWORD
Acts Similar to KeywordTokenizer.
|
static int |
SIMPLE
Acts like LetterTokenizer.
|
static int |
WHITESPACE
Acts Similar to WhitespaceTokenizer
|
Constructor and Description |
---|
MockTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
Reader input,
int pattern,
boolean lowerCase,
int maxTokenLength) |
MockTokenizer(Reader input)
|
MockTokenizer(Reader input,
int pattern,
boolean lowerCase) |
MockTokenizer(Reader input,
int pattern,
boolean lowerCase,
int maxTokenLength) |
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
end() |
boolean |
incrementToken() |
protected boolean |
isTokenChar(int c) |
protected int |
normalize(int c) |
protected int |
readCodePoint() |
void |
reset() |
void |
reset(Reader input) |
void |
setEnableChecks(boolean enableChecks)
Toggle consumer workflow checking: if your test consumes tokenstreams normally you
should leave this enabled.
|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public static final int WHITESPACE
public static final int KEYWORD
public static final int SIMPLE
public static final int DEFAULT_MAX_TOKEN_LENGTH
public MockTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader input, int pattern, boolean lowerCase, int maxTokenLength)
public MockTokenizer(Reader input, int pattern, boolean lowerCase, int maxTokenLength)
public MockTokenizer(Reader input, int pattern, boolean lowerCase)
public MockTokenizer(Reader input)
public final boolean incrementToken() throws IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
IOException
protected int readCodePoint() throws IOException
IOException
protected boolean isTokenChar(int c)
protected int normalize(int c)
public void reset() throws IOException
reset
in class org.apache.lucene.analysis.TokenStream
IOException
public void close() throws IOException
close
in interface Closeable
close
in class org.apache.lucene.analysis.Tokenizer
IOException
public void reset(Reader input) throws IOException
reset
in class org.apache.lucene.analysis.Tokenizer
IOException
public void end() throws IOException
end
in class org.apache.lucene.analysis.TokenStream
IOException
public void setEnableChecks(boolean enableChecks)