public class StreamingASTokenizer extends Object implements ASTokenTypes, IASTokenizer, Closeable
EOF, HIDDEN_TOKEN_BUILTIN_NS, HIDDEN_TOKEN_COMMENT, HIDDEN_TOKEN_E4X, HIDDEN_TOKEN_MULTI_LINE_COMMENT, HIDDEN_TOKEN_SINGLE_LINE_COMMENT, HIDDEN_TOKEN_STAR_ASSIGNMENT, NULL_TREE_LOOKAHEAD, TOKEN_ASDOC_COMMENT, TOKEN_ASDOC_TAG, TOKEN_ASDOC_TEXT, TOKEN_ATTRIBUTE, TOKEN_BLOCK_CLOSE, TOKEN_BLOCK_OPEN, TOKEN_COLON, TOKEN_COMMA, TOKEN_DIRECTIVE_DEFAULT_XML, TOKEN_E4X_BINDING_CLOSE, TOKEN_E4X_BINDING_OPEN, TOKEN_E4X_CDATA, TOKEN_E4X_CLOSE_TAG_START, TOKEN_E4X_COMMENT, TOKEN_E4X_DECIMAL_ENTITY, TOKEN_E4X_DOTTED_NAME_PART, TOKEN_E4X_EMPTY_TAG_END, TOKEN_E4X_ENTITY, TOKEN_E4X_EQUALS, TOKEN_E4X_HEX_ENTITY, TOKEN_E4X_NAME, TOKEN_E4X_NAME_DOT, TOKEN_E4X_OPEN_TAG_START, TOKEN_E4X_PROCESSING_INSTRUCTION, TOKEN_E4X_STRING, TOKEN_E4X_TAG_END, TOKEN_E4X_TEXT, TOKEN_E4X_WHITESPACE, TOKEN_E4X_XMLLIST_CLOSE, TOKEN_E4X_XMLNS, TOKEN_ELLIPSIS, TOKEN_IDENTIFIER, TOKEN_KEYWORD_AS, TOKEN_KEYWORD_BREAK, TOKEN_KEYWORD_CASE, TOKEN_KEYWORD_CATCH, TOKEN_KEYWORD_CLASS, TOKEN_KEYWORD_CONST, TOKEN_KEYWORD_CONTINUE, TOKEN_KEYWORD_DEFAULT, TOKEN_KEYWORD_DELETE, TOKEN_KEYWORD_DO, TOKEN_KEYWORD_ELSE, TOKEN_KEYWORD_FALSE, TOKEN_KEYWORD_FINALLY, TOKEN_KEYWORD_FOR, TOKEN_KEYWORD_FUNCTION, TOKEN_KEYWORD_IF, TOKEN_KEYWORD_IMPORT, TOKEN_KEYWORD_IN, TOKEN_KEYWORD_INCLUDE, TOKEN_KEYWORD_INSTANCEOF, TOKEN_KEYWORD_INTERFACE, TOKEN_KEYWORD_IS, TOKEN_KEYWORD_NEW, TOKEN_KEYWORD_NULL, TOKEN_KEYWORD_PACKAGE, TOKEN_KEYWORD_RETURN, TOKEN_KEYWORD_SUPER, TOKEN_KEYWORD_SWITCH, TOKEN_KEYWORD_THIS, TOKEN_KEYWORD_THROW, TOKEN_KEYWORD_TRUE, TOKEN_KEYWORD_TRY, TOKEN_KEYWORD_TYPEOF, TOKEN_KEYWORD_USE, TOKEN_KEYWORD_VAR, TOKEN_KEYWORD_VOID, TOKEN_KEYWORD_WHILE, TOKEN_KEYWORD_WITH, TOKEN_LITERAL_HEX_NUMBER, TOKEN_LITERAL_NUMBER, TOKEN_LITERAL_REGEXP, TOKEN_LITERAL_STRING, TOKEN_LITERAL_XMLLIST, TOKEN_MODIFIER_DYNAMIC, TOKEN_MODIFIER_FINAL, TOKEN_MODIFIER_NATIVE, TOKEN_MODIFIER_OVERRIDE, TOKEN_MODIFIER_STATIC, TOKEN_MODIFIER_VIRTUAL, TOKEN_NAMESPACE_ANNOTATION, TOKEN_NAMESPACE_NAME, TOKEN_OPERATOR_ASSIGNMENT, TOKEN_OPERATOR_ATSIGN, TOKEN_OPERATOR_BITWISE_AND, TOKEN_OPERATOR_BITWISE_AND_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT, TOKEN_OPERATOR_BITWISE_LEFT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_NOT, TOKEN_OPERATOR_BITWISE_OR, TOKEN_OPERATOR_BITWISE_OR_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT, TOKEN_OPERATOR_BITWISE_UNSIGNED_RIGHT_SHIFT_ASSIGNMENT, TOKEN_OPERATOR_BITWISE_XOR, TOKEN_OPERATOR_BITWISE_XOR_ASSIGNMENT, TOKEN_OPERATOR_DECREMENT, TOKEN_OPERATOR_DESCENDANT_ACCESS, TOKEN_OPERATOR_DIVISION, TOKEN_OPERATOR_DIVISION_ASSIGNMENT, TOKEN_OPERATOR_EQUAL, TOKEN_OPERATOR_GREATER_THAN, TOKEN_OPERATOR_GREATER_THAN_EQUALS, TOKEN_OPERATOR_INCREMENT, TOKEN_OPERATOR_LESS_THAN, TOKEN_OPERATOR_LESS_THAN_EQUALS, TOKEN_OPERATOR_LOGICAL_AND, TOKEN_OPERATOR_LOGICAL_AND_ASSIGNMENT, TOKEN_OPERATOR_LOGICAL_NOT, TOKEN_OPERATOR_LOGICAL_OR, TOKEN_OPERATOR_LOGICAL_OR_ASSIGNMENT, TOKEN_OPERATOR_MEMBER_ACCESS, TOKEN_OPERATOR_MINUS, TOKEN_OPERATOR_MINUS_ASSIGNMENT, TOKEN_OPERATOR_MODULO, TOKEN_OPERATOR_MODULO_ASSIGNMENT, TOKEN_OPERATOR_MULTIPLICATION_ASSIGNMENT, TOKEN_OPERATOR_NOT_EQUAL, TOKEN_OPERATOR_NS_QUALIFIER, TOKEN_OPERATOR_PLUS, TOKEN_OPERATOR_PLUS_ASSIGNMENT, TOKEN_OPERATOR_STAR, TOKEN_OPERATOR_STRICT_EQUAL, TOKEN_OPERATOR_STRICT_NOT_EQUAL, TOKEN_OPERATOR_TERNARY, TOKEN_PAREN_CLOSE, TOKEN_PAREN_OPEN, TOKEN_RESERVED_WORD_CONFIG, TOKEN_RESERVED_WORD_EACH, TOKEN_RESERVED_WORD_EXTENDS, TOKEN_RESERVED_WORD_GET, TOKEN_RESERVED_WORD_GOTO, TOKEN_RESERVED_WORD_IMPLEMENTS, TOKEN_RESERVED_WORD_NAMESPACE, TOKEN_RESERVED_WORD_SET, TOKEN_SEMICOLON, TOKEN_SQUARE_CLOSE, TOKEN_SQUARE_OPEN, TOKEN_TYPED_COLLECTION_CLOSE, TOKEN_TYPED_COLLECTION_OPEN, TOKEN_TYPED_LITERAL_CLOSE, TOKEN_TYPED_LITERAL_OPEN, TOKEN_VOID_0
Constructor and Description |
---|
StreamingASTokenizer()
You should probably not use this constructor.
|
StreamingASTokenizer(Reader reader)
You should probably not use this constructor.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes the underlying reader
|
protected static StreamingASTokenizer |
create(IFileSpecification fileSpec,
IncludeHandler includeHandler)
Create a tokenizer from a source file.
|
protected static StreamingASTokenizer |
createForASParser(IFileSpecification fileSpec,
IncludeHandler includeHandler,
boolean followIncludes,
List<String> includedFiles)
Create a tokenizer for
ASParser#parseFile() . |
protected static StreamingASTokenizer |
createForInlineExpressionParsing(Reader reader,
String path)
Create a tokenizer to parse an Expression.
|
protected static StreamingASTokenizer |
createForInlineScriptScopeBuilding(Reader reader,
String path,
IncludeHandler includeHandler,
int offsetAdjustment,
int lineAdjustment,
int columnAdjustment)
This creator doesn't "enter file" on creation.
|
static StreamingASTokenizer |
createForMXMLIndexing(String fileName)
Creates a tokenizer suitable for the mxml indexing code. fragments the
new tokenizer will tokenize.
|
static StreamingASTokenizer |
createForRepairingASTokenizer(Reader reader,
String path,
IncludeHandler includeHandler)
This method can create a
StreamingASTokenizer with optional
"follow includes". |
int |
getEndOffset()
Retrieve the end offset of the file.
|
String |
getSourcePath()
Gets the source path to the file being tokenized.
|
List<ICompilerProblem> |
getTokenizationProblems()
Returns a collection of problems that have been encountered while
scanning.
|
ASToken[] |
getTokens(Reader reader)
|
ASToken[] |
getTokens(Reader reader,
ITokenStreamFilter filter) |
IASToken[] |
getTokens(String range)
|
boolean |
hasEncounteredIncludeStatements()
Indicated whether this tokenizer has encountered include statements,
regardless of whether it is set to follow them or not
|
boolean |
hasTokenizationProblems()
Indicated that we have tokenization problems.
|
ASToken |
next()
Returns the next token that can be produced from the underlying reader.
|
ASToken |
next(ITokenStreamFilter filter)
Returns the next token that can be produced from the underlying reader
|
void |
setCollectComments(boolean collect)
Sets whether we comments are collected: single line and multi-line.
|
void |
setFollowIncludes(boolean followIncludes)
Sets whether we follow include statements, including their tokens.
|
void |
setIgnoreKeywords(boolean ignore)
Sets whether we ignore keywords while scanning.
|
void |
setIncludeHandler(IncludeHandler handler)
Sets the include handler used by this tokenizer to get
IFileSpecification for included files. |
void |
setIsMetadataAware(boolean aware)
Sets whether we will find metadata constructs Default is
true |
void |
setPath(String path)
Sets the path to the file this tokenizer is scanning
|
void |
setReader(Reader reader)
Sets the
Reader that supplies the content to this tokenizer. |
void |
setReader(Reader reader,
int offset,
int line,
int column)
Sets the
Reader that supplies the content to this tokenizer. |
void |
setScanningFullContent(boolean full)
Sets whether we are scanning a full file, or a fragment.
|
void |
setSourcePositionAdjustment(int offset,
int line,
int column)
Allows for the adjustment of offset, line and column information when
parsing subsequences of text.
|
void |
setTokenFilter(ITokenStreamFilter filter)
Sets the
ITokenStreamFilter used to filter out unwanted tokens |
public StreamingASTokenizer(Reader reader)
TODO: make this private.
public StreamingASTokenizer()
TODO: make this private.
public static StreamingASTokenizer createForMXMLIndexing(String fileName)
protected static StreamingASTokenizer create(IFileSpecification fileSpec, IncludeHandler includeHandler) throws FileNotFoundException
ASCompilationUnit
.fileSpec
- File specification provides the reader and the file path.includeHandler
- Include handler.FileNotFoundException
- errorprotected static StreamingASTokenizer createForASParser(IFileSpecification fileSpec, IncludeHandler includeHandler, boolean followIncludes, List<String> includedFiles) throws FileNotFoundException
ASParser#parseFile()
.fileSpec
- File specification provides the reader and the file path.includeHandler
- Include handler.followIncludes
- True if included files are also parsed.includedFiles
- A list of included file paths.FileNotFoundException
- errorprotected static StreamingASTokenizer createForInlineScriptScopeBuilding(Reader reader, String path, IncludeHandler includeHandler, int offsetAdjustment, int lineAdjustment, int columnAdjustment)
protected static StreamingASTokenizer createForInlineExpressionParsing(Reader reader, String path)
public static StreamingASTokenizer createForRepairingASTokenizer(Reader reader, String path, IncludeHandler includeHandler)
StreamingASTokenizer
with optional
"follow includes". If IncludeHandler
is not null, it will follow
include
directives.reader
- Input to the tokenizer.path
- File path of the input.includeHandler
- If not null, the created tokenizer will follow
include
directives.StreamingASTokenizer
.public void setReader(Reader reader)
Reader
that supplies the content to this tokenizer. It
is up to the client to close any previous readers that have been in use.
It is also up to the client to close the reader once it has been usedreader
- a Reader
public void setReader(Reader reader, int offset, int line, int column)
Reader
that supplies the content to this tokenizer. It
is up to the client to close any previous readers that have been in use.
It is also up to the client to close the reader once it has been usedreader
- a Reader
offset
- Offset adjustment. If the specified reader is reading from
a string extracted from a source file, this should be the offset of the
first character read from the reader in the source file.line
- Line adjustment.column
- Column adjustmentpublic void setPath(String path)
setPath
in interface IASTokenizer
path
- a file pathpublic void setSourcePositionAdjustment(int offset, int line, int column)
offset
- The offset where the fragment starts.line
- The line where the fragment starts. This should be a
ZERO-based line numbercolumn
- The column where the fragment starts. This should be a
ZERO-based column numberpublic void setCollectComments(boolean collect)
false
setCollectComments
in interface IASTokenizer
collect
- true if we should collect commentspublic void setFollowIncludes(boolean followIncludes)
true
setFollowIncludes
in interface IASTokenizer
followIncludes
- true if we should follow includespublic void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
public void setIgnoreKeywords(boolean ignore)
false
ignore
- true if we should ignore keywordspublic void setScanningFullContent(boolean full)
true
full
- true if we are scanning a full file.public void setIsMetadataAware(boolean aware)
true
aware
- true if we will find metadatapublic void setTokenFilter(ITokenStreamFilter filter)
ITokenStreamFilter
used to filter out unwanted tokensfilter
- the token filter to alter the stream returned from the
tokenizerpublic void setIncludeHandler(IncludeHandler handler)
IFileSpecification
for included files.handler
- IncludeHandler
this tokenizer should use.public boolean hasTokenizationProblems()
public boolean hasEncounteredIncludeStatements()
public List<ICompilerProblem> getTokenizationProblems()
public ASToken[] getTokens(Reader reader, ITokenStreamFilter filter)
public ASToken[] getTokens(Reader reader)
IASTokenizer
getTokens
in interface IASTokenizer
reader
- a Java Reader
IASToken
objectspublic IASToken[] getTokens(String range)
IASTokenizer
getTokens
in interface IASTokenizer
IASToken
objectspublic final ASToken next(ITokenStreamFilter filter)
filter
- an ITokenStreamFilter
to restrict the tokens that
are returnedpublic final ASToken next()
If the forked "include file tokenizer" is open (not null), return the next token from it. If the forked tokenizer reaches the end of the included file, close (set to null) the forked tokenizer and return token from the main source file.
public final int getEndOffset()
The result is the end offset of the file, not the offset of the last
token, this allows any trailing space to be included so that the parser
can span the result FileNode
to the entire file.
public String getSourcePath()
Copyright © 2016 The Apache Software Foundation. All rights reserved.