/[Apache-SVN]
ViewVC logotype

Revision 1619730


Jump to revision: Previous Next
Author: sarowe
Date: Fri Aug 22 10:19:06 2014 UTC (9 years, 7 months ago)
Changed paths: 14
Log Message:
LUCENE-5897, LUCENE-5400: JFlex-based tokenizers StandardTokenizer and UAX29URLEmailTokenizer tokenize extremely slowly over long sequences of text partially matching certain grammar rules.  The scanner default buffer size was reduced, and scanner buffer growth was disabled, resulting in much, much faster tokenization for these text sequences.

Changed paths

Path Details
Directorylucene/dev/trunk/lucene/CHANGES.txt modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/build.xml modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/charfilter/HTMLStripCharFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/ClassicTokenizerImpl.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/ClassicTokenizerImpl.jflex modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerImpl.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerImpl.jflex modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestStandardAnalyzer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestUAX29URLEmailTokenizer.java modified , text changed

infrastructure at apache.org
ViewVC Help
Powered by ViewVC 1.1.26