/[Apache-SVN]
ViewVC logotype

Revision 1625458


Jump to revision: Previous Next
Author: sarowe
Date: Wed Sep 17 03:58:21 2014 UTC (9 years, 6 months ago)
Changed paths: 114 (showing only 100; show all)
Log Message:
LUCENE-5897, LUCENE-5400: JFlex-based tokenizers StandardTokenizer and UAX29URLEmailTokenizer tokenize extremely slowly over long sequences of text partially matching certain grammar rules.  The scanner default buffer size was reduced, and scanner buffer growth was disabled, resulting in much, much faster tokenization for these text sequences. (merged branch_4x r1619773)

Changed paths

Path Details
Directorylucene/dev/branches/lucene_solr_4_9/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/dev-tools/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/BUILD.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/CHANGES.txt modified , text changed , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/JRE_VERSION_MIGRATION.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/LICENSE.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/MIGRATE.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/NOTICE.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/README.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/SYSTEM_REQUIREMENTS.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/build.xml modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/Lucene47WordDelimiterFilter.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/ClassicTokenizerImpl.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/ClassicTokenizerImpl.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/SUPPLEMENTARY.jflex-macro modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizer.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerImpl.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerImpl.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std31/StandardTokenizerImpl31.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std31/StandardTokenizerImpl31.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std31/UAX29URLEmailTokenizerImpl31.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std31/UAX29URLEmailTokenizerImpl31.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std34/StandardTokenizerImpl34.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std34/StandardTokenizerImpl34.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std34/UAX29URLEmailTokenizerImpl34.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std34/UAX29URLEmailTokenizerImpl34.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std36/UAX29URLEmailTokenizerImpl36.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std36/UAX29URLEmailTokenizerImpl36.jflex modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/ASCIITLD.jflex-macro modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/SUPPLEMENTARY.jflex-macro modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/StandardTokenizerImpl40.java modified , text changed , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/StandardTokenizerImpl40.jflex modified , text changed , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/UAX29URLEmailTokenizerImpl40.java modified , text changed , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/UAX29URLEmailTokenizerImpl40.jflex modified , text changed , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/std40/package.html modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestStandardAnalyzer.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestUAX29URLEmailTokenizer.java modified , text changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestLucene47WordDelimiterFilter.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/analysis/icu/src/java/org/apache/lucene/collation/ICUCollationKeyFilterFactory.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/backwards/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/benchmark/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/classification/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/classification/build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/classification/ivy.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/classification/src/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/codecs/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/common-build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/TestIndexWriterExceptions2.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/index.40.cfs.zip modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/index.40.nocfs.zip modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/index.40.optimized.cfs.zip modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/index/index.40.optimized.nocfs.zip modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/search/TestSort.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/search/TestSortDocValues.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/search/TestTopFieldCollector.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/core/src/test/org/apache/lucene/search/TestTotalHitCountCollector.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/demo/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/expressions/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/facet/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/grouping/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/highlighter/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/ivy-ignore-conflicts.properties modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/ivy-settings.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/ivy-versions.properties modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/join/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/licenses/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/memory/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/misc/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/module-build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/queries/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/queries/src/test/org/apache/lucene/queries/function/TestFunctionQuerySort.java modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/queryparser/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/replicator/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/sandbox/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/site/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/suggest/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/test-framework/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/test-framework/src/java/org/apache/lucene/codecs/cranky/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/lucene/tools/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/CHANGES.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/LICENSE.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/NOTICE.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/README.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/SYSTEM_REQUIREMENTS.txt modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/cloud-dev/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/common-build.xml modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/contrib/ modified , props changed
Directorylucene/dev/branches/lucene_solr_4_9/solr/core/ modified , props changed
[...]

infrastructure at apache.org
ViewVC Help
Powered by ViewVC 1.1.26