/[Apache-SVN]
ViewVC logotype

Revision 1492185

Jump to revision: Previous Next
Author: jpountz
Date: Wed Jun 12 13:17:49 2013 UTC (11 years, 5 months ago)
Changed paths: 21
Log Message: 
LUCENE-5042: Fix the n-gram tokenizers and filters.

This commit fixes n-gram tokenizers and filters so that they handle
supplementary characters correctly and adds the ability to pre-tokenize the
stream in tokenizers.

Changed paths

Path Details
Directorylucene/dev/trunk/lucene/CHANGES.txt modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/el/GreekLowerCaseFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/ngram/EdgeNGramTokenFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/ngram/EdgeNGramTokenizer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/ngram/NGramTokenFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/ngram/NGramTokenizer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/tr/TurkishLowerCaseFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/CharArrayMap.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/CharTokenizer.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/java/org/apache/lucene/analysis/util/CharacterUtils.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestAnalyzers.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestStemmerOverrideFilter.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/ngram/EdgeNGramTokenFilterTest.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/ngram/EdgeNGramTokenizerTest.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/ngram/NGramTokenFilterTest.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/ngram/NGramTokenizerTest.java modified , text changed
Directorylucene/dev/trunk/lucene/analysis/common/src/test/org/apache/lucene/analysis/util/TestCharacterUtils.java modified , text changed
Directorylucene/dev/trunk/lucene/build.xml modified , text changed
Directorylucene/dev/trunk/lucene/core/src/java/org/apache/lucene/util/fst/Util.java modified , text changed
Directorylucene/dev/trunk/lucene/tools/forbiddenApis/chars.txt added
Directorylucene/dev/trunk/solr/build.xml modified , text changed
infrastructure at apache.org
 ViewVC Help
Powered by ViewVC 1.1.26