Class BM25Similarity
- java.lang.Object
-
- org.apache.lucene.search.similarities.Similarity
-
- org.apache.lucene.search.similarities.BM25Similarity
-
public class BM25Similarity extends Similarity
BM25 Similarity. Introduced in Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 1994). Gaithersburg, USA, November 1994.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity
Similarity.SimScorer, Similarity.SimWeight
-
-
Field Summary
Fields Modifier and Type Field Description protected boolean
discountOverlaps
True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.
-
Constructor Summary
Constructors Constructor Description BM25Similarity()
BM25 with these default values:k1 = 1.2
,b = 0.75
.BM25Similarity(float k1, float b)
BM25 with the supplied parameter values.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected float
avgFieldLength(CollectionStatistics collectionStats)
The default implementation computes the average assumTotalTermFreq / maxDoc
, or returns1
if the index does not store sumTotalTermFreq (Lucene 3.x indexes or any field that omits frequency information).long
computeNorm(FieldInvertState state)
Computes the normalization value for a field, given the accumulated state of term processing for this field (seeFieldInvertState
).Similarity.SimWeight
computeWeight(float queryBoost, CollectionStatistics collectionStats, TermStatistics... termStats)
Compute any collection-level weight (e.g.protected float
decodeNormValue(byte b)
protected byte
encodeNormValue(float boost, int fieldLength)
The default implementation encodesboost / sqrt(length)
withSmallFloat.floatToByte315(float)
.float
getB()
Returns theb
parameterboolean
getDiscountOverlaps()
Returns true if overlap tokens are discounted from the document's length.float
getK1()
Returns thek1
parameterprotected float
idf(long docFreq, long numDocs)
Implemented aslog(1 + (numDocs - docFreq + 0.5)/(docFreq + 0.5))
.Explanation
idfExplain(CollectionStatistics collectionStats, TermStatistics termStats)
Computes a score factor for a simple term and returns an explanation for that score factor.Explanation
idfExplain(CollectionStatistics collectionStats, TermStatistics[] termStats)
Computes a score factor for a phrase.protected float
scorePayload(int doc, int start, int end, BytesRef payload)
The default implementation returns1
void
setDiscountOverlaps(boolean v)
Sets whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm.Similarity.SimScorer
simScorer(Similarity.SimWeight stats, AtomicReaderContext context)
Creates a newSimilarity.SimScorer
to score matching documents from a segment of the inverted index.protected float
sloppyFreq(int distance)
Implemented as1 / (distance + 1)
.java.lang.String
toString()
-
Methods inherited from class org.apache.lucene.search.similarities.Similarity
coord, queryNorm
-
-
-
-
Constructor Detail
-
BM25Similarity
public BM25Similarity(float k1, float b)
BM25 with the supplied parameter values.- Parameters:
k1
- Controls non-linear term frequency normalization (saturation).b
- Controls to what degree document length normalizes tf values.
-
BM25Similarity
public BM25Similarity()
BM25 with these default values:k1 = 1.2
,b = 0.75
.
-
-
Method Detail
-
idf
protected float idf(long docFreq, long numDocs)
Implemented aslog(1 + (numDocs - docFreq + 0.5)/(docFreq + 0.5))
.
-
sloppyFreq
protected float sloppyFreq(int distance)
Implemented as1 / (distance + 1)
.
-
scorePayload
protected float scorePayload(int doc, int start, int end, BytesRef payload)
The default implementation returns1
-
avgFieldLength
protected float avgFieldLength(CollectionStatistics collectionStats)
The default implementation computes the average assumTotalTermFreq / maxDoc
, or returns1
if the index does not store sumTotalTermFreq (Lucene 3.x indexes or any field that omits frequency information).
-
encodeNormValue
protected byte encodeNormValue(float boost, int fieldLength)
The default implementation encodesboost / sqrt(length)
withSmallFloat.floatToByte315(float)
. This is compatible with Lucene's default implementation. If you change this, then you should changedecodeNormValue(byte)
to match.
-
decodeNormValue
protected float decodeNormValue(byte b)
-
setDiscountOverlaps
public void setDiscountOverlaps(boolean v)
Sets whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm. By default this is true, meaning overlap tokens do not count when computing norms.
-
getDiscountOverlaps
public boolean getDiscountOverlaps()
Returns true if overlap tokens are discounted from the document's length.- See Also:
setDiscountOverlaps(boolean)
-
computeNorm
public final long computeNorm(FieldInvertState state)
Description copied from class:Similarity
Computes the normalization value for a field, given the accumulated state of term processing for this field (seeFieldInvertState
).Matches in longer fields are less precise, so implementations of this method usually set smaller values when
state.getLength()
is large, and larger values whenstate.getLength()
is small.- Specified by:
computeNorm
in classSimilarity
- Parameters:
state
- current processing state for this field- Returns:
- computed norm value
-
idfExplain
public Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics termStats)
Computes a score factor for a simple term and returns an explanation for that score factor.The default implementation uses:
idf(docFreq, searcher.maxDoc());
Note thatCollectionStatistics.maxDoc()
is used instead ofIndexReader#numDocs()
because alsoTermStatistics.docFreq()
is used, and when the latter is inaccurate, so isCollectionStatistics.maxDoc()
, and in the same direction. In addition,CollectionStatistics.maxDoc()
is more efficient to compute- Parameters:
collectionStats
- collection-level statisticstermStats
- term-level statistics for the term- Returns:
- an Explain object that includes both an idf score factor and an explanation for the term.
-
idfExplain
public Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics[] termStats)
Computes a score factor for a phrase.The default implementation sums the idf factor for each term in the phrase.
- Parameters:
collectionStats
- collection-level statisticstermStats
- term-level statistics for the terms in the phrase- Returns:
- an Explain object that includes both an idf score factor for the phrase and an explanation for each term.
-
computeWeight
public final Similarity.SimWeight computeWeight(float queryBoost, CollectionStatistics collectionStats, TermStatistics... termStats)
Description copied from class:Similarity
Compute any collection-level weight (e.g. IDF, average document length, etc) needed for scoring a query.- Specified by:
computeWeight
in classSimilarity
- Parameters:
queryBoost
- the query-time boost.collectionStats
- collection-level statistics, such as the number of tokens in the collection.termStats
- term-level statistics, such as the document frequency of a term across the collection.- Returns:
- SimWeight object with the information this Similarity needs to score a query.
-
simScorer
public final Similarity.SimScorer simScorer(Similarity.SimWeight stats, AtomicReaderContext context) throws java.io.IOException
Description copied from class:Similarity
Creates a newSimilarity.SimScorer
to score matching documents from a segment of the inverted index.- Specified by:
simScorer
in classSimilarity
- Parameters:
stats
- collection information fromSimilarity.computeWeight(float, CollectionStatistics, TermStatistics...)
context
- segment of the inverted index to be scored.- Returns:
- SloppySimScorer for scoring documents across
context
- Throws:
java.io.IOException
- if there is a low-level I/O error
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
getK1
public float getK1()
Returns thek1
parameter- See Also:
BM25Similarity(float, float)
-
getB
public float getB()
Returns theb
parameter- See Also:
BM25Similarity(float, float)
-
-