Lucene.Net
3.0.3
Lucene.Net is a .NET port of the Java Lucene Indexing Library
|
Generate "more like this" similarity queries. Based on this mail: More...
Public Member Functions | |
MoreLikeThis (IndexReader ir) | |
Constructor requiring an IndexReader. | |
MoreLikeThis (IndexReader ir, Lucene.Net.Search.Similarity sim) | |
void | SetMaxDocFreqPct (int maxPercentage) |
Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored. | |
System.String[] | GetFieldNames () |
Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used is DEFAULT_FIELD_NAMES. | |
void | SetFieldNames (System.String[] fieldNames) |
Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor. | |
void | SetStopWords (ISet< string > stopWords) |
Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting". | |
ISet< string > | GetStopWords () |
Get the current stop words being used. | |
Query | Like (int docNum) |
Return a query that will return docs like the passed lucene document ID. | |
Query | Like (System.IO.FileInfo f) |
Return a query that will return docs like the passed file. | |
Query | Like (System.Uri u) |
Return a query that will return docs like the passed URL. | |
Query | Like (System.IO.Stream is_Renamed) |
Return a query that will return docs like the passed stream. | |
Query | Like (System.IO.TextReader r) |
Return a query that will return docs like the passed Reader. | |
System.String | DescribeParams () |
Describe the parameters that control how the "more like this" query is formed. | |
PriorityQueue< object[]> | RetrieveTerms (System.IO.TextReader r) |
Find words for a more-like-this query former. The result is a priority queue of arrays with one entry for every word in the document. Each array has 6 elements. The elements are: | |
System.String[] | RetrieveInterestingTerms (int docNum) |
System.String[] | RetrieveInterestingTerms (System.IO.TextReader r) |
Convenience routine to make it easy to return the most interesting words in a document. More advanced users will call RetrieveTerms(System.IO.TextReader) directly. | |
Static Public Member Functions | |
static void | Main (System.String[] a) |
Test driver. Pass in "-i INDEX" and then either "-fn FILE" or "-url URL". | |
Public Attributes | |
const int | DEFAULT_MAX_NUM_TOKENS_PARSED = 5000 |
Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support. | |
const int | DEFAULT_MIN_TERM_FREQ = 2 |
Ignore terms with less than this frequency in the source doc. | |
const int | DEFAULT_MIN_DOC_FREQ = 5 |
Ignore words which do not occur in at least this many docs. | |
const int | DEFAULT_MAX_DOC_FREQ = int.MaxValue |
Ignore words wich occur in more than this many docs | |
const bool | DEFAULT_BOOST = false |
Boost terms in query based on score. | |
const int | DEFAULT_MIN_WORD_LENGTH = 0 |
Ignore words less than this length or if 0 then this has no effect. | |
const int | DEFAULT_MAX_WORD_LENGTH = 0 |
Ignore words greater than this length or if 0 then this has no effect. | |
const int | DEFAULT_MAX_QUERY_TERMS = 25 |
Return a Query with no more than this many terms. | |
Static Public Attributes | |
static readonly Analyzer | DEFAULT_ANALYZER = new StandardAnalyzer(Util.Version.LUCENE_CURRENT) |
Default analyzer to parse source doc with. | |
static readonly System.String[] | DEFAULT_FIELD_NAMES = new System.String[] { "contents" } |
Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader. | |
static readonly ISet< string > | DEFAULT_STOP_WORDS = null |
Default set of stopwords. If null means to allow stop words. | |
Properties | |
float | BoostFactor [get, set] |
Gets or sets the boost factor used when boosting terms | |
Similarity | Similarity [get, set] |
Analyzer | Analyzer [get, set] |
Gets or sets the analyzer used to parse source doc with. The default analyzer is the DEFAULT_ANALYZER. | |
int | MinTermFreq [get, set] |
Gets or sets the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ. | |
int | MinDocFreq [get, set] |
Gets or sets the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ. | |
int | MaxDocFreq [get, set] |
Gets or sets the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency is DEFAULT_MAX_DOC_FREQ | |
bool | Boost [get, set] |
Gets or sets a boolean indicating whether to boost terms in query based on "score" or not. The default is DEFAULT_BOOST. | |
int | MinWordLen [get, set] |
Gets or sets the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH. | |
int | MaxWordLen [get, set] |
Gets or sets the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH. | |
int | MaxQueryTerms [get, set] |
Gets or sets the maximum number of query terms that will be included in any generated query. The default is DEFAULT_MAX_QUERY_TERMS. | |
int | MaxNumTokensParsed [get, set] |
Gets or sets the maximum number of tokens to parse in each example doc field that is not stored with TermVector support | |
Generate "more like this" similarity queries. Based on this mail:
Lucene does let you access the document frequency of terms, with IndexReader.DocFreq(). Term frequencies can be computed by re-tokenizing the text, which, for a single document, is usually fast enough. But looking up the DocFreq() of every term in the document is probably too slow.
You can use some heuristics to prune the set of terms, to avoid calling DocFreq() too much, or at all. Since you're trying to maximize a tf*idf score, you're probably most interested in terms with a high tf. Choosing a tf threshold even as low as two or three will radically reduce the number of terms under consideration. Another heuristic is that terms with a high idf (i.e., a low df) tend to be longer. So you could threshold the terms by the number of characters, not selecting anything less than, e.g., six or seven characters. With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms that do a pretty good job of characterizing a document.
It all depends on what you're trying to do. If you're trying to eek out that last percent of precision and recall regardless of computational difficulty so that you can win a TREC competition, then the techniques I mention above are useless. But if you're trying to provide a "more like this" button on a search results page that does a decent job and has good performance, such techniques might be useful.
An efficient, effective "more-like-this" query generator would be a great contribution, if anyone's interested. I'd imagine that it would take a Reader or a String (the document's text), analyzer Analyzer, and return a set of representative terms using heuristics like those above. The frequency and length thresholds could be parameters, etc.
Doug
This class has lots of options to try to make it efficient and flexible. See the body of Main below in the source for real code, or if you want pseudo code, the simpliest possible usage is as follows. The bold fragment is specific to this class.
IndexReader ir = ... IndexSearcher is = ... MoreLikeThis mlt = new MoreLikeThis(ir); Reader target = ... // orig source of doc you want to find similarities to Query query = mlt.Like( target); Hits hits = is.Search(query); // now the usual iteration thru 'hits' - the only thing to watch for is to make sure you ignore the doc if it matches your 'target' document, as it should be similar to itself
Thus you:
You may want to use SetFieldNames so you can examine multiple fields (e.g. body and title) for similarity.
Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated:
Changes: Mark Harwood 29/02/04 Some bugfixing, some refactoring, some optimisation.
Definition at line 138 of file MoreLikeThis.cs.
Lucene.Net.Search.Similar.MoreLikeThis.MoreLikeThis | ( | IndexReader | ir | ) |
Constructor requiring an IndexReader.
Definition at line 274 of file MoreLikeThis.cs.
Lucene.Net.Search.Similar.MoreLikeThis.MoreLikeThis | ( | IndexReader | ir, |
Lucene.Net.Search.Similarity | sim | ||
) |
Definition at line 278 of file MoreLikeThis.cs.
System.String Lucene.Net.Search.Similar.MoreLikeThis.DescribeParams | ( | ) |
Describe the parameters that control how the "more like this" query is formed.
Definition at line 613 of file MoreLikeThis.cs.
System.String [] Lucene.Net.Search.Similar.MoreLikeThis.GetFieldNames | ( | ) |
Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used is DEFAULT_FIELD_NAMES.
Definition at line 364 of file MoreLikeThis.cs.
ISet<string> Lucene.Net.Search.Similar.MoreLikeThis.GetStopWords | ( | ) |
Get the current stop words being used.
Definition at line 423 of file MoreLikeThis.cs.
Query Lucene.Net.Search.Similar.MoreLikeThis.Like | ( | int | docNum | ) |
Return a query that will return docs like the passed lucene document ID.
docNum | the documentID of the lucene doc to generate the 'More Like This" query for. |
Definition at line 453 of file MoreLikeThis.cs.
Query Lucene.Net.Search.Similar.MoreLikeThis.Like | ( | System.IO.FileInfo | f | ) |
Return a query that will return docs like the passed file.
Definition at line 470 of file MoreLikeThis.cs.
Query Lucene.Net.Search.Similar.MoreLikeThis.Like | ( | System.Uri | u | ) |
Return a query that will return docs like the passed URL.
Definition at line 487 of file MoreLikeThis.cs.
Query Lucene.Net.Search.Similar.MoreLikeThis.Like | ( | System.IO.Stream | is_Renamed | ) |
Return a query that will return docs like the passed stream.
Definition at line 497 of file MoreLikeThis.cs.
Query Lucene.Net.Search.Similar.MoreLikeThis.Like | ( | System.IO.TextReader | r | ) |
Return a query that will return docs like the passed Reader.
Definition at line 507 of file MoreLikeThis.cs.
|
static |
Test driver. Pass in "-i INDEX" and then either "-fn FILE" or "-url URL".
Definition at line 638 of file MoreLikeThis.cs.
System.String [] Lucene.Net.Search.Similar.MoreLikeThis.RetrieveInterestingTerms | ( | int | docNum | ) |
Definition at line 872 of file MoreLikeThis.cs.
System.String [] Lucene.Net.Search.Similar.MoreLikeThis.RetrieveInterestingTerms | ( | System.IO.TextReader | r | ) |
Convenience routine to make it easy to return the most interesting words in a document. More advanced users will call RetrieveTerms(System.IO.TextReader) directly.
r | the source document |
the most interesting words in the document
Definition at line 901 of file MoreLikeThis.cs.
PriorityQueue<object[]> Lucene.Net.Search.Similar.MoreLikeThis.RetrieveTerms | ( | System.IO.TextReader | r | ) |
Find words for a more-like-this query former. The result is a priority queue of arrays with one entry for every word in the document. Each array has 6 elements. The elements are:
This is a somewhat "advanced" routine, and in general only the 1st entry in the array is of interest. This method is exposed so that you can identify the "interesting words" in a document. For an easier method to call see RetrieveInterestingTerms(System.IO.TextReader).
r | the reader that has the content of the document |
the most intresting words in the document ordered by score, with the highest scoring, or best entry, first
Definition at line 860 of file MoreLikeThis.cs.
void Lucene.Net.Search.Similar.MoreLikeThis.SetFieldNames | ( | System.String[] | fieldNames | ) |
Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.
fieldNames | the field names that will be used when generating the 'More Like This' query. |
Definition at line 377 of file MoreLikeThis.cs.
void Lucene.Net.Search.Similar.MoreLikeThis.SetMaxDocFreqPct | ( | int | maxPercentage | ) |
Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored.
maxPercentage | the maximum percentage of documents (0-100) that a term may appear in to be still considered relevant |
Definition at line 344 of file MoreLikeThis.cs.
void Lucene.Net.Search.Similar.MoreLikeThis.SetStopWords | ( | ISet< string > | stopWords | ) |
Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".
stopWords | set of stopwords, if null it means to allow stop words |
Definition at line 415 of file MoreLikeThis.cs.
|
static |
Default analyzer to parse source doc with.
Definition at line 150 of file MoreLikeThis.cs.
const bool Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_BOOST = false |
Boost terms in query based on score.
Definition at line 178 of file MoreLikeThis.cs.
|
static |
Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader.
Definition at line 183 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_DOC_FREQ = int.MaxValue |
Ignore words wich occur in more than this many docs
Definition at line 171 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_NUM_TOKENS_PARSED = 5000 |
Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.
Definition at line 144 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_QUERY_TERMS = 25 |
Return a Query with no more than this many terms.
Definition at line 221 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MAX_WORD_LENGTH = 0 |
Ignore words greater than this length or if 0 then this has no effect.
Definition at line 197 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_DOC_FREQ = 5 |
Ignore words which do not occur in at least this many docs.
Definition at line 164 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_TERM_FREQ = 2 |
Ignore terms with less than this frequency in the source doc.
Definition at line 157 of file MoreLikeThis.cs.
const int Lucene.Net.Search.Similar.MoreLikeThis.DEFAULT_MIN_WORD_LENGTH = 0 |
Ignore words less than this length or if 0 then this has no effect.
Definition at line 190 of file MoreLikeThis.cs.
|
static |
Default set of stopwords. If null means to allow stop words.
Definition at line 207 of file MoreLikeThis.cs.
|
getset |
Gets or sets the analyzer used to parse source doc with. The default analyzer is the DEFAULT_ANALYZER.
An analyzer is not required for generating a query with the Like(int) method, all other 'like' methods require an analyzer.
the analyzer that will be used to parse source doc with.
Definition at line 300 of file MoreLikeThis.cs.
|
getset |
Gets or sets a boolean indicating whether to boost terms in query based on "score" or not. The default is DEFAULT_BOOST.
Definition at line 353 of file MoreLikeThis.cs.
|
getset |
Gets or sets the boost factor used when boosting terms
Definition at line 268 of file MoreLikeThis.cs.
|
getset |
Gets or sets the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency is DEFAULT_MAX_DOC_FREQ
Definition at line 331 of file MoreLikeThis.cs.
|
getset |
Gets or sets the maximum number of tokens to parse in each example doc field that is not stored with TermVector support
Definition at line 445 of file MoreLikeThis.cs.
|
getset |
Gets or sets the maximum number of query terms that will be included in any generated query. The default is DEFAULT_MAX_QUERY_TERMS.
Definition at line 434 of file MoreLikeThis.cs.
|
getset |
Gets or sets the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH.
Definition at line 397 of file MoreLikeThis.cs.
|
getset |
Gets or sets the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ.
Definition at line 320 of file MoreLikeThis.cs.
|
getset |
Gets or sets the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ.
Definition at line 310 of file MoreLikeThis.cs.
|
getset |
Gets or sets the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH.
Definition at line 387 of file MoreLikeThis.cs.
|
getset |
Definition at line 285 of file MoreLikeThis.cs.