IndexWriter creates and maintains an index.
/// The create argument to the
/// constructor
/// determines whether a new index is created, or whether an existing index is
/// opened. Note that you
/// can open an index with create=true even while readers are
/// using the index. The old readers will continue to search
/// the "point in time" snapshot they had opened, and won't
/// see the newly created index until they re-open. There are
/// also constructors
/// with no create argument which
/// will create a new index if there is not already an index at the
/// provided path and otherwise open the existing index.
In either case, documents are added with addDocument /// and removed with deleteDocuments(Term) /// or deleteDocuments(Query). /// A document can be updated with updateDocument /// (which just deletes and then adds the entire document). /// When finished adding, deleting and updating documents, close should be called.
/// /// ///These changes are buffered in memory and periodically /// flushed to the {@link Directory} (during the above method /// calls). A flush is triggered when there are enough /// buffered deletes (see {@link #setMaxBufferedDeleteTerms}) /// or enough added documents since the last flush, whichever /// is sooner. For the added documents, flushing is triggered /// either by RAM usage of the documents (see {@link /// #setRAMBufferSizeMB}) or the number of added documents. /// The default is to flush when RAM usage hits 16 MB. For /// best indexing speed you should flush by RAM usage with a /// large RAM buffer. Note that flushing just moves the /// internal buffered state in IndexWriter into the index, but /// these changes are not visible to IndexReader until either /// {@link #Commit()} or {@link #close} is called. A flush may /// also trigger one or more segment merges which by default /// run with a background thread so as not to block the /// addDocument calls (see below /// for changing the {@link MergeScheduler}).
/// /// ///The optional autoCommit argument to the constructors
/// controls visibility of the changes to {@link IndexReader}
/// instances reading the same index. When this is
/// false, changes are not visible until {@link
/// #Close()} or {@link #Commit()} is called. Note that changes will still be
/// flushed to the {@link org.apache.lucene.store.Directory}
/// as new files, but are not committed (no new
/// segments_N file is written referencing the
/// new files, nor are the files sync'd to stable storage)
/// until {@link #Close()} or {@link #Commit()} is called. If something
/// goes terribly wrong (for example the JVM crashes), then
/// the index will reflect none of the changes made since the
/// last commit, or the starting state if commit was not called.
/// You can also call {@link #rollback}, which closes the writer
/// without committing any changes, and removes any index
/// files that had been flushed but are now unreferenced.
/// This mode is useful for preventing readers from refreshing
/// at a bad time (for example after you've done all your
/// deletes but before you've done your adds). It can also be
/// used to implement simple single-writer transactional
/// semantics ("all or none"). You can do a two-phase commit
/// by calling {@link #PrepareCommit()}
/// followed by {@link #Commit()}. This is necessary when
/// Lucene is working with an external resource (for example,
/// a database) and both must either commit or rollback the
/// transaction.
When autoCommit is true then
/// the writer will periodically commit on its own. [Deprecated: Note that in 3.0, IndexWriter will
/// no longer accept autoCommit=true (it will be hardwired to
/// false). You can always call {@link #Commit()} yourself
/// when needed]. There is
/// no guarantee when exactly an auto commit will occur (it
/// used to be after every flush, but it is now after every
/// completed merge, as of 2.4). If you want to force a
/// commit, call {@link #Commit()}, or, close the writer. Once
/// a commit has finished, newly opened {@link IndexReader} instances will
/// see the changes to the index as of that commit. When
/// running in this mode, be careful not to refresh your
/// readers while optimize or segment merges are taking place
/// as this can tie up substantial disk space.
Regardless of autoCommit, an {@link
/// IndexReader} or {@link org.apache.lucene.search.IndexSearcher} will only see the
/// index as of the "point in time" that it was opened. Any
/// changes committed to the index after the reader was opened
/// are not visible until the reader is re-opened.
If an index will not have more documents added for a while and optimal search /// performance is desired, then either the full optimize /// method or partial {@link #Optimize(int)} method should be /// called before the index is closed.
/// ///Opening an IndexWriter creates a lock file for the directory in use. Trying to open
/// another IndexWriter on the same directory will lead to a
/// {@link LockObtainFailedException}. The {@link LockObtainFailedException}
/// is also thrown if an IndexReader on the same directory is used to delete documents
/// from the index.
Expert: IndexWriter allows an optional
/// {@link IndexDeletionPolicy} implementation to be
/// specified. You can use this to control when prior commits
/// are deleted from the index. The default policy is {@link
/// KeepOnlyLastCommitDeletionPolicy} which removes all prior
/// commits as soon as a new commit is done (this matches
/// behavior before 2.2). Creating your own policy can allow
/// you to explicitly keep previous "point in time" commits
/// alive in the index for some time, to allow readers to
/// refresh to the new commit without having the old commit
/// deleted out from under them. This is necessary on
/// filesystems like NFS that do not support "delete on last
/// close" semantics, which Lucene's "point in time" search
/// normally relies on.
Expert:
/// IndexWriter allows you to separately change
/// the {@link MergePolicy} and the {@link MergeScheduler}.
/// The {@link MergePolicy} is invoked whenever there are
/// changes to the segments in the index. Its role is to
/// select which merges to do, if any, and return a {@link
/// MergePolicy.MergeSpecification} describing the merges. It
/// also selects merges to do for Optimize(). (The default is
/// {@link LogByteSizeMergePolicy}. Then, the {@link
/// MergeScheduler} is invoked with the requested merges and
/// it decides when and how to run the merges. The default is
/// {@link ConcurrentMergeScheduler}.
Get the current setting of whether newly flushed /// segments will use the compound file format. Note that /// this just returns the value previously set with /// setUseCompoundFile(bool), or the default value /// (true). You cannot use this to query the status of /// previously flushed segments.
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.GetUseCompoundFile as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
/// ///Setting to turn on usage of a compound file. When on, /// multiple files for each segment are merged into a /// single file when a new segment is flushed.
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.SetUseCompoundFile as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
///This defaults to the current value of {@link Similarity#GetDefault()}. ///
numUniqueTerms/interval terms are read into
/// memory by an IndexReader, and, on average, interval/2 terms
/// must be scanned for each random term access.
///
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// path, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
/// Maximum field Length: LIMITED, UNLIMITED, or user-specified
public IndexWriter(string path, Analyzer a, bool create, MaxFieldLength mfl)
{
InitBlock();
Init(FSDirectory.GetDirectory(path), a, create, true, null, false, mfl.GetLimit());
}
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// path, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
[System.Obsolete("This constructor will be removed in the 3.0 release. Use IndexWriter(string, Analyzer, bool, MaxFieldLength) instead, and call Commit() when needed")]
public IndexWriter(string path, Analyzer a, bool create)
{
InitBlock();
Init(FSDirectory.GetDirectory(path), a, create, true, null, true, DEFAULT_MAX_FIELD_LENGTH);
}
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// path, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
/// Maximum field Length: LIMITED, UNLIMITED, or user-specified
public IndexWriter(System.IO.FileInfo path, Analyzer a, bool create, MaxFieldLength mfl)
{
InitBlock();
Init(FSDirectory.GetDirectory(path), a, create, true, null, false, mfl.GetLimit());
}
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// path, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
[System.Obsolete("This constructor will be removed in the 3.0 release. Use IndexWriter(FileInfo, Analyzer, bool, MaxFieldLength) instead, and call Commit() when needed")]
public IndexWriter(System.IO.FileInfo path, Analyzer a, bool create)
{
InitBlock();
Init(FSDirectory.GetDirectory(path), a, create, true, null, true, DEFAULT_MAX_FIELD_LENGTH);
}
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// d, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
/// Maximum field Length: LIMITED, UNLIMITED, or user-specified
public IndexWriter(Directory d, Analyzer a, bool create, MaxFieldLength mfl)
{
InitBlock();
Init(d, a, create, false, null, false, mfl.GetLimit());
}
/// path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// d, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
[System.Obsolete("This constructor will be removed in the 3.0 release. Use IndexWriter(Directory, Analyzer, bool, MaxFieldLength) instead, and call Commit() when needed")]
public IndexWriter(Directory d, Analyzer a, bool create)
{
InitBlock();
Init(d, a, create, false, null, true, DEFAULT_MAX_FIELD_LENGTH);
}
/// path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a.
/// write.lock could not be obtained)path.
/// Text will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// d, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
[System.Obsolete("This constructor will be removed in the 3.0 release. Use IndexWriter(Directory, Analyzer, bool, MaxFieldLength) instead, and call Commit() when needed")]
public IndexWriter(Directory d, bool autoCommit, Analyzer a, bool create)
{
InitBlock();
Init(d, a, create, false, null, autoCommit, DEFAULT_MAX_FIELD_LENGTH);
}
/// d,
/// first creating it if it does not already exist. Text
/// will be analyzed with a.
/// write.lock could not be obtained)d,
/// first creating it if it does not already exist. Text
/// will be analyzed with a.
/// write.lock could not be obtained)d,
/// first creating it if it does not already exist. Text
/// will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// d, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
/// see deletionPolicy above
/// Maximum field Length: LIMITED, UNLIMITED, or user-specified
public IndexWriter(Directory d, Analyzer a, bool create, IndexDeletionPolicy deletionPolicy, MaxFieldLength mfl)
{
InitBlock();
Init(d, a, create, false, deletionPolicy, false, mfl.GetLimit());
}
/// d,
/// first creating it if it does not already exist. Text
/// will be analyzed with a. If create
/// is true, then a new, empty index will be created in
/// d, replacing the index already there, if any.
/// write.lock could not be obtained)create is false or if there is any other low-level IO errortrue to create the index or overwrite the existing one; false to append to the existing index
/// see deletionPolicy above
/// Maximum field Length: LIMITED, UNLIMITED, or user-specified
[System.Obsolete("This constructor will be removed in the 3.0 release. Use IndexWriter(Directory, Analyzer, bool, IndexDeletionPolicy, MaxFieldLength) instead, and call Commit() when needed")]
public IndexWriter(Directory d, bool autoCommit, Analyzer a, bool create, IndexDeletionPolicy deletionPolicy)
{
InitBlock();
Init(d, a, create, false, deletionPolicy, autoCommit, DEFAULT_MAX_FIELD_LENGTH);
}
private void Init(Directory d, Analyzer a, bool closeDir, IndexDeletionPolicy deletionPolicy, bool autoCommit, int maxFieldLength)
{
if (IndexReader.IndexExists(d))
{
Init(d, a, false, closeDir, deletionPolicy, autoCommit, maxFieldLength);
}
else
{
Init(d, a, true, closeDir, deletionPolicy, autoCommit, maxFieldLength);
}
}
private void Init(Directory d, Analyzer a, bool create, bool closeDir, IndexDeletionPolicy deletionPolicy, bool autoCommit, int maxFieldLength)
{
this.closeDir = closeDir;
directory = d;
analyzer = a;
SetMessageID(defaultInfoStream);
this.maxFieldLength = maxFieldLength;
if (create)
{
// Clear the write lock in case it's leftover:
directory.ClearLock(WRITE_LOCK_NAME);
}
Lock writeLock = directory.MakeLock(WRITE_LOCK_NAME);
if (!writeLock.Obtain(writeLockTimeout))
// obtain write lock
{
throw new LockObtainFailedException("Index locked for write: " + writeLock);
}
this.writeLock = writeLock; // save it
try
{
if (create)
{
// Try to read first. This is to allow create
// against an index that's currently open for
// searching. In this case we write the next
// segments_N file with no segments:
try
{
segmentInfos.Read(directory);
segmentInfos.Clear();
}
catch (System.IO.IOException)
{
// Likely this means it's a fresh directory
}
segmentInfos.Commit(directory);
}
else
{
segmentInfos.Read(directory);
// We assume that this segments_N was previously
// properly sync'd:
for (int i = 0; i < segmentInfos.Count; i++)
{
SegmentInfo info = segmentInfos.Info(i);
ListDetermines the largest segment (measured by /// document count) that may be merged with other segments. /// Small values (e.g., less than 10,000) are best for /// interactive indexing, as this limits the length of /// pauses while indexing to a few seconds. Larger values /// are best for batched indexing and speedier /// searches.
/// ///The default value is {@link Integer#MAX_VALUE}.
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.SetMaxMergeDocs as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
/// ///The default merge policy ({@link /// LogByteSizeMergePolicy}) also allows you to set this /// limit by net size (in MB) of the segment, using {@link /// LogByteSizeMergePolicy#setMaxMergeMB}.
///Returns the largest segment (measured by document /// count) that may be merged with other segments.
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.GetMaxMergeDocs as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
/// ///When this is set, the writer will flush every /// maxBufferedDocs added documents. Pass in {@link /// #DISABLE_AUTO_FLUSH} to prevent triggering a flush due /// to number of buffered documents. Note that if flushing /// by RAM usage is also enabled, then the flush will be /// triggered by whichever comes first.
/// ///Disabled by default (writer flushes by RAM usage).
/// ///When this is set, the writer will flush whenever /// buffered documents use this much RAM. Pass in {@link /// #DISABLE_AUTO_FLUSH} to prevent triggering a flush due /// to RAM usage. Note that if flushing by document count /// is also enabled, then the flush will be triggered by /// whichever comes first.
/// ///The default value is {@link #DEFAULT_RAM_BUFFER_SIZE_MB}.
/// ///Determines the minimal number of delete terms required before the buffered /// in-memory delete terms are applied and flushed. If there are documents /// buffered in memory at the time, they are merged and a new segment is /// created.
///Disabled by default (writer flushes by RAM usage).
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.SetMergeFactor as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
/// ///This must never be less than 2. The default value is 10. ///
Returns the number of segments that are merged at /// once and also controls the total number of segments /// allowed to accumulate in the index.
/// ///Note that this method is a convenience method: it /// just calls mergePolicy.GetMergeFactor as long as /// mergePolicy is an instance of {@link LogMergePolicy}. /// Otherwise an System.ArgumentException is thrown.
/// ///If an Exception is hit during close, eg due to disk /// full or some other reason, then both the on-disk index /// and the internal state of the IndexWriter instance will /// be consistent. However, the close will not be complete /// even though part of it (flushing buffered documents) /// may have succeeded, so the write lock will still be /// held.
/// ///If you can correct the underlying cause (eg free up /// some disk space) then you can call Close() again. /// Failing that, if you want to force the write lock to be /// released (dangerous, because you may then lose buffered /// docs in the IndexWriter instance) then you can do /// something like this:
/// ///
/// try
/// {
/// writer.Close();
/// }
/// finally
/// {
/// if (IndexWriter.isLocked(directory))
/// {
/// IndexWriter.unlock(directory);
/// }
/// }
///
///
/// after which, you must be certain not to use the writer
/// instance anymore.
/// Note that if an Exception is hit (for example disk full) /// then the index will be consistent, but this document /// may not have been added. Furthermore, it's possible /// the index will have one segment in non-compound format /// even when using compound files (when a merge has /// partially succeeded).
/// ///This method periodically flushes pending documents /// to the Directory (see Flush(), above), /// and also periodically triggers segment merges in the index /// according to the MergePolicy in use. /// ///
Merges temporarily consume space in the directory. The amout of space /// required is up to 1X the size of all segments being merged, when no /// readers/searchers are open against the index, and up to 2X the size of /// all segments being merged when readers/searchers are open against the /// index (see Optimize() for details). The sequence of primitive merge /// operations performed is governed by the merge policy. /// ///
Note that each term in the document can be no longer /// than 16383 characters, otherwise an /// ArgumentException will be thrown.
/// ///See {@link #AddDocument(Document)} for details on /// index and IndexWriter state after an Exception, and /// flushing/merging temporary free space requirements.
/// ///term.term and then adding the new
/// document. The delete and then add are atomic as seen
/// by a reader on the same index (flush may happen only after
/// the add).
/// term and then adding the new
/// document. The delete and then add are atomic as seen
/// by a reader on the same index (flush may happen only after
/// the add).
/// It is recommended that this method be called upon completion of indexing. In /// environments with frequent updates, optimize is best done during low volume times, if at all. /// ///
///See http://www.gossamer-threads.com/lists/lucene/java-dev/47895 for more discussion.
/// ///Note that this can require substantial temporary free /// space in the Directory (see LUCENE-764 /// for details):
/// ///If no readers/searchers are open against the index, /// then free space required is up to 1X the total size of /// the starting index. For example, if the starting /// index is 10 GB, then you must have up to 10 GB of free /// space before calling optimize.
/// ///If readers/searchers are using the index, then free /// space required is up to 2X the size of the starting /// index. This is because in addition to the 1X used by /// optimize, the original 1X of the starting index is /// still consuming space in the Directory as the readers /// are holding the segments files open. Even on Unix, /// where it will appear as if the files are gone ("ls" /// won't list them), they still consume storage due to /// "delete on last close" semantics.
/// ///Furthermore, if some but not all readers re-open /// while the optimize is underway, this will cause > 2X /// temporary space to be consumed as those new readers /// will then hold open the partially optimized segments at /// that time. It is best not to re-open readers while /// optimize is running.
/// ///The actual temporary usage could be much less than /// these figures (it depends on many factors).
/// ///In general, once the optimize completes, the total size of the /// index will be less than the size of the starting index. /// It could be quite a bit smaller (if there were many /// pending deletes) or just slightly smaller.
/// ///If an Exception is hit during Optimize(), for example /// due to disk full, the index will not be corrupt and no /// documents will have been lost. However, it may have /// been partially optimized (some segments were merged but /// not all), and it's possible that one of the segments in /// the index will be in non-compound format even when /// using compound file format. This will occur when the /// Exception is hit during conversion of the segment into /// compound format.
/// ///This call will optimize those segments present in /// the index when the call started. If other threads are /// still adding documents and flushing segments, those /// newly created segments will not be optimized unless you /// call optimize again.
/// ///IndexWriter without committing
/// any of the changes that have occurred since it was
/// opened. This removes any temporary files that had been
/// created, after which the state of the index will be the
/// same as it was when this writer was first opened. This
/// can only be called when this IndexWriter was opened
/// with autoCommit=false. This also clears a previous
/// call to PrepareCommit().
/// autoCommit=true.
/// This may be used to parallelize batch indexing. A large document * collection can be broken into sub-collections. Each sub-collection can be * indexed in parallel, on a different thread, process or machine. The * complete index can then be created by merging sub-collection indexes * with this method. * *
NOTE: the index in each Directory must not be * changed (opened by a writer) while this method is * running. This method does not acquire a write lock in * each input Directory, so it is up to the caller to * enforce this. * *
NOTE: while this is running, any attempts to * add or delete documents (with another thread) will be * paused until this method completes. * *
This method is transactional in how Exceptions are * handled: it does not commit a new segments_N file until * all indexes are added. This means if an Exception * occurs (for example disk full), then either no indexes * will have been added or they all will have been.
* *Note that this requires temporary free space in the * Directory up to 2X the sum of all input indexes * (including the starting index). If readers/searchers * are open against the starting index, then temporary * free space required will be higher by the size of the * starting index (see {@link #Optimize()} for details). *
* *Once this completes, the final size of the index * will be less than the sum of all input index sizes * (including the starting index). It could be quite a * bit smaller (if there were many pending deletes) or * just slightly smaller.
* ** This requires this index not be among those to be added. * * @throws CorruptIndexException if the index is corrupt * @throws System.IO.IOException if there is a low-level IO error */ public void AddIndexesNoOptimize(Directory[] dirs) { EnsureOpen(); NoDupDirs(dirs); // Do not allow add docs or deletes while we are running: docWriter.PauseAllThreads(); try { if (infoStream != null) Message("flush at addIndexesNoOptimize"); Flush(true, false, true); bool success = false; StartTransaction(false); try { int docCount = 0; lock (this) { EnsureOpen(); for (int i = 0; i < dirs.Length; i++) { if (directory == dirs[i]) { // cannot add this index: segments may be deleted in merge before added throw new System.ArgumentException("Cannot add this index to itself"); } SegmentInfos sis = new SegmentInfos(); // read infos from dir sis.Read(dirs[i]); for (int j = 0; j < sis.Count; j++) { SegmentInfo info = sis.Info(j); System.Diagnostics.Debug.Assert(!segmentInfos.Contains(info), "dup info dir=" + info.dir + " name=" + info.name); docCount += info.docCount; segmentInfos.Add(info); // add each info } } } // Notify DocumentsWriter that the flushed count just increased docWriter.UpdateFlushedDocCount(docCount); MaybeMerge(); EnsureOpen(); // If after merging there remain segments in the index // that are in a different directory, just copy these // over into our index. This is necessary (before // finishing the transaction) to avoid leaving the // index in an unusable (inconsistent) state. ResolveExternalSegments(); EnsureOpen(); success = true; } finally { if (success) { CommitTransaction(); } else { RollbackTransaction(); } } } catch (System.OutOfMemoryException oom) { hitOOM = true; throw oom; } finally { docWriter.ResumeAllThreads(); } } private bool HasExternalSegments() { return HasExternalSegments(segmentInfos); } private bool HasExternalSegments(SegmentInfos infos) { int numSegments = infos.Count; for (int i = 0; i < numSegments; i++) if (infos.Info(i).dir != directory) return true; return false; } /* If any of our segments are using a directory != ours * then we have to either copy them over one by one, merge * them (if merge policy has chosen to) or wait until * currently running merges (in the background) complete. * We don't return until the SegmentInfos has no more * external segments. Currently this is only used by * AddIndexesNoOptimize(). */ private void ResolveExternalSegments() { bool any = false; bool done = false; while (!done) { SegmentInfo info = null; MergePolicy.OneMerge merge = null; lock (this) { if (stopMerges) throw new MergePolicy.MergeAbortedException("rollback() was called or AddIndexes* hit an unhandled exception"); int numSegments = segmentInfos.Count; done = true; for (int i = 0; i < numSegments; i++) { info = segmentInfos.Info(i); if (info.dir != directory) { done = false; MergePolicy.OneMerge newMerge = new MergePolicy.OneMerge(segmentInfos.Range(i, 1 + i), info.GetUseCompoundFile()); // Returns true if no running merge conflicts // with this one (and, records this merge as // pending), ie, this segment is not currently // being merged: if (RegisterMerge(newMerge)) { merge = newMerge; // If this segment is not currently being // merged, then advance it to running & run // the merge ourself (below): pendingMerges.Remove(merge); runningMerges[merge] = merge; break; } } } if (!done && merge == null) // We are not yet done (external segments still // exist in segmentInfos), yet, all such segments // are currently "covered" by a pending or running // merge. We now try to grab any pending merge // that involves external segments: merge = GetNextExternalMerge(); if (!done && merge == null) // We are not yet done, and, all external segments // fall under merges that the merge scheduler is // currently running. So, we now wait and check // back to see if the merge has completed. DoWait(); } if (merge != null) { any = true; Merge(merge); } } if (any) // Sometimes, on copying an external segment over, // more merges may become necessary: mergeScheduler.Merge(this); } /** Merges the provided indexes into this index. *
After this completes, the index is optimized.
*The provided IndexReaders are not closed.
*NOTE: the index in each Directory must not be * changed (opened by a writer) while this method is * running. This method does not acquire a write lock in * each input Directory, so it is up to the caller to * enforce this. * *
NOTE: while this is running, any attempts to * add or delete documents (with another thread) will be * paused until this method completes. * *
See {@link #AddIndexesNoOptimize(Directory[])} for * details on transactional semantics, temporary free * space required in the Directory, and non-CFS segments * on an Exception.
* @throws CorruptIndexException if the index is corrupt * @throws System.IO.IOException if there is a low-level IO error */ public void AddIndexes(IndexReader[] readers) { EnsureOpen(); // Do not allow add docs or deletes while we are running: docWriter.PauseAllThreads(); // We must pre-acquire the write lock here (and not in // StartTransaction below) so that no other AddIndexes // is allowed to start up after we have flushed & // optimized but before we then start our transaction. // This is because the merging below requires that only // one segment is present in the index: AcquireWrite(); try { bool success = false; SegmentInfo info = null; string mergedName = null; SegmentMerger merger = null; try { Flush(true, false, true); Optimize(); // start with zero or 1 seg success = true; } finally { // Take care to release the write lock if we hit an // exception before starting the transaction if (!success) ReleaseWrite(); } success = false; // true means we already have write lock; if this call // hits an exception it will release the write lock: StartTransaction(true); try { mergedName = NewSegmentName(); merger = new SegmentMerger(this, mergedName, null); IndexReader sReader = null; lock (this) { if (segmentInfos.Count == 1) { // add existing index, if any sReader = SegmentReader.Get(true, segmentInfos.Info(0)); } } try { if (sReader != null) merger.Add(sReader); for (int i = 0; i < readers.Length; i++) // add new indexes merger.Add(readers[i]); int docCount = merger.Merge(); // merge 'em if (sReader != null) { sReader.Close(); sReader = null; } lock (this) { segmentInfos.Clear(); // pop old infos & add new info = new SegmentInfo(mergedName, docCount, directory, false, true, -1, null, false, merger.HasProx()); segmentInfos.Add(info); } // Notify DocumentsWriter that the flushed count just increased docWriter.UpdateFlushedDocCount(docCount); success = true; } finally { if (sReader != null) { sReader.Close(); } } } finally { if (!success) { if (infoStream != null) Message("hit exception in AddIndexes during merge"); RollbackTransaction(); } else { CommitTransaction(); } } if (mergePolicy is LogMergePolicy && GetUseCompoundFile()) { ListNote: while this will force buffered docs to be * pushed into the index, it will not make these docs * visible to a reader. Use {@link #Commit()} instead * @throws CorruptIndexException if the index is corrupt * @throws System.IO.IOException if there is a low-level IO error * @deprecated please call {@link #Commit()}) instead */ public void Flush() { Flush(true, false, true); } /**
Expert: prepare for commit. This does the first * phase of 2-phase commit. You can only call this when * autoCommit is false. This method does all steps * necessary to commit changes since this writer was * opened: flushes pending added and deleted docs, syncs * the index files, writes most of next segments_N file. * After calling this you must call either {@link * #Commit()} to finish the commit, or {@link * #rollback()} to revert the commit and undo all changes * done since the writer was opened.
* * You can also just call {@link #Commit()} directly * without prepareCommit first in which case that method * will internally call prepareCommit. */ public void PrepareCommit() { EnsureOpen(); PrepareCommit(false); } private void PrepareCommit(bool internal_Renamed) { if (hitOOM) throw new System.Exception("this writer hit an System.OutOfMemoryException; cannot commit"); if (autoCommit && !internal_Renamed) throw new System.Exception("this method can only be used when autoCommit is false"); if (!autoCommit && pendingCommit != null) throw new System.Exception("prepareCommit was already called with no corresponding call to commit"); Message("prepareCommit: flush"); Flush(true, true, true); StartCommit(0); } private void Commit(long sizeInBytes) { StartCommit(sizeInBytes); FinishCommit(); } private bool committing; private void WaitForCommit() { lock (this) { // Only allow a single thread to do the commit, at a time: while (committing) DoWait(); committing = true; } } private void DoneCommit() { lock (this) { committing = false; System.Threading.Monitor.PulseAll(this); } } /** *Commits all pending updates (added & deleted * documents) to the index, and syncs all referenced index * files, such that a reader will see the changes and the * index updates will survive an OS or machine crash or * power loss. Note that this does not wait for any * running background merges to finish. This may be a * costly operation, so you should test the cost in your * application and do it only when really necessary.
* *Note that this operation calls Directory.sync on * the index files. That call should not return until the * file contents & metadata are on stable storage. For * FSDirectory, this calls the OS's fsync. But, beware: * some hardware devices may in fact cache writes even * during fsync, and return before the bits are actually * on stable storage, to give the appearance of faster * performance. If you have such a device, and it does * not have a battery backup (for example) then on power * loss it may still lose data. Lucene cannot guarantee * consistency on such devices.
* * @see #prepareCommit */ public void Commit() { EnsureOpen(); // Only let one thread do the prepare/finish at a time WaitForCommit(); try { Message("commit: start"); if (autoCommit || pendingCommit == null) { Message("commit: now prepare"); PrepareCommit(true); } else Message("commit: already prepared"); FinishCommit(); } finally { DoneCommit(); } } private void FinishCommit() { lock (this) { if (pendingCommit != null) { try { Message("commit: pendingCommit != null"); pendingCommit.FinishCommit(directory); lastCommitChangeCount = pendingCommitChangeCount; segmentInfos.UpdateGeneration(pendingCommit); SetRollbackSegmentInfos(pendingCommit); deleter.Checkpoint(pendingCommit, true); } finally { deleter.DecRef(pendingCommit); pendingCommit = null; System.Threading.Monitor.PulseAll(this); } } else Message("commit: pendingCommit == null; skip"); Message("commit: done"); } } /** * Flush all in-memory buffered udpates (adds and deletes) * to the Directory. * @param triggerMerge if true, we may merge segments (if * deletes or docs were flushed) if necessary * @param flushDocStores if false we are allowed to keep * doc stores open to share with the next segment * @param flushDeletes whether pending deletes should also * be flushed */ public /* for nunit tests, was: protected */ void Flush(bool triggerMerge, bool flushDocStores, bool flushDeletes) { // We can be called during close, when closing==true, so we must pass false to EnsureOpen: EnsureOpen(false); if (DoFlush(flushDocStores, flushDeletes) && triggerMerge) MaybeMerge(); } // TODO: this method should not have to be entirely // synchronized, ie, merges should be allowed to commit // even while a flush is happening private bool DoFlush(bool flushDocStores, bool flushDeletes) { lock (this) { EnsureOpen(false); System.Diagnostics.Debug.Assert(TestPoint("startDoFlush")); flushCount++; // Make sure no threads are actively adding a document flushDeletes |= docWriter.DeletesFull(); // When autoCommit=true we must always flush deletes // when flushing a segment; otherwise deletes may become // visible before their corresponding added document // from an updateDocument call flushDeletes |= autoCommit; // Returns true if docWriter is currently aborting, in // which case we skip flushing this segment if (docWriter.PauseAllThreads()) { docWriter.ResumeAllThreads(); return false; } try { SegmentInfo newSegment = null; int numDocs = docWriter.GetNumDocsInRAM(); // Always flush docs if there are any bool flushDocs = numDocs > 0; // With autoCommit=true we always must flush the doc // stores when we flush flushDocStores |= autoCommit; string docStoreSegment = docWriter.GetDocStoreSegment(); if (docStoreSegment == null) flushDocStores = false; int docStoreOffset = docWriter.GetDocStoreOffset(); // docStoreOffset should only be non-zero when // autoCommit == false System.Diagnostics.Debug.Assert(!autoCommit || 0 == docStoreOffset); bool docStoreIsCompoundFile = false; if (infoStream != null) { Message(" flush: segment=" + docWriter.GetSegment() + " docStoreSegment=" + docWriter.GetDocStoreSegment() + " docStoreOffset=" + docStoreOffset + " flushDocs=" + flushDocs + " flushDeletes=" + flushDeletes + " flushDocStores=" + flushDocStores + " numDocs=" + numDocs + " numBufDelTerms=" + docWriter.GetNumBufferedDeleteTerms()); Message(" index before flush " + SegString()); } // Check if the doc stores must be separately flushed // because other segments, besides the one we are about // to flush, reference it if (flushDocStores && (!flushDocs || !docWriter.GetSegment().Equals(docWriter.GetDocStoreSegment()))) { // We must separately flush the doc store if (infoStream != null) Message(" flush shared docStore segment " + docStoreSegment); docStoreIsCompoundFile = FlushDocStores(); flushDocStores = false; } string segment = docWriter.GetSegment(); // If we are flushing docs, segment must not be null: System.Diagnostics.Debug.Assert(segment != null || !flushDocs); if (flushDocs) { bool success = false; int flushedDocCount; try { flushedDocCount = docWriter.Flush(flushDocStores); success = true; } finally { if (!success) { if (infoStream != null) Message("hit exception flushing segment " + segment); deleter.Refresh(segment); } } if (0 == docStoreOffset && flushDocStores) { // This means we are flushing private doc stores // with this segment, so it will not be shared // with other segments System.Diagnostics.Debug.Assert(docStoreSegment != null); System.Diagnostics.Debug.Assert(docStoreSegment.Equals(segment)); docStoreOffset = -1; docStoreIsCompoundFile = false; docStoreSegment = null; } // Create new SegmentInfo, but do not add to our // segmentInfos until deletes are flushed // successfully. newSegment = new SegmentInfo(segment, flushedDocCount, directory, false, true, docStoreOffset, docStoreSegment, docStoreIsCompoundFile, docWriter.HasProx()); } docWriter.PushDeletes(); if (flushDocs) segmentInfos.Add(newSegment); if (flushDeletes) { flushDeletesCount++; ApplyDeletes(); } DoAfterFlush(); if (flushDocs) Checkpoint(); if (flushDocs && mergePolicy.UseCompoundFile(segmentInfos, newSegment)) { // Now build compound file bool success = false; try { docWriter.CreateCompoundFile(segment); success = true; } finally { if (!success) { if (infoStream != null) Message("hit exception creating compound file for newly flushed segment " + segment); deleter.DeleteFile(segment + "." + IndexFileNames.COMPOUND_FILE_EXTENSION); } } newSegment.SetUseCompoundFile(true); Checkpoint(); } return flushDocs; } catch (System.OutOfMemoryException oom) { hitOOM = true; throw oom; } finally { docWriter.ClearFlushPending(); docWriter.ResumeAllThreads(); } } } /** Expert: Return the total size of all index files currently cached in memory. * Useful for size management with flushRamDocs() */ public long RamSizeInBytes() { EnsureOpen(); return docWriter.GetRAMUsed(); } /** Expert: Return the number of documents currently * buffered in RAM. */ public int NumRamDocs() { lock (this) { EnsureOpen(); return docWriter.GetNumDocsInRAM(); } } private int EnsureContiguousMerge(MergePolicy.OneMerge merge) { int first = segmentInfos.IndexOf(merge.segments.Info(0)); if (first == -1) throw new MergePolicy.MergeException("could not find segment " + merge.segments.Info(0).name + " in current segments", directory); int numSegments = segmentInfos.Count; int numSegmentsToMerge = merge.segments.Count; for (int i = 0; i < numSegmentsToMerge; i++) { SegmentInfo info = merge.segments.Info(i); if (first + i >= numSegments || !segmentInfos.Info(first + i).Equals(info)) { if (segmentInfos.IndexOf(info) == -1) throw new MergePolicy.MergeException("MergePolicy selected a segment (" + info.name + ") that is not in the index", directory); else throw new MergePolicy.MergeException("MergePolicy selected non-contiguous segments to merge (" + merge.SegString(directory) + " vs " + SegString() + "), which IndexWriter (currently) cannot handle", directory); } } return first; } /** Carefully merges deletes for the segments we just * merged. This is tricky because, although merging will * clear all deletes (compacts the documents), new * deletes may have been flushed to the segments since * the merge was started. This method "carries over" * such new deletes onto the newly merged segment, and * saves the resulting deletes file (incrementing the * delete generation for merge.info). If no deletes were * flushed, no new deletes file is saved. */ private void CommitMergedDeletes(MergePolicy.OneMerge merge) { lock (this) { System.Diagnostics.Debug.Assert(TestPoint("startCommitMergeDeletes")); SegmentInfos sourceSegmentsClone = merge.segmentsClone; SegmentInfos sourceSegments = merge.segments; if (infoStream != null) Message("commitMergeDeletes " + merge.SegString(directory)); // Carefully merge deletes that occurred after we // started merging: BitVector deletes = null; int docUpto = 0; int delCount = 0; int numSegmentsToMerge = sourceSegments.Count; for (int i = 0; i < numSegmentsToMerge; i++) { SegmentInfo previousInfo = sourceSegmentsClone.Info(i); SegmentInfo currentInfo = sourceSegments.Info(i); System.Diagnostics.Debug.Assert(currentInfo.docCount == previousInfo.docCount); int docCount = currentInfo.docCount; if (previousInfo.HasDeletions()) { // There were deletes on this segment when the merge // started. The merge has collapsed away those // deletes, but, if new deletes were flushed since // the merge started, we must now carefully keep any // newly flushed deletes but mapping them to the new // docIDs. System.Diagnostics.Debug.Assert(currentInfo.HasDeletions()); // Load deletes present @ start of merge, for this segment: BitVector previousDeletes = new BitVector(previousInfo.dir, previousInfo.GetDelFileName()); if (!currentInfo.GetDelFileName().Equals(previousInfo.GetDelFileName())) { // This means this segment has had new deletes // committed since we started the merge, so we // must merge them: if (deletes == null) deletes = new BitVector(merge.info.docCount); BitVector currentDeletes = new BitVector(currentInfo.dir, currentInfo.GetDelFileName()); for (int j = 0; j < docCount; j++) { if (previousDeletes.Get(j)) System.Diagnostics.Debug.Assert(currentDeletes.Get(j)); else { if (currentDeletes.Get(j)) { deletes.Set(docUpto); delCount++; } docUpto++; } } } else docUpto += docCount - previousDeletes.Count(); } else if (currentInfo.HasDeletions()) { // This segment had no deletes before but now it // does: if (deletes == null) deletes = new BitVector(merge.info.docCount); BitVector currentDeletes = new BitVector(directory, currentInfo.GetDelFileName()); for (int j = 0; j < docCount; j++) { if (currentDeletes.Get(j)) { deletes.Set(docUpto); delCount++; } docUpto++; } } else // No deletes before or after docUpto += currentInfo.docCount; } if (deletes != null) { merge.info.AdvanceDelGen(); Message("commit merge deletes to " + merge.info.GetDelFileName()); deletes.Write(directory, merge.info.GetDelFileName()); merge.info.SetDelCount(delCount); System.Diagnostics.Debug.Assert(delCount == deletes.Count()); } } } /* FIXME if we want to support non-contiguous segment merges */ private bool CommitMerge(MergePolicy.OneMerge merge, SegmentMerger merger, int mergedDocCount) { lock (this) { System.Diagnostics.Debug.Assert(TestPoint("startCommitMerge")); if (hitOOM) return false; if (infoStream != null) Message("commitMerge: " + merge.SegString(directory) + " index=" + SegString()); System.Diagnostics.Debug.Assert(merge.registerDone); // If merge was explicitly aborted, or, if rollback() or // RollbackTransaction() had been called since our merge // started (which results in an unqualified // deleter.Refresh() call that will remove any index // file that current segments does not reference), we // abort this merge if (merge.IsAborted()) { if (infoStream != null) Message("commitMerge: skipping merge " + merge.SegString(directory) + ": it was aborted"); deleter.Refresh(merge.info.name); return false; } int start = EnsureContiguousMerge(merge); CommitMergedDeletes(merge); docWriter.RemapDeletes(segmentInfos, merger.GetDocMaps(), merger.GetDelCounts(), merge, mergedDocCount); // Simple optimization: if the doc store we are using // has been closed and is in now compound format (but // wasn't when we started), then we will switch to the // compound format as well: string mergeDocStoreSegment = merge.info.GetDocStoreSegment(); if (mergeDocStoreSegment != null && !merge.info.GetDocStoreIsCompoundFile()) { int size = segmentInfos.Count; for (int i = 0; i < size; i++) { SegmentInfo info = segmentInfos.Info(i); string docStoreSegment = info.GetDocStoreSegment(); if (docStoreSegment != null && docStoreSegment.Equals(mergeDocStoreSegment) && info.GetDocStoreIsCompoundFile()) { merge.info.SetDocStoreIsCompoundFile(true); break; } } } merge.info.SetHasProx(merger.HasProx()); //segmentInfos.RemoveRange(start, start + merge.segments.Count); segmentInfos.RemoveRange(start, merge.segments.Count); System.Diagnostics.Debug.Assert(!segmentInfos.Contains(merge.info)); segmentInfos.Insert(start, merge.info); // Must checkpoint before decrefing so any newly // referenced files in the new merge.info are incref'd // first: Checkpoint(); DecrefMergeSegments(merge); if (merge.optimize) segmentsToOptimize[merge.info] = merge.info; return true; } } private void DecrefMergeSegments(MergePolicy.OneMerge merge) { SegmentInfos sourceSegmentsClone = merge.segmentsClone; int numSegmentsToMerge = sourceSegmentsClone.Count; System.Diagnostics.Debug.Assert(merge.increfDone); merge.increfDone = false; for (int i = 0; i < numSegmentsToMerge; i++) { SegmentInfo previousInfo = sourceSegmentsClone.Info(i); // Decref all files for this SegmentInfo (this // matches the incref in mergeInit): if (previousInfo.dir == directory) deleter.DecRef(previousInfo.Files()); } } private void HandleMergeException(System.Exception t, MergePolicy.OneMerge merge) { // Set the exception on the merge, so if // Optimize() is waiting on us it sees the root // cause exception: merge.SetException(t); AddMergeException(merge); if (t is MergePolicy.MergeAbortedException) { // We can ignore this exception (it happens when // close(false) or rollback is called), unless the // merge involves segments from external directories, // in which case we must throw it so, for example, the // rollbackTransaction code in AddIndexes* is // executed. if (merge.isExternal) throw (MergePolicy.MergeAbortedException)t; } else if (t is System.IO.IOException) throw (System.IO.IOException)t; else if (t is System.SystemException) throw (System.SystemException)t; else if (t is System.Exception) throw (System.Exception)t; else // Should not get here throw new System.SystemException(null, t); } /** * Merges the indicated segments, replacing them in the stack with a * single segment. */ public /* for nunit test, was: internal */ void Merge(MergePolicy.OneMerge merge) { bool success = false; try { try { try { MergeInit(merge); if (infoStream != null) Message("now merge\n merge=" + merge.SegString(directory) + "\n merge=" + merge + "\n index=" + SegString()); MergeMiddle(merge); success = true; } catch (System.Exception t) { HandleMergeException(t, merge); } } finally { lock (this) { try { MergeFinish(merge); if (!success) { if (infoStream != null) Message("hit exception during merge"); if (merge.info != null && !segmentInfos.Contains(merge.info)) deleter.Refresh(merge.info.name); } // This merge (and, generally, any change to the // segments) may now enable new merges, so we call // merge policy & update pending merges. if (success && !merge.IsAborted() && !closed && !closing) UpdatePendingMerges(merge.maxNumSegmentsOptimize, merge.optimize); } finally { runningMerges.Remove(merge); } } } } catch (System.OutOfMemoryException oom) { hitOOM = true; throw oom; } } /** Checks whether this merge involves any segments * already participating in a merge. If not, this merge * is "registered", meaning we record that its segments * are now participating in a merge, and true is * returned. Else (the merge conflicts) false is * returned. */ internal bool RegisterMerge(MergePolicy.OneMerge merge) { lock (this) { if (merge.registerDone) return true; if (stopMerges) { merge.Abort(); throw new MergePolicy.MergeAbortedException("merge is aborted: " + merge.SegString(directory)); } int count = merge.segments.Count; bool isExternal = false; for (int i = 0; i < count; i++) { SegmentInfo info = merge.segments.Info(i); if (mergingSegments.ContainsKey(info)) return false; if (segmentInfos.IndexOf(info) == -1) return false; if (info.dir != directory) isExternal = true; } EnsureContiguousMerge(merge); pendingMerges.Add(merge); if (infoStream != null) Message("add merge to pendingMerges: " + merge.SegString(directory) + " [total " + pendingMerges.Count + " pending]"); merge.mergeGen = mergeGen; merge.isExternal = isExternal; // OK it does not conflict; now record that this merge // is running (while synchronized) to avoid race // condition where two conflicting merges from different // threads, start for (int i = 0; i < count; i++) mergingSegments[merge.segments.Info(i)] = merge.segments.Info(i); // Merge is now registered merge.registerDone = true; return true; } } /** Does initial setup for a merge, which is fast but holds * the synchronized lock on IndexWriter instance. */ internal void MergeInit(MergePolicy.OneMerge merge) { lock (this) { bool success = false; try { _MergeInit(merge); success = true; } finally { if (!success) { MergeFinish(merge); runningMerges.Remove(merge); } } } } private void _MergeInit(MergePolicy.OneMerge merge) { lock (this) { System.Diagnostics.Debug.Assert(TestPoint("startMergeInit")); System.Diagnostics.Debug.Assert(merge.registerDone); System.Diagnostics.Debug.Assert(!merge.optimize || merge.maxNumSegmentsOptimize > 0); if (merge.info != null) // mergeInit already done return; if (merge.IsAborted()) return; bool changed = ApplyDeletes(); // If autoCommit == true then all deletes should have // been flushed when we flushed the last segment System.Diagnostics.Debug.Assert(!changed || !autoCommit); SegmentInfos sourceSegments = merge.segments; int end = sourceSegments.Count; // Check whether this merge will allow us to skip // merging the doc stores (stored field & vectors). // This is a very substantial optimization (saves tons // of IO) that can only be applied with // autoCommit=false. Directory lastDir = directory; string lastDocStoreSegment = null; int next = -1; bool mergeDocStores = false; bool doFlushDocStore = false; string currentDocStoreSegment = docWriter.GetDocStoreSegment(); // Test each segment to be merged: check if we need to // flush/merge doc stores for (int i = 0; i < end; i++) { SegmentInfo si = sourceSegments.Info(i); // If it has deletions we must merge the doc stores if (si.HasDeletions()) mergeDocStores = true; // If it has its own (private) doc stores we must // merge the doc stores if (-1 == si.GetDocStoreOffset()) mergeDocStores = true; // If it has a different doc store segment than // previous segments, we must merge the doc stores string docStoreSegment_Renamed = si.GetDocStoreSegment(); if (docStoreSegment_Renamed == null) mergeDocStores = true; else if (lastDocStoreSegment == null) lastDocStoreSegment = docStoreSegment_Renamed; else if (!lastDocStoreSegment.Equals(docStoreSegment_Renamed)) mergeDocStores = true; // Segments' docScoreOffsets must be in-order, // contiguous. For the default merge policy now // this will always be the case but for an arbitrary // merge policy this may not be the case if (-1 == next) next = si.GetDocStoreOffset() + si.docCount; else if (next != si.GetDocStoreOffset()) mergeDocStores = true; else next = si.GetDocStoreOffset() + si.docCount; // If the segment comes from a different directory // we must merge if (lastDir != si.dir) mergeDocStores = true; // If the segment is referencing the current "live" // doc store outputs then we must merge if (si.GetDocStoreOffset() != -1 && currentDocStoreSegment != null && si.GetDocStoreSegment().Equals(currentDocStoreSegment)) { doFlushDocStore = true; } } int docStoreOffset; string docStoreSegment; bool docStoreIsCompoundFile; if (mergeDocStores) { docStoreOffset = -1; docStoreSegment = null; docStoreIsCompoundFile = false; } else { SegmentInfo si = sourceSegments.Info(0); docStoreOffset = si.GetDocStoreOffset(); docStoreSegment = si.GetDocStoreSegment(); docStoreIsCompoundFile = si.GetDocStoreIsCompoundFile(); } if (mergeDocStores && doFlushDocStore) { // SegmentMerger intends to merge the doc stores // (stored fields, vectors), and at least one of the // segments to be merged refers to the currently // live doc stores. // TODO: if we know we are about to merge away these // newly flushed doc store files then we should not // make compound file out of them... if (infoStream != null) Message("now flush at merge"); DoFlush(true, false); //Flush(false, true, false); } // We must take a full copy at this point so that we can // properly merge deletes in CommitMerge() merge.segmentsClone = (SegmentInfos)merge.segments.Clone(); for (int i = 0; i < end; i++) { SegmentInfo si = merge.segmentsClone.Info(i); // IncRef all files for this segment info to make sure // they are not removed while we are trying to merge. if (si.dir == directory) deleter.IncRef(si.Files()); } merge.increfDone = true; merge.mergeDocStores = mergeDocStores; // Bind a new segment name here so even with // ConcurrentMergePolicy we keep deterministic segment // names. merge.info = new SegmentInfo(NewSegmentName(), 0, directory, false, true, docStoreOffset, docStoreSegment, docStoreIsCompoundFile, false); // Also enroll the merged segment into mergingSegments; // this prevents it from getting selected for a merge // after our merge is done but while we are building the // CFS: mergingSegments[merge.info] = merge.info; } } /** This is called after merging a segment and before * building its CFS. Return true if the files should be * sync'd. If you return false, then the source segment * files that were merged cannot be deleted until the CFS * file is built & sync'd. So, returning false consumes * more transient disk space, but saves performance of * not having to sync files which will shortly be deleted * anyway. * @deprecated -- this will be removed in 3.0 when * autoCommit is hardwired to false */ [System.Obsolete("this will be removed in 3.0 when autoCommit is hardwired to false")] private bool DoCommitBeforeMergeCFS(MergePolicy.OneMerge merge) { lock (this) { long freeableBytes = 0; int size = merge.segments.Count; for (int i = 0; i < size; i++) { SegmentInfo info = merge.segments.Info(i); // It's only important to sync if the most recent // commit actually references this segment, because if // it doesn't, even without syncing we will free up // the disk space: if (rollbackSegments.ContainsKey(info)) { int loc = rollbackSegments[info]; SegmentInfo oldInfo = rollbackSegmentInfos.Info(loc); if (oldInfo.GetUseCompoundFile() != info.GetUseCompoundFile()) freeableBytes += info.SizeInBytes(); } } // If we would free up more than 1/3rd of the index by // committing now, then do so: long totalBytes = 0; int numSegments = segmentInfos.Count; for (int i = 0; i < numSegments; i++) totalBytes += segmentInfos.Info(i).SizeInBytes(); if (3 * freeableBytes > totalBytes) return true; else return false; } } /** Does fininishing for a merge, which is fast but holds * the synchronized lock on IndexWriter instance. */ internal void MergeFinish(MergePolicy.OneMerge merge) { lock (this) { // Optimize, AddIndexes or finishMerges may be waiting // on merges to finish. System.Threading.Monitor.PulseAll(this); if (merge.increfDone) DecrefMergeSegments(merge); System.Diagnostics.Debug.Assert(merge.registerDone); SegmentInfos sourceSegments = merge.segments; int end = sourceSegments.Count; for (int i = 0; i < end; i++) mergingSegments.Remove(sourceSegments.Info(i)); if (merge.info != null) mergingSegments.Remove(merge.info); merge.registerDone = false; } } /** Does the actual (time-consuming) work of the merge, * but without holding synchronized lock on IndexWriter * instance */ private int MergeMiddle(MergePolicy.OneMerge merge) { merge.CheckAborted(directory); string mergedName = merge.info.name; SegmentMerger merger = null; int mergedDocCount = 0; SegmentInfos sourceSegments = merge.segments; SegmentInfos sourceSegmentsClone = merge.segmentsClone; int numSegments = sourceSegments.Count; if (infoStream != null) Message("merging " + merge.SegString(directory)); merger = new SegmentMerger(this, mergedName, merge); bool success = false; // This is try/finally to make sure merger's readers are // closed: try { int totDocCount = 0; for (int i = 0; i < numSegments; i++) { SegmentInfo si = sourceSegmentsClone.Info(i); IndexReader reader = SegmentReader.Get(true, si, MERGE_READ_BUFFER_SIZE, merge.mergeDocStores); // no need to set deleter (yet) merger.Add(reader); totDocCount += reader.NumDocs(); } if (infoStream != null) { Message("merge: total " + totDocCount + " docs"); } merge.CheckAborted(directory); // This is where all the work happens: mergedDocCount = merge.info.docCount = merger.Merge(merge.mergeDocStores); System.Diagnostics.Debug.Assert(mergedDocCount == totDocCount); success = true; } finally { // close readers before we attempt to delete // now-obsolete segments if (merger != null) { merger.CloseReaders(); } } if (!CommitMerge(merge, merger, mergedDocCount)) // commitMerge will return false if this merge was aborted return 0; if (merge.useCompoundFile) { // Maybe force a sync here to allow reclaiming of the // disk space used by the segments we just merged: if (autoCommit && DoCommitBeforeMergeCFS(merge)) { long size; lock (this) { size = merge.info.SizeInBytes(); } Commit(size); } success = false; string compoundFileName = mergedName + "." + IndexFileNames.COMPOUND_FILE_EXTENSION; try { merger.CreateCompoundFile(compoundFileName); success = true; } catch (System.IO.IOException ioe) { lock (this) { if (merge.IsAborted()) { // This can happen if rollback or close(false) // is called -- fall through to logic below to // remove the partially created CFS: success = true; } else HandleMergeException(ioe, merge); } } catch (System.Exception t) { HandleMergeException(t, merge); } finally { if (!success) { if (infoStream != null) Message("hit exception creating compound file during merge"); lock (this) { deleter.DeleteFile(compoundFileName); } } } if (merge.IsAborted()) { if (infoStream != null) Message("abort merge after building CFS"); deleter.DeleteFile(compoundFileName); return 0; } lock (this) { if (segmentInfos.IndexOf(merge.info) == -1 || merge.IsAborted()) { // Our segment (committed in non-compound // format) got merged away while we were // building the compound format. deleter.DeleteFile(compoundFileName); } else { merge.info.SetUseCompoundFile(true); Checkpoint(); } } } // Force a sync after commiting the merge. Once this // sync completes then all index files referenced by the // current segmentInfos are on stable storage so if the // OS/machine crashes, or power cord is yanked, the // index will be intact. Note that this is just one // (somewhat arbitrary) policy; we could try other // policies like only sync if it's been > X minutes or // more than Y bytes have been written, etc. if (autoCommit) { long size; lock (this) { size = merge.info.SizeInBytes(); } Commit(size); } return mergedDocCount; } internal void AddMergeException(MergePolicy.OneMerge merge) { lock (this) { System.Diagnostics.Debug.Assert(merge.GetException() != null); if (!mergeExceptions.Contains(merge) && mergeGen == merge.mergeGen) mergeExceptions.Add(merge); } } // Apply buffered deletes to all segments. private bool ApplyDeletes() { lock (this) { System.Diagnostics.Debug.Assert(TestPoint("startApplyDeletes")); SegmentInfos rollback = (SegmentInfos)segmentInfos.Clone(); bool success = false; bool changed; try { changed = docWriter.ApplyDeletes(segmentInfos); success = true; } finally { if (!success) { if (infoStream != null) Message("hit exception flushing deletes"); // Carefully remove any partially written .del // files int size = rollback.Count; for (int i = 0; i < size; i++) { string newDelFileName = segmentInfos.Info(i).GetDelFileName(); string delFileName = rollback.Info(i).GetDelFileName(); if (newDelFileName != null && !newDelFileName.Equals(delFileName)) deleter.DeleteFile(newDelFileName); } // Fully replace the segmentInfos since flushed // deletes could have changed any of the // SegmentInfo instances: segmentInfos.Clear(); for (int i = 0; i < rollback.Count; i++) segmentInfos.Add(rollback.Info(i)); } } if (changed) Checkpoint(); return changed; } } // For test purposes. public int GetBufferedDeleteTermsSize() { lock (this) { return docWriter.GetBufferedDeleteTerms().Count; } } // For test purposes. public int GetNumBufferedDeleteTerms() { lock (this) { return docWriter.GetNumBufferedDeleteTerms(); } } // utility routines for tests public SegmentInfo NewestSegment() { return segmentInfos.Info(segmentInfos.Count - 1); } public string SegString() { lock (this) { return SegString(segmentInfos); } } private string SegString(SegmentInfos infos) { lock (this) { System.Text.StringBuilder buffer = new System.Text.StringBuilder(); int count = infos.Count; for (int i = 0; i < count; i++) { if (i > 0) { buffer.Append(' '); } SegmentInfo info = infos.Info(i); buffer.Append(info.SegString(directory)); if (info.dir != directory) buffer.Append("**"); } return buffer.ToString(); } } // Files that have been sync'd already private Dictionarytrue iff the index in the named directory is
* currently locked.
* @param directory the directory to check for a lock
* @throws System.IO.IOException if there is a low-level IO error
*/
public static bool IsLocked(Directory directory)
{
return directory.MakeLock(WRITE_LOCK_NAME).IsLocked();
}
/**
* Returns true iff the index in the named directory is
* currently locked.
* @param directory the directory to check for a lock
* @throws System.IO.IOException if there is a low-level IO error
*/
public static bool IsLocked(string directory)
{
Directory dir = FSDirectory.GetDirectory(directory);
try
{
return IsLocked(dir);
}
finally
{
dir.Close();
}
}
/**
* Forcibly unlocks the index in the named directory.
*
* Caution: this should only be used by failure recovery code,
* when it is known that no other process nor thread is in fact
* currently accessing this index.
*/
public static void Unlock(Directory directory)
{
directory.MakeLock(IndexWriter.WRITE_LOCK_NAME).Release();
}
/**
* Specifies maximum field length in {@link IndexWriter} constructors.
* {@link #setMaxFieldLength(int)} overrides the value set by
* the constructor.
*/
public sealed class MaxFieldLength
{
private int limit;
private string name;
/**
* Private type-safe-enum-pattern constructor.
*
* @param name instance name
* @param limit maximum field length
*/
private MaxFieldLength(string name, int limit)
{
this.name = name;
this.limit = limit;
}
/**
* Public constructor to allow users to specify the maximum field size limit.
*
* @param limit The maximum field length
*/
public MaxFieldLength(int limit)
: this("User-specified", limit)
{
}
public int GetLimit()
{
return limit;
}
public override string ToString()
{
return name + ":" + limit;
}
/** Sets the maximum field length to {@link Integer#MAX_VALUE}. */
public static readonly MaxFieldLength UNLIMITED = new MaxFieldLength("UNLIMITED", int.MaxValue);
/**
* Sets the maximum field length to
* {@link #DEFAULT_MAX_FIELD_LENGTH}
* */
public static readonly MaxFieldLength LIMITED = new MaxFieldLength("LIMITED", DEFAULT_MAX_FIELD_LENGTH);
}
// Used only by assert for testing. Current points:
// startDoFlush
// startCommitMerge
// startStartCommit
// midStartCommit
// midStartCommit2
// midStartCommitSuccess
// finishStartCommit
// startCommitMergeDeletes
// startMergeInit
// startApplyDeletes
// DocumentsWriter.ThreadState.init start
protected internal virtual bool TestPoint(string name)
{
return true;
}
}
}
/// This may be used to parallelize batch indexing. A large document
// /// collection can be broken into sub-collections. Each sub-collection can be
// /// indexed in parallel, on a different thread, process or machine. The
// /// complete index can then be created by merging sub-collection indexes
// /// with this method.
// ///
// /// NOTE: the index in each Directory must not be
// /// changed (opened by a writer) while this method is
// /// running. This method does not acquire a write lock in
// /// each input Directory, so it is up to the caller to
// /// enforce this.
// ///
// /// NOTE: while this is running, any attempts to
// /// add or delete documents (with another thread) will be
// /// paused until this method completes.
// ///
// /// After this completes, the index is optimized.
// ///
// /// This method is transactional in how Exceptions are
// /// handled: it does not commit a new segments_N file until
// /// all indexes are added. This means if an Exception
// /// occurs (for example disk full), then either no indexes
// /// will have been added or they all will have been. If an Exception is hit, it's still possible that all
// /// indexes were successfully added. This happens when the
// /// Exception is hit when trying to build a CFS file. In
// /// this case, one segment in the index will be in non-CFS
// /// format, even when using compound file format. Also note that on an Exception, the index may still
// /// have been partially or fully optimized even though none
// /// of the input indexes were added. Note that this requires temporary free space in the
// /// Directory up to 2X the sum of all input indexes
// /// (including the starting index). If readers/searchers
// /// are open against the starting index, then temporary
// /// free space required will be higher by the size of the
// /// starting index (see {@link #Optimize()} for details).
// /// Once this completes, the final size of the index
// /// will be less than the sum of all input index sizes
// /// (including the starting index). It could be quite a
// /// bit smaller (if there were many pending deletes) or
// /// just slightly smaller. See LUCENE-702
// /// for details.
// /// This is similar to AddIndexes(Directory[]). However, no Optimize()
// /// is called either at the beginning or at the end. Instead, merges
// /// are carried out as necessary.
// ///
// /// NOTE: the index in each Directory must not be
// /// changed (opened by a writer) while this method is
// /// running. This method does not acquire a write lock in
// /// each input Directory, so it is up to the caller to
// /// enforce this.
// ///
// /// NOTE: while this is running, any attempts to
// /// add or delete documents (with another thread) will be
// /// paused until this method completes.
// ///
// ///
// /// This requires this index not be among those to be added, and the
// /// upper bound* of those segment doc counts not exceed maxMergeDocs.
// ///
// /// See {@link #AddIndexes(Directory[])} for
// /// details on transactional semantics, temporary free
// /// space required in the Directory, and non-CFS segments
// /// on an Exception. After this completes, the index is optimized. The provided IndexReaders are not closed. NOTE: the index in each Directory must not be
// /// changed (opened by a writer) while this method is
// /// running. Thiw method does not acquire a write lock in
// /// each input Directory, so it is up to the caller to
// /// enforce this.
// /// NOTE: while this is running, any attempts to
// /// add or delete documents (with another thread) will be
// /// paused until this method completes. See {@link #AddIndexes(Directory[])} for
// /// details on transactional semantics, temporary free
// /// space required in the Directory, and non-CFS segments
// /// on an Exception. Note: if autoCommit=false, flushed data would still
// /// not be visible to readers, until {@link #close} is called.
// ///