Package org.apache.lucene.store
Class NRTCachingDirectory
- java.lang.Object
-
- org.apache.lucene.store.Directory
-
- org.apache.lucene.store.NRTCachingDirectory
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public class NRTCachingDirectory extends Directory
Wraps aRAMDirectory
around any provided delegate directory, to be used during NRT search.This class is likely only useful in a near-real-time context, where indexing rate is lowish but reopen rate is highish, resulting in many tiny files being written. This directory keeps such segments (as well as the segments produced by merging them, as long as they are small enough), in RAM.
This is safe to use: when your app calls {IndexWriter#commit}, all cached files will be flushed from the cached and sync'd.
Here's a simple example usage:
Directory fsDir = FSDirectory.open(new File("/path/to/index")); NRTCachingDirectory cachedFSDir = new NRTCachingDirectory(fsDir, 5.0, 60.0); IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_47, analyzer); IndexWriter writer = new IndexWriter(cachedFSDir, conf);
This will cache all newly flushed segments, all merges whose expected segment size is <= 5 MB, unless the net cached bytes exceeds 60 MB at which point all writes will not be cached (until the net bytes falls below 60 MB).
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.store.Directory
Directory.IndexInputSlicer
-
-
Constructor Summary
Constructors Constructor Description NRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is <= maxMergeSizeMB, and 2) the total cached bytes is <= maxCachedMB
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
clearLock(java.lang.String name)
Attempt to clear (forcefully unlock and remove) the specified lock.void
close()
Close this directory, which flushes any cached files to the delegate and then closes the delegate.IndexOutput
createOutput(java.lang.String name, IOContext context)
Creates a new, empty file in the directory with the given name.Directory.IndexInputSlicer
createSlicer(java.lang.String name, IOContext context)
Creates anDirectory.IndexInputSlicer
for the given file name.void
deleteFile(java.lang.String name)
Removes an existing file in the directory.protected boolean
doCacheWrite(java.lang.String name, IOContext context)
Subclass can override this to customize logic; return true if this file should be written to the RAMDirectory.boolean
fileExists(java.lang.String name)
Returns true iff a file with the given name exists.long
fileLength(java.lang.String name)
Returns the length of a file in the directory.Directory
getDelegate()
LockFactory
getLockFactory()
Get the LockFactory that this Directory instance is using for its locking implementation.java.lang.String
getLockID()
Return a string identifier that uniquely differentiates this Directory instance from other Directory instances.java.lang.String[]
listAll()
Returns an array of strings, one for each file in the directory.java.lang.String[]
listCachedFiles()
Lock
makeLock(java.lang.String name)
Construct aLock
.IndexInput
openInput(java.lang.String name, IOContext context)
Returns a stream reading an existing file, with the specified read buffer size.void
setLockFactory(LockFactory lf)
Set the LockFactory that this Directory instance should use for its locking implementation.long
sizeInBytes()
Returns how many bytes are being used by the RAMDirectory cachevoid
sync(java.util.Collection<java.lang.String> fileNames)
Ensure that any writes to these files are moved to stable storage.java.lang.String
toString()
-
Methods inherited from class org.apache.lucene.store.Directory
copy, ensureOpen
-
-
-
-
Constructor Detail
-
NRTCachingDirectory
public NRTCachingDirectory(Directory delegate, double maxMergeSizeMB, double maxCachedMB)
We will cache a newly created output if 1) it's a flush or a merge and the estimated size of the merged segment is <= maxMergeSizeMB, and 2) the total cached bytes is <= maxCachedMB
-
-
Method Detail
-
getDelegate
public Directory getDelegate()
-
getLockFactory
public LockFactory getLockFactory()
Description copied from class:Directory
Get the LockFactory that this Directory instance is using for its locking implementation. Note that this may be null for Directory implementations that provide their own locking implementation.- Specified by:
getLockFactory
in classDirectory
-
setLockFactory
public void setLockFactory(LockFactory lf) throws java.io.IOException
Description copied from class:Directory
Set the LockFactory that this Directory instance should use for its locking implementation. Each * instance of LockFactory should only be used for one directory (ie, do not share a single instance across multiple Directories).- Specified by:
setLockFactory
in classDirectory
- Parameters:
lf
- instance ofLockFactory
.- Throws:
java.io.IOException
-
getLockID
public java.lang.String getLockID()
Description copied from class:Directory
Return a string identifier that uniquely differentiates this Directory instance from other Directory instances. This ID should be the same if two Directory instances (even in different JVMs and/or on different machines) are considered "the same index". This is how locking "scopes" to the right index.
-
makeLock
public Lock makeLock(java.lang.String name)
Description copied from class:Directory
Construct aLock
.
-
clearLock
public void clearLock(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Attempt to clear (forcefully unlock and remove) the specified lock. Only call this at a time when you are certain this lock is no longer in use.
-
listAll
public java.lang.String[] listAll() throws java.io.IOException
Description copied from class:Directory
Returns an array of strings, one for each file in the directory.- Specified by:
listAll
in classDirectory
- Throws:
NoSuchDirectoryException
- if the directory is not prepared for any write operations (such asDirectory.createOutput(String, IOContext)
).java.io.IOException
- in case of other IO errors
-
sizeInBytes
public long sizeInBytes()
Returns how many bytes are being used by the RAMDirectory cache
-
fileExists
public boolean fileExists(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Returns true iff a file with the given name exists.- Specified by:
fileExists
in classDirectory
- Throws:
java.io.IOException
-
deleteFile
public void deleteFile(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Removes an existing file in the directory.- Specified by:
deleteFile
in classDirectory
- Throws:
java.io.IOException
-
fileLength
public long fileLength(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Returns the length of a file in the directory. This method follows the following contract:- Must throw
FileNotFoundException
if the file does not exist (notjava.nio.file.NoSuchFileException
of Java 7). - Returns a value ≥0 if the file exists, which specifies its length.
- Specified by:
fileLength
in classDirectory
- Parameters:
name
- the name of the file for which to return the length.- Throws:
java.io.IOException
- if there was an IO error while retrieving the file's length.
- Must throw
-
listCachedFiles
public java.lang.String[] listCachedFiles()
-
createOutput
public IndexOutput createOutput(java.lang.String name, IOContext context) throws java.io.IOException
Description copied from class:Directory
Creates a new, empty file in the directory with the given name. Returns a stream writing this file.- Specified by:
createOutput
in classDirectory
- Throws:
java.io.IOException
-
sync
public void sync(java.util.Collection<java.lang.String> fileNames) throws java.io.IOException
Description copied from class:Directory
Ensure that any writes to these files are moved to stable storage. Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.
NOTE: Clients may call this method for same files over and over again, so some impls might optimize for that. For other impls the operation can be a noop, for various reasons.
-
openInput
public IndexInput openInput(java.lang.String name, IOContext context) throws java.io.IOException
Description copied from class:Directory
Returns a stream reading an existing file, with the specified read buffer size. The particular Directory implementation may ignore the buffer size. Currently the only Directory implementations that respect this parameter areFSDirectory
andCompoundFileDirectory
.- Must throw
FileNotFoundException
if the file does not exist (notjava.nio.file.NoSuchFileException
of Java 7). - Must throw
-
createSlicer
public Directory.IndexInputSlicer createSlicer(java.lang.String name, IOContext context) throws java.io.IOException
Description copied from class:Directory
Creates anDirectory.IndexInputSlicer
for the given file name. IndexInputSlicer allows otherDirectory
implementations to efficiently open one or more slicedIndexInput
instances from a single file handle. The underlying file handle is kept open until theDirectory.IndexInputSlicer
is closed.- Must throw
FileNotFoundException
if the file does not exist (notjava.nio.file.NoSuchFileException
of Java 7).- Overrides:
createSlicer
in classDirectory
- Throws:
java.io.IOException
- if anIOException
occurs
- Must throw
-
close
public void close() throws java.io.IOException
Close this directory, which flushes any cached files to the delegate and then closes the delegate.
-
doCacheWrite
protected boolean doCacheWrite(java.lang.String name, IOContext context)
Subclass can override this to customize logic; return true if this file should be written to the RAMDirectory.
-
-