Package org.apache.lucene.tests.index
Class BaseKnnVectorsFormatTestCase
- java.lang.Object
-
- org.junit.Assert
-
- org.apache.lucene.tests.util.LuceneTestCase
-
- org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase
-
public abstract class BaseKnnVectorsFormatTestCase extends LuceneTestCase
Base class aiming at testingvectors formats
. To test a new format, all you need is to register a newCodec
which uses it and extend this class and overridegetCodec()
.- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.tests.util.LuceneTestCase
LuceneTestCase.AwaitsFix, LuceneTestCase.BadApple, LuceneTestCase.Monster, LuceneTestCase.Nightly, LuceneTestCase.SuppressCodecs, LuceneTestCase.SuppressFileSystems, LuceneTestCase.SuppressFsync, LuceneTestCase.SuppressReproduceLine, LuceneTestCase.SuppressSysoutChecks, LuceneTestCase.SuppressTempFileChecks, LuceneTestCase.ThrowingConsumer<T>, LuceneTestCase.ThrowingRunnable, LuceneTestCase.Weekly
-
-
Field Summary
-
Fields inherited from class org.apache.lucene.tests.util.LuceneTestCase
assertsAreEnabled, classRules, DEFAULT_LINE_DOCS_FILE, INFOSTREAM, JENKINS_LARGE_LINE_DOCS_FILE, LEAVE_TEMPORARY, MAYBE_CACHE_POLICY, RANDOM_MULTIPLIER, ruleChain, suiteFailureMarker, SYSPROP_AWAITSFIX, SYSPROP_BADAPPLES, SYSPROP_FAILFAST, SYSPROP_MAXFAILURES, SYSPROP_MONSTER, SYSPROP_NIGHTLY, SYSPROP_WEEKLY, TEST_ASSERTS_ENABLED, TEST_AWAITSFIX, TEST_BADAPPLES, TEST_CODEC, TEST_DIRECTORY, TEST_DOCVALUESFORMAT, TEST_LINE_DOCS_FILE, TEST_MONSTER, TEST_NIGHTLY, TEST_POSTINGSFORMAT, TEST_THROTTLING, TEST_WEEKLY, VERBOSE
-
-
Constructor Summary
Constructors Constructor Description BaseKnnVectorsFormatTestCase()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected void
addRandomFields(Document doc)
Add random fields to the provided document.protected <D extends Directory>
DapplyCreatedVersionMajor(D d)
Set the created version of the givenDirectory
and return it.protected Collection<String>
excludedExtensionsFromByteCounts()
Return the list of extensions that should be excluded from byte counts when comparing indices that store the same content.protected abstract Codec
getCodec()
Returns the codec to run tests againstprotected int
getCreatedVersionMajor()
Returns the major version that this codec is compatible with.void
init()
protected DirectoryReader
maybeWrapWithMergingReader(DirectoryReader r)
protected boolean
mergeIsStable()
protected float[]
randomNormalizedVector(int dim)
protected VectorSimilarityFunction
randomSimilarity()
protected float[]
randomVector(int dim)
protected VectorEncoding
randomVectorEncoding()
This method is overrideable since old codec versions only supportVectorEncoding.FLOAT32
.void
setUp()
For subclasses to override.protected boolean
shouldTestMergeInstance()
Returnsfalse
if only the regular fields reader should be tested, andtrue
if only the merge instance should be tested.void
tearDown()
For subclasses to override.void
testAddIndexesDirectory0()
void
testAddIndexesDirectory01()
void
testAddIndexesDirectory1()
void
testAdvance()
void
testCheckIndexIncludesVectors()
void
testCheckIntegrityReadsAllBytes()
This test is a best effort at verifying that checkIntegrity doesn't miss any files.void
testDeleteAllVectorDocs()
void
testDifferentCodecs1()
void
testDifferentCodecs2()
void
testFieldConstructor()
void
testFieldConstructorExceptions()
void
testFieldSetValue()
void
testIllegalDimChangeTwoDocs()
void
testIllegalDimChangeTwoWriters()
void
testIllegalDimChangeViaAddIndexesCodecReader()
void
testIllegalDimChangeViaAddIndexesDirectory()
void
testIllegalDimChangeViaAddIndexesSlowCodecReader()
void
testIllegalDimensionTooLarge()
void
testIllegalEmptyVector()
void
testIllegalMultipleValues()
void
testIllegalSimilarityFunctionChange()
void
testIllegalSimilarityFunctionChangeTwoWriters()
void
testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader()
void
testIllegalSimilarityFunctionChangeViaAddIndexesDirectory()
void
testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader()
void
testIndexedValueNotAliased()
void
testIndexMultipleKnnVectorFields()
void
testInvalidKnnVectorFieldUsage()
void
testKnnVectorFieldMissingFromOneSegment()
void
testMergeStability()
The purpose of this test is to make sure that bulk merge doesn't accumulate useless data over runs.void
testMultiClose()
Calls close multiple times on closeable codec apisvoid
testRandom()
Index random vectors, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.void
testRandomBytes()
Index random vectors as bytes, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.void
testRandomExceptions()
Tests exception handling on write and openInput/createOutputvoid
testRandomWithUpdatesAndGraph()
Index random vectors, sometimes skipping documents, sometimes updating a document, sometimes merging, sometimes sorting the index, using an HNSW similarity function so as to also produce a graph, and verify that the expected values can be read back consistently.void
testSearchWithVisitedLimit()
Tests whetherKnnVectorsReader.search(java.lang.String, float[], org.apache.lucene.search.KnnCollector, org.apache.lucene.util.Bits)
implementations obey the limit on the number of visited vectors.void
testSimilarityFunctionIdentifiers()
void
testSortedIndex()
void
testSortedIndexBytes()
void
testSparseVectors()
void
testVectorEncodingOrdinals()
void
testVectorValuesReportCorrectDocs()
-
Methods inherited from class org.apache.lucene.tests.util.LuceneTestCase
addVirusChecker, assertDeletedDocsEquals, assertDocsAndPositionsEnumEquals, assertDocsEnumEquals, assertDocsSkippingEquals, assertDocValuesEquals, assertDocValuesEquals, assertFieldInfosEquals, assertNormsEquals, assertPointsEquals, assertPositionsSkippingEquals, assertReaderEquals, assertReaderStatisticsEquals, assertStoredFieldEquals, assertStoredFieldsEquals, assertTermsEnumEquals, assertTermsEquals, assertTermsEquals, assertTermsStatisticsEquals, assertTermStatsEquals, assertTermVectorsEquals, asSet, assumeFalse, assumeNoException, assumeTrue, assumeWorkingMMapOnWindows, atLeast, atLeast, callStackContains, callStackContains, callStackContainsAnyOf, closeAfterSuite, closeAfterTest, collate, createTempDir, createTempDir, createTempFile, createTempFile, dumpArray, dumpIterator, ensureSaneIWCOnNightly, expectThrows, expectThrows, expectThrows, expectThrowsAnyOf, expectThrowsAnyOf, getDataInputStream, getDataPath, getJvmForkArguments, getOnlyLeafReader, getTestClass, getTestName, hasWorkingMMapOnWindows, isTestThread, localeForLanguageTag, maybeChangeLiveIndexWriterConfig, maybeWrapReader, newAlcoholicMergePolicy, newAlcoholicMergePolicy, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newDirectory, newDirectory, newDirectory, newDirectory, newDirectory, newField, newField, newFSDirectory, newFSDirectory, newIndexWriterConfig, newIndexWriterConfig, newIndexWriterConfig, newIOContext, newIOContext, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newMaybeVirusCheckingDirectory, newMaybeVirusCheckingFSDirectory, newMergePolicy, newMergePolicy, newMergePolicy, newMockDirectory, newMockDirectory, newMockDirectory, newMockFSDirectory, newMockFSDirectory, newSearcher, newSearcher, newSearcher, newSearcher, newSnapshotIndexWriterConfig, newStringField, newStringField, newStringField, newStringField, newTextField, newTextField, newTieredMergePolicy, newTieredMergePolicy, overrideDefaultQueryCache, overrideTestDefaultQueryCache, random, randomLocale, randomTimeZone, rarely, rarely, replaceMaxFailureRule, resetDefaultQueryCache, restoreCPUCoreCount, restoreIndexWriterMaxDocs, runWithRestrictedPermissions, setIndexWriterMaxDocs, setupCPUCoreCount, setUpExecutorService, shutdownExecutorService, slowFileExists, usually, usually, wrapReader
-
Methods inherited from class org.junit.Assert
assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertThat, assertThat, assertThrows, assertThrows, assertTrue, assertTrue, fail, fail
-
-
-
-
Method Detail
-
init
public void init()
-
addRandomFields
protected void addRandomFields(Document doc)
Add random fields to the provided document.
-
testFieldConstructor
public void testFieldConstructor()
-
testFieldConstructorExceptions
public void testFieldConstructorExceptions()
-
testFieldSetValue
public void testFieldSetValue()
-
testIllegalDimChangeTwoDocs
public void testIllegalDimChangeTwoDocs() throws Exception
- Throws:
Exception
-
testIllegalSimilarityFunctionChange
public void testIllegalSimilarityFunctionChange() throws Exception
- Throws:
Exception
-
testIllegalDimChangeTwoWriters
public void testIllegalDimChangeTwoWriters() throws Exception
- Throws:
Exception
-
testIllegalSimilarityFunctionChangeTwoWriters
public void testIllegalSimilarityFunctionChangeTwoWriters() throws Exception
- Throws:
Exception
-
testAddIndexesDirectory01
public void testAddIndexesDirectory01() throws Exception
- Throws:
Exception
-
testIllegalDimChangeViaAddIndexesDirectory
public void testIllegalDimChangeViaAddIndexesDirectory() throws Exception
- Throws:
Exception
-
testIllegalSimilarityFunctionChangeViaAddIndexesDirectory
public void testIllegalSimilarityFunctionChangeViaAddIndexesDirectory() throws Exception
- Throws:
Exception
-
testIllegalDimChangeViaAddIndexesCodecReader
public void testIllegalDimChangeViaAddIndexesCodecReader() throws Exception
- Throws:
Exception
-
testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader
public void testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader() throws Exception
- Throws:
Exception
-
testIllegalDimChangeViaAddIndexesSlowCodecReader
public void testIllegalDimChangeViaAddIndexesSlowCodecReader() throws Exception
- Throws:
Exception
-
testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader
public void testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader() throws Exception
- Throws:
Exception
-
testIllegalMultipleValues
public void testIllegalMultipleValues() throws Exception
- Throws:
Exception
-
testIllegalDimensionTooLarge
public void testIllegalDimensionTooLarge() throws Exception
- Throws:
Exception
-
testInvalidKnnVectorFieldUsage
public void testInvalidKnnVectorFieldUsage()
-
testKnnVectorFieldMissingFromOneSegment
public void testKnnVectorFieldMissingFromOneSegment() throws Exception
- Throws:
Exception
-
randomSimilarity
protected VectorSimilarityFunction randomSimilarity()
-
randomVectorEncoding
protected VectorEncoding randomVectorEncoding()
This method is overrideable since old codec versions only supportVectorEncoding.FLOAT32
.
-
testIndexedValueNotAliased
public void testIndexedValueNotAliased() throws Exception
- Throws:
Exception
-
testIndexMultipleKnnVectorFields
public void testIndexMultipleKnnVectorFields() throws Exception
- Throws:
Exception
-
testRandom
public void testRandom() throws Exception
Index random vectors, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.- Throws:
Exception
-
testRandomBytes
public void testRandomBytes() throws Exception
Index random vectors as bytes, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.- Throws:
Exception
-
testSearchWithVisitedLimit
public void testSearchWithVisitedLimit() throws Exception
Tests whetherKnnVectorsReader.search(java.lang.String, float[], org.apache.lucene.search.KnnCollector, org.apache.lucene.util.Bits)
implementations obey the limit on the number of visited vectors. This test is a best-effort attempt to capture the right behavior, and isn't meant to define a strict requirement on behavior.- Throws:
Exception
-
testRandomWithUpdatesAndGraph
public void testRandomWithUpdatesAndGraph() throws Exception
Index random vectors, sometimes skipping documents, sometimes updating a document, sometimes merging, sometimes sorting the index, using an HNSW similarity function so as to also produce a graph, and verify that the expected values can be read back consistently.- Throws:
Exception
-
randomVector
protected float[] randomVector(int dim)
-
randomNormalizedVector
protected float[] randomNormalizedVector(int dim)
-
testCheckIndexIncludesVectors
public void testCheckIndexIncludesVectors() throws Exception
- Throws:
Exception
-
testSimilarityFunctionIdentifiers
public void testSimilarityFunctionIdentifiers()
-
testVectorEncodingOrdinals
public void testVectorEncodingOrdinals()
-
testVectorValuesReportCorrectDocs
public void testVectorValuesReportCorrectDocs() throws Exception
- Throws:
Exception
-
getCodec
protected abstract Codec getCodec()
Returns the codec to run tests against
-
getCreatedVersionMajor
protected int getCreatedVersionMajor()
Returns the major version that this codec is compatible with.
-
applyCreatedVersionMajor
protected final <D extends Directory> D applyCreatedVersionMajor(D d) throws IOException
Set the created version of the givenDirectory
and return it.- Throws:
IOException
-
setUp
public void setUp() throws Exception
Description copied from class:LuceneTestCase
For subclasses to override. Overrides must callsuper.setUp()
.- Overrides:
setUp
in classLuceneTestCase
- Throws:
Exception
-
tearDown
public void tearDown() throws Exception
Description copied from class:LuceneTestCase
For subclasses to override. Overrides must callsuper.tearDown()
.- Overrides:
tearDown
in classLuceneTestCase
- Throws:
Exception
-
excludedExtensionsFromByteCounts
protected Collection<String> excludedExtensionsFromByteCounts()
Return the list of extensions that should be excluded from byte counts when comparing indices that store the same content.
-
testMergeStability
public void testMergeStability() throws Exception
The purpose of this test is to make sure that bulk merge doesn't accumulate useless data over runs.- Throws:
Exception
-
mergeIsStable
protected boolean mergeIsStable()
-
testMultiClose
public void testMultiClose() throws IOException
Calls close multiple times on closeable codec apis- Throws:
IOException
-
testRandomExceptions
public void testRandomExceptions() throws Exception
Tests exception handling on write and openInput/createOutput- Throws:
Exception
-
shouldTestMergeInstance
protected boolean shouldTestMergeInstance()
Returnsfalse
if only the regular fields reader should be tested, andtrue
if only the merge instance should be tested.
-
maybeWrapWithMergingReader
protected final DirectoryReader maybeWrapWithMergingReader(DirectoryReader r) throws IOException
- Throws:
IOException
-
testCheckIntegrityReadsAllBytes
public void testCheckIntegrityReadsAllBytes() throws Exception
This test is a best effort at verifying that checkIntegrity doesn't miss any files. It tests that the combination of opening a reader and calling checkIntegrity on it reads all bytes of all files.- Throws:
Exception
-
-