Class Lucene99HnswVectorsFormat
java.lang.Object
org.apache.lucene.codecs.KnnVectorsFormat
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat
- All Implemented Interfaces:
NamedSPILoader.NamedSPI
Lucene 9.9 vector format, which encodes numeric vector values into an associated graph connecting
the documents having values. The graph is used to power HNSW search. The format consists of two
files, and requires a
FlatVectorsFormat to store the actual vectors:
.vex (vector index)
Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:
- For each level:
- For each node:
- [vint] the number of neighbor nodes
- array[vint] the delta encoded neighbor ordinals
- For each node:
- After all levels are encoded, memory offsets for each node's neighbor nodes are appended to
the end of the file. The offsets are encoded by
DirectMonotonicWriter.
.vem (vector metadata) file
For each field:
- [int32] field number
- [int32] vector similarity function ordinal
- [vlong] offset to this field's index in the .vex file
- [vlong] length of this field's index data, in bytes
- [vint] dimension of this field's vectors
- [int] the number of documents having values for this field
- [vint] the maximum number of connections (neighbours) that each node can have
- [vint] number of levels in the graph
- Graph nodes by level. For each level
- [vint] the number of nodes on this level
- array[vint] for levels greater than 0 list of nodes on this level, stored as the level 0th delta encoded nodes' ordinals.
- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final intDefault number of the size of the queue maintained while searching during a graph construction.static final intDefault number of maximum connections per nodestatic final intDefault to use single thread mergestatic final intThe maximum size of the queue to maintain while searching during graph construction.static final intA maximum configurable maximum max conn.static final intstatic final intFields inherited from class org.apache.lucene.codecs.KnnVectorsFormat
DEFAULT_MAX_DIMENSIONS, EMPTY -
Constructor Summary
ConstructorsConstructorDescriptionConstructs a format using default graph construction parametersLucene99HnswVectorsFormat(int maxConn, int beamWidth) Constructs a format using the given graph construction parameters.Lucene99HnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec) Constructs a format using the given graph construction parameters and scalar quantization. -
Method Summary
Modifier and TypeMethodDescriptionfieldsReader(SegmentReadState state) Returns aKnnVectorsReaderto read the vectors from the index.fieldsWriter(SegmentWriteState state) Returns aKnnVectorsWriterto write the vectors to the index.intgetMaxDimensions(String fieldName) Returns the maximum number of vector dimensions supported by this codec for the given field nametoString()Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat
availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat
-
Field Details
-
VERSION_START
public static final int VERSION_START- See Also:
-
VERSION_CURRENT
public static final int VERSION_CURRENT- See Also:
-
MAXIMUM_MAX_CONN
public static final int MAXIMUM_MAX_CONNA maximum configurable maximum max conn.NOTE: We eagerly populate `float[MAX_CONN*2]` and `int[MAX_CONN*2]`, so exceptionally large numbers here will use an inordinate amount of heap
- See Also:
-
DEFAULT_MAX_CONN
public static final int DEFAULT_MAX_CONNDefault number of maximum connections per node- See Also:
-
MAXIMUM_BEAM_WIDTH
public static final int MAXIMUM_BEAM_WIDTHThe maximum size of the queue to maintain while searching during graph construction. This maximum value preserves the ratio of the `DEFAULT_BEAM_WIDTH`/`DEFAULT_MAX_CONN` (i.e. `6.25 * 16 = 3200`).- See Also:
-
DEFAULT_BEAM_WIDTH
public static final int DEFAULT_BEAM_WIDTHDefault number of the size of the queue maintained while searching during a graph construction.- See Also:
-
DEFAULT_NUM_MERGE_WORKER
public static final int DEFAULT_NUM_MERGE_WORKERDefault to use single thread merge- See Also:
-
-
Constructor Details
-
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat()Constructs a format using default graph construction parameters -
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat(int maxConn, int beamWidth) Constructs a format using the given graph construction parameters.- Parameters:
maxConn- the maximum number of connections to a node in the HNSW graphbeamWidth- the size of the queue maintained during graph construction.
-
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec) Constructs a format using the given graph construction parameters and scalar quantization.- Parameters:
maxConn- the maximum number of connections to a node in the HNSW graphbeamWidth- the size of the queue maintained during graph construction.numMergeWorkers- number of workers (threads) that will be used when doing merge. If larger than 1, a non-nullExecutorServicemust be passed as mergeExecmergeExec- theExecutorServicethat will be used by ALL vector writers that are generated by this format to do the merge. If null, the configuredMergeScheduler.getIntraMergeExecutor(MergePolicy.OneMerge)is used.
-
-
Method Details
-
fieldsWriter
Description copied from class:KnnVectorsFormatReturns aKnnVectorsWriterto write the vectors to the index.- Specified by:
fieldsWriterin classKnnVectorsFormat- Throws:
IOException
-
fieldsReader
Description copied from class:KnnVectorsFormatReturns aKnnVectorsReaderto read the vectors from the index.- Specified by:
fieldsReaderin classKnnVectorsFormat- Throws:
IOException
-
getMaxDimensions
Description copied from class:KnnVectorsFormatReturns the maximum number of vector dimensions supported by this codec for the given field nameCodecs implement this method to specify the maximum number of dimensions they support.
- Specified by:
getMaxDimensionsin classKnnVectorsFormat- Parameters:
fieldName- the field name- Returns:
- the maximum number of vector dimensions.
-
toString
-