Vector Search

A vector search compares vectors that you index in your IDOL documents with a vector that you use to query. A vector is an array of floating point numbers that represents an object, which might be text or an image. You can generate vectors by using many different models, and index them into IDOL in VectorType fields.

Configure Vector Search

To perform vector comparisons in queries, you must configure VectorType and index your documents with the vector values in these fields.

To configure vector fields

  1. Open the IDOL Content component configuration file in a text editor.

  2. In the [FieldProcessing] section of the IDOL Content component configuration file, set up a new field process for your vector fields. For example:

    [FieldProcessing]
    0=VectorFields
  3. Create a section for the synonym process you listed.

    • Set Property to the name of the property configuration section for the process. The property that you create must not have the same name as the process.

    • Set PropertyFieldCSVs to the list of fields that you want to associate with the process.

      To identify the fields, use the format /FieldName to match root-level fields, */FieldName to match all fields except root-level, or /Path/FieldName to match fields that the specified path points to.

    For example:

    [VectorFields]
    Property=VectorProperties
    PropertyFieldCSVs=*/VECTOR*

    In this example, Content indexes values in the VECTOR field as vectors for vector distance comparisons.

  4. Create a section for the property. In this section, set VectorType to True.

    [VectorProperties]
    VectorType=True
  5. (Optional) Change the distance metric to use for your vector comparisons, by adding or modifying the value of the DistanceMetric configuration parameter. For example:

    [VectorIndex]
    DistanceMetric=L2

    NOTE: The value that you need to use depends on the vector model that you use to generate your vectors. To find out what distance metric to use, refer to the documentation for your vector model.

  6. (Optional) If you already have documents in your IDOL index, you can configure IDOL to regenerate the vector index for these documents by using the RegenerateVectorIndex configuration parameter.

    [Server]
    ...
    RegenerateVectorIndex=True

    TIP: Alternatively, you can regenerate the vector index by using the DREREGENERATE index action.

  7. Save and close the configuration file.

  8. Restart the IDOL Content component for your changes to take effect.

Generate Vectors and Index Documents

After you configure vectors in the IDOL Content component, you can index documents that contain these vector fields.

You can generate vectors by using a third-party model. You then add the vector to a field in your IDOL documents.

A vector in a document is a comma-separated list of floating point values. For example, in an IDX field, you might have the following value:

#DREFIELD VECTOR="0.24275297, 0.6721721, 0.38784778, 0.9476961, 0.7166045, 0.91357803, 0.1593279, 0.8701281, 0.5187642 , 0.6577334 , 0.60528547, 0.83356684, 0.33284906, 0.0672832 , 0.957869  , 0.33899164, 0.8574585 , 0.08680364, 0.16952382, 0.07177012, 0.6952732 , 0.9542396 , 0.62435275, 0.9369151, 0.75291044, 0.6654858, 0.9690549, 0.96977615, 0.93315464, 0.36870354, 0.77450484, 0.8728181" 

All vectors in a particular field name must have the same number of dimensions (the previous example has 32 dimensions).

A document can contain multiple vectors with the same field name.

Send Vector Queries

To use vector search, you use the VECTOR operator in the Query action Text parameter.

The VECTOR operator allows you to search for vectors in your documents that are similar to the vector that you provide. The syntax for the VECTOR operator is:

action=Query&text=VECTOR{v1,v2,...,vn:maxdist}[:maxneighbors]:fieldname

For more information about this syntax, see Vector Search.

You can also use vectors for the Suggest action, to suggest documents that contain vectors similar to the vectors in the document you provide. In this case, you must set the UseVectors action parameter to True. For example:

action=Suggest&ID=200&UseVectors=True