Hi @Kishore Nallan, how should we make the JSONL file in order to implement hybrid search (auto embedding) ? Right now, I have fields to return in my JSONL documents,

id

,

name

,

category

. Do I explicitly need to mention a field named

embedding

? I have added this

embedding

field in the step while creating the collection:

{
  "name": "embedding",
  "type": "float[]",
  "embed": {
      "from": ["name", "category"],
      "model_config": {"model_name": "all-MiniLM-L12-v2"},
  },
}

But, running this with and without the embedding field in JSONL documents give me the error:

[Errno 400] Model file not found

Should i have the model stored in some directory within my project so that typesense can access it?

I see syntax errors in that schema...

all-MiniLM-L12-v2

should be

ts/all-MiniLM-L12-v2

Here's a step-by-step guide you should be able to copy-paste: https://typesense.org/docs/guide/tips-for-searching-common-types-of-data.html#long-pieces-of-text

Hey @Jason Bosco Adding

ts/

didn't help, still the same error.

Could you share a full curl command that creates the schema, that throws the error?

Here is the collection schema:

{
  "name": "TEST",
  "fields": [
    {
      "name": "testId",
      "type": "string",
      "facet": true
    },
    {
      "name": "name",
      "type": "string",
      "facet": true
    },
    {
      "name": "category",
      "type": "string",
      "facet": true
    },
    {
      "name": "userId",
      "type": "string",
      "facet": true,
      "optional": true
    },
    {
      "name": "embedding",
      "type": "float[]",
      "embed": {
        "from": [
          "name",
          "category"
        ],
        "model_config": {
          "model_name": "ts/all-MiniLM-L12-v2"
        }
      }
    }
  ]
}

I just tried running it locally and it worked without any issues

Yes, on it

Hmm no, you don't need a GPU build

I'm running typesense service in linux

Could you try using

0.26.0.rc62

of Typesense Server?

The error wasn't there in

0.26.0.rc62

but i received an empty response. ### The response:

{
  "results": [
    {
      "facet_counts": [],
      "found": 0,
      "hits": [],
      "out_of": 0,
      "page": 1,
      "request_params": {
        "collection_name": "TEST",
        "first_q": "test",
        "per_page": 10,
        "q": "test"
      },
      "search_cutoff": false,
      "search_time_ms": 8
    }
  ]
}

### The collection schema

{
  "name": "TEST",
  "fields": [
    {
      "name": "testId",
      "type": "string",
      "facet": true
    },
    {
      "name": "name",
      "type": "string",
      "facet": true
    },
    {
      "name": "category",
      "type": "string",
      "facet": true
    },
    {
      "name": "userId",
      "type": "string",
      "facet": true,
      "optional": true
    },
    {
      "name": "embedding",
      "type": "float[]",
      "embed": {
        "from": [
          "name",
          "category"
        ],
        "model_config": {
          "model_name": "ts/all-MiniLM-L12-v2"
        }
      }
    }
  ]
}

### SEARCH REQUESTS

{
  "searches": [
    {
      "collection": "TEST",
      "q": "test",
      "sort_by": "_vector_distance:asc,_text_match:desc",
      "prioritize_token_position": "true"
    }
  ]
}

### COMMON SEARCH PARAMS

{
  "query_by": "embedding,name,category"
}

### Sample Document from JSONL File

{
  "testId": "1",
  "name": "Test String",
  "userId": "10",
  "category": "test-category",
  "embedding": []
}

Yes that means no doc has been indexed

Are my documents correct?

Check if you get error during indexing.

Here's that:

['Field `embedding` contains an invalid embedding.']

Yeah so some invalid values

I'm not getting it. I need to keep the embedding field empty, right? Typesense will automatically populate that field.

Don't send that field at all if you are using auto embedding

Worked. for both, RC62 and

0.25.2

You can set the alpha parameter to give more weightage to keyword matches. There's also a parameter for fetching only records with min vector similarity value.

Can you send me the reference links for those?

Everything is documented on our docs page on vector search.

Can we specify a threshold for rank fusion score? Also sorting based on the same param?

_text_match

is already rank fusion score in hybrid search so you can just sort on

_text_match:desc

They are different. DIdn't get that part Here's the response

{
  "results": [
    {
      "facet_counts": [],
      "found": 2,
      "hits": [
        {
          "document": {
            "category": "test-category",
            "id": "2",
            "name": "Gift laptop to your friend",
            "testId": "1011",
            "userId": "15"
          },
          "highlight": {
            "name": {
              "matched_tokens": [
                "laptop"
              ],
              "snippet": "Gift <mark>laptop</mark> to your friend"
            }
          },
          "highlights": [
            {
              "field": "name",
              "matched_tokens": [
                "laptop"
              ],
              "snippet": "Gift <mark>laptop</mark> to your friend"
            }
          ],
          "hybrid_search_info": {
            "rank_fusion_score": 1
          },
          "text_match": 1060320051,
          "text_match_info": {
            "best_field_score": "517734",
            "best_field_weight": 102,
            "fields_matched": 3,
            "score": "1060320051",
            "tokens_matched": 0
          },
          "vector_distance": 0.39046764373779297
        },
        {
          "document": {
            "category": "test-category",
            "id": "3",
            "name": "Gift a gaming computer to your friend",
            "testId": "10111",
            "userId": "15"
          },
          "highlight": {},
          "highlights": [],
          "hybrid_search_info": {
            "rank_fusion_score": 0.15000000596046448
          },
          "text_match": 0,
          "text_match_info": {
            "best_field_score": "0",
            "best_field_weight": 0,
            "fields_matched": 0,
            "score": "0",
            "tokens_matched": 0
          },
          "vector_distance": 0.5736579895019531
        }
      ],
      "out_of": 15,
      "page": 1,
      "request_params": {
        "collection_name": "TEST",
        "per_page": 10,
        "q": "laptop"
      },
      "search_cutoff": false,
      "search_time_ms": 5
    }
  ]
}

In hybrid search when you sort on

_text_match

you are actually sorting on fusion score.

Okay

Could you run the same set of curl commands I shared with you yesterday, on

0.26.0.rc62

and share the output of each command and also the Typesenese logs from the beginning of the process start?