#community-help

Benchmarking Typesense Search Results

TLDR Narayan asked about the search algorithm in Typesense and how it compares to TF-IDF/BM25. Kishore Nallan explained that Typesense focuses on smaller documents and provided links to their test suites.

1mo

Join the chat

Oct 30, 2023 (1 month ago)

Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72

Narayan

05:05 AM

I am trying to benchmark search results from typesense against standard information retrieval datasets and compare it with baselines like BM25. Can someone point me to any documentation, if one exists, for how the search in typesense works under the hood? I could not find anything on the website and comments like these suggests the algorithm is a bit involved 😅

05:07

Narayan

05:07 AM

Also, are there a set of parameters such that search reduces to something very close to TF-IDF/BM25?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

05:12 AM

Most of the standard information retrieval benchmarks focus on searching of queries against large pieces of text. We focus primarily on searching on smaller documents, across multiple fields. In these cases, the term frequency is not a very useful metric. Therefore we do not consider term frequency for ranking. We instead consider the extent of overlap of tokens from the query with the text, typo count etc.

Narayan

05:24 AM

Understood. That would explain the poor results I got in my first run 😅 How do you quantitatively measure any changes to the algorithm then? And, by smaller documents I assume you mean things like names, chapter titles, news headlines and the likes and not stuff like product descriptions, 5-6 sentence paragraphs, correct?

Kishore Nallan

05:33 AM

We have our own test suites for checking ranking correctness.

By smaller docs, it includes description as well, but typically descriptions are never used standlone but weighted lower when compared to other fields like title etc. None of the MTEB type of benchmarks really reflect searches for these use cases.

Narayan

05:41 AM

Can you please point me to these tests if they exist in github? I agree with what you said, but as we look to move away from Elasticsearch and Manticore and completely commit to typesense - some sort of benchmarking for pure token based search (no vector/hybrid search component) on standard datasets would really help us take that leap with more confidence.

Kishore Nallan

05:45 AM

Sure, see here: https://github.com/typesense/typesense/blob/main/test/collection_specific_test.cpp#L1470

These types of tests are present throughout the test suite as we test ranking with various other subfeatures like filtering etc.

05:46

Kishore Nallan

05:46 AM

This test suite as well: https://github.com/typesense/typesense/blob/main/test/collection_test.cpp#L3131

Narayan

05:47 AM

thank you, I’ll take a look.

05:49

Narayan

05:49 AM

In any case, would you be able to add a section somewhere in the docs explaining briefly the search algorithm? 🙏

Kishore Nallan

05:51 AM

We have documented the high level components used for deriving the text match score.: https://typesense.org/docs/guide/ranking-and-relevance.html#text-match-score

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Understanding Indexing and Search-As-You-Type In Typesense

Steven had queries about indexing and search-as-you-type in Typesense. Jason clarified that bulk updates are faster and search-as-you-type is resource intensive but worth it. The discussion also included querying benchmarks and Typesense's drop_tokens_threshold parameter, with participation from bnfd.

28mo

Improving Typesense Query Performance

Jonathan queried about slower than expected typesense query performance. Jason and Kishore Nallan offered solutions and explanations. After a series of tests, Jonathan found other queries returned results quickly, indicating the issue was specific to the original query.

13mo

Optimizing Dataset of Podcast Feeds for a Searchable Database

Alexander seeks advice on optimizing a podcast database for search. Kishore Nallan suggests data size and stopwords impact RAM usage, and that benchmarking on 1M records would be useful. satish raises the potential need for vector searching. Both recommend feeding user activity data into ML models for relevancy ranking. Collaboration was suggested.

20mo

Resolving Typesense Search Issues

Conversation started by Maximilian about Typesense search behavior led to Users Kishore Nallan and Mike discussing and suggesting workaround, with Kishore Nallan promising an official solution soon. No final confirmation of resolution provided.

21mo

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

111

8mo