Discussing Access to Lower-Level APIs to Access Index in Typesense
TLDR Janaka inquired if lower-level APIs can access index tokens in a document for similarity calculation in Typesense. Jason advised extracting keywords for a regular search, informing that indices are only stored in memory. The thread concluded with a discussion about prospective Typesense extensions.
1
Sep 29, 2021 (26 months ago)
Janaka
10:50 PMJanaka
10:52 PM
Jason
10:53 PMJason
10:53 PMIndices are only stored in memory
Janaka
10:56 PMJanaka
10:59 PMJanaka
11:01 PMJason
11:02 PMJanaka
11:02 PMJanaka
11:03 PM1
Jason
11:04 PMThat should help in your use-case as well (when we have it ๐ )
Oct 06, 2021 (25 months ago)
Janaka
05:27 PMIf so exposing the index via an API would allow both cases to be solved and not tightly couple the similarity algorithm to the core of Typesense. In theory that should also make it easier for other to collaborate on those algos ๐
Jason
08:28 PM
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Integrating Semantic Search with Typesense
Krish wants to integrate a semantic search functionality with typesense but struggles with the limitations. Kishore Nallan provides resources, clarifications and workarounds to the raised issues.
Optimizing Dataset of Podcast Feeds for a Searchable Database
Alexander seeks advice on optimizing a podcast database for search. Kishore Nallan suggests data size and stopwords impact RAM usage, and that benchmarking on 1M records would be useful. satish raises the potential need for vector searching. Both recommend feeding user activity data into ML models for relevancy ranking. Collaboration was suggested.
Discussing Document Indexing Speeds and Typesense Features
Thomas asks about the speed of indexing and associated factors. The conversation reveals that larger batch sizes and NVMe disk usage can improve speed, but the index size is limited by RAM. Jason shares plans on supporting nested fields, and they explore a solution for products in multiple categories and catalogs.
Querying and Indexing Multiple Elements Issues
Krish queried fields with multiple elements, which Kishore Nallan suggested checking `drop_tokens_threshold`. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.
Understanding Indexing and Search-As-You-Type In Typesense
Steven had queries about indexing and search-as-you-type in Typesense. Jason clarified that bulk updates are faster and search-as-you-type is resource intensive but worth it. The discussion also included querying benchmarks and Typesense's drop_tokens_threshold parameter, with participation from bnfd.