Implementing Typesense Updates with JSONL Import and Aliases
TLDR Ken is building a search solution for a website using Typesense. They consulted Kishore Nallan about the implementation of updates using JSONL import and aliases and how to know when the new collection is indexed and ready. Measures, such as dividing large imports into small batches, were suggested to address the issue.
2
Feb 15, 2022 (23 months ago)
Ken
03:31 PM1
Ken
03:31 PMhttps://typesense.org/docs/0.22.2/api/collection-alias.html
Ken
03:32 PMdocuments/import, toggle collection alias to point to new collection)
Kishore Nallan
03:33 PMThe alias-based approach works well if your content refresh happens only periodically so you can just a full refresh of the index.
1
Kishore Nallan
03:34 PMupsert or update action of the import end-point.Ken
03:35 PMGET /collections/foo or is there a callback that can notify me when the new collection is ready?Ken
03:37 PMKishore Nallan
03:52 PMKen
03:54 PMKishore Nallan
03:59 PMKen
04:03 PMGET collection/foo requests as well as search requests.Kishore Nallan
04:16 PMTo prevent this from happening, split your imports into batches that are not too large. We have some work planned to make the import endpoint automatically slow down for large uploads which should make this easier.
Ken
04:35 PMFeb 16, 2022 (23 months ago)
Kishore Nallan
03:31 AMKishore Nallan
03:32 AMKen
09:19 PM
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
Implementing Semantic Search with Typesense
Erik sought advice for semantic search implementation in Typesense and raised issues around slow document import and excessive latency. Upon implementing advice from Kishore Nallan to try different models, Erik reported faster times, ultimately deciding to rate-limit imports.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.