Implementing Typesense Updates with JSONL Import and Aliases
TLDR Ken is building a search solution for a website using Typesense. They consulted Kishore Nallan about the implementation of updates using JSONL import and aliases and how to know when the new collection is indexed and ready. Measures, such as dividing large imports into small batches, were suggested to address the issue.
Feb 15, 2022 (23 months ago)
documents/import, toggle collection alias to point to new collection)
Kishore Nallan03:33 PM
The alias-based approach works well if your content refresh happens only periodically so you can just a full refresh of the index.
Kishore Nallan03:34 PM
updateaction of the import end-point.
GET /collections/fooor is there a callback that can notify me when the new collection is ready?
Kishore Nallan03:52 PM
Kishore Nallan03:59 PM
GET collection/foorequests as well as
Kishore Nallan04:16 PM
To prevent this from happening, split your imports into batches that are not too large. We have some work planned to make the import endpoint automatically slow down for large uploads which should make this easier.
Feb 16, 2022 (23 months ago)
Kishore Nallan03:31 AM
Kishore Nallan03:32 AM
Indexed 3015 threads (79% resolved)
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
Implementing Semantic Search with Typesense
Erik sought advice for semantic search implementation in Typesense and raised issues around slow document import and excessive latency. Upon implementing advice from Kishore Nallan to try different models, Erik reported faster times, ultimately deciding to rate-limit imports.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.