Implementing Typesense Updates with JSONL Import and Aliases
TLDR Ken is building a search solution for a website using Typesense. They consulted Kishore Nallan about the implementation of updates using JSONL import and aliases and how to know when the new collection is indexed and ready. Measures, such as dividing large imports into small batches, were suggested to address the issue.
2
Feb 15, 2022 (21 months ago)
Ken
03:31 PM1
Ken
03:31 PMhttps://typesense.org/docs/0.22.2/api/collection-alias.html
Ken
03:32 PMdocuments/import
, toggle collection alias to point to new collection)Kishore Nallan
03:33 PMThe alias-based approach works well if your content refresh happens only periodically so you can just a full refresh of the index.
1
Kishore Nallan
03:34 PMupsert
or update
action of the import end-point.Ken
03:35 PMGET /collections/foo
or is there a callback that can notify me when the new collection is ready?Ken
03:37 PMKishore Nallan
03:52 PMKen
03:54 PMKishore Nallan
03:59 PMKen
04:03 PMGET collection/foo
requests as well as search
requests.Kishore Nallan
04:16 PMTo prevent this from happening, split your imports into batches that are not too large. We have some work planned to make the import endpoint automatically slow down for large uploads which should make this easier.
Ken
04:35 PMFeb 16, 2022 (21 months ago)
Kishore Nallan
03:31 AMKishore Nallan
03:32 AMKen
09:19 PMTypesense
Indexed 2786 threads (79% resolved)
Similar Threads
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.