Resolving Excessive Document Addition and Import Errors
TLDR Rushil experienced problems with overloading and failing imports when adding documents. Kishore Nallan advised adjusting batch size, increasing client timeout, and checking import responses for errors. Rushil resolved the issues by following these suggestions.
1
Nov 01, 2022 (14 months ago)
Rushil
08:36 AMKishore Nallan
08:36 AMRushil
08:52 AMKishore Nallan
09:04 AMRushil
09:12 AMRushil
09:12 AMRushil
09:13 AMKishore Nallan
09:29 AMRushil
11:38 AMKishore Nallan
11:40 AM{"success": true}
or the actual error on why the document was not imported.Rushil
11:50 AMRushil
11:50 AMKishore Nallan
11:56 AMKishore Nallan
11:57 AMRushil
12:09 PMerror.importResults
from the raised exception to get a detailed error reason for each documentRushil
12:09 PMKishore Nallan
12:16 PMerror.importResults
to get the details. Doesn't that work?Rushil
12:32 PM1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Resolving Timeout Errors in Large Document Imports
Ken had issues with importing over 360k documents due to operation timeout. Jason advised increasing the timeout in the client library.
Resolving Timeout Issues in Bulk Document Imports
Aljosa encountered timeouts when importing 32k documents. Kishore Nallan advised increasing client side timeout. Bruno suggested import in chunks and a retry for individual chunks. Kishore Nallan mentioned the next release of Typesense will make imports more reliable.
Issues with Duplicate Entries in New Collections
Ivan reported an issue wherein duplicate entries appear in his database collection. This occurs sporadically when he deletes a collection, creates a new one with the same name, and adds the same entries. Kishore Nallan identified this as a race condition, suggesting a larger timeout value to avoid premature disconnection. Unable to increase the timeout, Ivan instead reduced his import chunks to 500 entries, which successfully eliminated the error.
Troubleshooting Write Timeouts in Typesense with Large CSVs
Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.