Bulk Import 50MB JSON Files Error - Timeout and Solutions
TLDR madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.
3
1
Nov 19, 2021 (24 months ago)
madhweep
11:09 AMRequest #1637320016147: Request to Node 0 failed due to "undefined Too many properties to enumerate"
Kishore Nallan
11:09 AMKishore Nallan
11:11 AMcurl
to import the full file. Typesense can support imports of several GBs of file directly, so this is just a JS client limitation at the moment.madhweep
11:28 AMKishore Nallan
11:41 AMcurl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -X POST --data-binary @/path/to/file.jsonl " "
The file needs to be in JSONL. Described here: https://typesense.org/docs/0.21.0/api/documents.html#import-a-jsonl-file
madhweep
12:23 PMmadhweep
12:23 PMKishore Nallan
01:58 PMJason
03:28 PMJason
03:28 PMmadhweep
05:05 PMJason
05:34 PMDaniel
05:49 PMJason
05:52 PMJason
05:52 PMDaniel
06:04 PMDaniel
06:05 PMJason
06:05 PM1
Jason
06:07 PMSo I'm wondering what's happening...
Jason
11:15 PM1
1
Nov 20, 2021 (24 months ago)
Daniel
12:36 PM1
Typesense
Indexed 2786 threads (79% resolved)
Similar Threads
Troubleshooting Write Timeouts in Typesense with Large CSVs
Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
Issues with Importing Typesense Collection to Different Server
Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.
Importing data to Typesense Cloud with JS client
Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.
Discussion on Document Inserting Speed and Process
David inquired about document insertion speed, and Jason provided reference values and recommended sending more documents per API call. Both David and Chetan acknowledged the suggestions, with David stating to report back on their experience.