Issues with Indexing Large JSON File in Node
TLDR Alexander had issues with indexing a large JSON file in Node. Jason suggested the problem may be due to node/js limitations and proposed to reduce the batch size to around 5000.
Jul 12, 2021 (31 months ago)
Jul 13, 2021 (31 months ago)
Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate" Request #1626169731197: Sleeping for 0.1s and then retrying request... Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate" Request #1626169731197: Sleeping for 0.1s and then retrying request... RangeError: Too many properties to enumerate at Function.keys (<anonymous>) at ApiCall._callee$ (/home/alexander/Documents/dt-site-search/node_modules/typesense/lib/Typesense/ApiCall.js:155:46) at tryCatch (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:63:40) at Generator.invoke [as _invoke] (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:293:22) at Generator.next (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:118:21) at asyncGeneratorStep (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24) at _next (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
May be try reducing the batch size to say 5000 at a time
Indexed 3005 threads (79% resolved)
Handling Large JSON File for Typesense
Matt struggled with processing a large JSON file for Typesense. Kishore Nallan explained how to create a schema, convert to JSONL, and import the file. They also identified the necessary keys from the JSON.
Resolving JSONL File Import Issues in Python
Jon struggles importing a large JSONL file using Python, encountering decode errors and size restrictions. Kishore Nallan instructs to use curl for imports under 10GB, and references an update to the Python client which could more capably handle large imports.
Resolving Typesense Documents Import Error
Aljosa experienced an error while using typesense `documents().import()`, related to handling of large document arrays. Jason clarified that batch_size controls server-side batching, not client-side. He advised splitting arrays to address the issue and committed to elaborating its functionality in the docs. Aljosa proposed amending the TypeScript types to accommodate batch_size in the import options.