#community-help

Issues with Indexing Large JSON File in Node

TLDR Alexander had issues with indexing a large JSON file in Node. Jason suggested the problem may be due to node/js limitations and proposed to reduce the batch size to around 5000.

Powered by Struct AI
Jul 12, 2021 (31 months ago)
Alexander
Photo of md5-faf0fdba0b6739a6706f05c15b6738c6
Alexander
04:25 PM
What is the maximum size of json file possible (in terms of number of documents) that can be indexed at a time? Have a file that holds a quarter of a million records and I'm getting an error: Request to Node 0 failed due to "undefined Too many properties to enumerate".
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:37 PM
Typesense itself doesn’t have a limit - it’s dependent on RAM available. But in this case, that error is coming from JavaScript / Node
04:38
Jason
04:38 PM
You want to read the file in smaller batches and import them in smaller chunks
Jul 13, 2021 (31 months ago)
Alexander
Photo of md5-faf0fdba0b6739a6706f05c15b6738c6
Alexander
11:57 AM
Jason Thank you. I am still having trouble indexing the data set. The problem occurs when the number of documents in the json array goes over 12857. In the case of a memory bottleneck I tried increasing the allowed memory usage for node however that did not solve the problem. Here is the exact error message that I am getting:
Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate"
Request #1626169731197: Sleeping for 0.1s and then retrying request...
Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate"
Request #1626169731197: Sleeping for 0.1s and then retrying request...
RangeError: Too many properties to enumerate
    at Function.keys (<anonymous>)
    at ApiCall._callee$ (/home/alexander/Documents/dt-site-search/node_modules/typesense/lib/Typesense/ApiCall.js:155:46)
    at tryCatch (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:63:40)
    at Generator.invoke [as _invoke] (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:293:22)
    at Generator.next (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:118:21)
    at asyncGeneratorStep (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:27 PM
This is a node / js limitation, this thread has some options: https://stackoverflow.com/questions/9282869/are-there-limits-to-the-number-of-properties-in-a-javascript-object

May be try reducing the batch size to say 5000 at a time