#community-help

Issues with Indexing Large JSON File in Node

TLDR Alexander had issues with indexing a large JSON file in Node. Jason suggested the problem may be due to node/js limitations and proposed to reduce the batch size to around 5000.

31mo

Join the chat

Jul 12, 2021 (31 months ago)

Photo of md5-faf0fdba0b6739a6706f05c15b6738c6

Alexander

04:25 PM

What is the maximum size of json file possible (in terms of number of documents) that can be indexed at a time? Have a file that holds a quarter of a million records and I'm getting an error: Request to Node 0 failed due to "undefined Too many properties to enumerate".

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

04:37 PM

Typesense itself doesn’t have a limit - it’s dependent on RAM available. But in this case, that error is coming from JavaScript / Node

04:38

Jason

04:38 PM

You want to read the file in smaller batches and import them in smaller chunks

Jul 13, 2021 (31 months ago)

Alexander

11:57 AM

Jason Thank you. I am still having trouble indexing the data set. The problem occurs when the number of documents in the json array goes over 12857. In the case of a memory bottleneck I tried increasing the allowed memory usage for node however that did not solve the problem. Here is the exact error message that I am getting:

Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate"
Request #1626169731197: Sleeping for 0.1s and then retrying request...
Request #1626169731197: Request to Node 0 failed due to "undefined Too many properties to enumerate"
Request #1626169731197: Sleeping for 0.1s and then retrying request...
RangeError: Too many properties to enumerate
    at Function.keys (<anonymous>)
    at ApiCall._callee$ (/home/alexander/Documents/dt-site-search/node_modules/typesense/lib/Typesense/ApiCall.js:155:46)
    at tryCatch (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:63:40)
    at Generator.invoke [as _invoke] (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:293:22)
    at Generator.next (/home/alexander/Documents/dt-site-search/node_modules/regenerator-runtime/runtime.js:118:21)
    at asyncGeneratorStep (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/home/alexander/Documents/dt-site-search/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)

Jason

03:27 PM

This is a node / js limitation, this thread has some options: https://stackoverflow.com/questions/9282869/are-there-limits-to-the-number-of-properties-in-a-javascript-object

May be try reducing the batch size to say 5000 at a time

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Handling Large JSON File for Typesense

Matt struggled with processing a large JSON file for Typesense. Kishore Nallan explained how to create a schema, convert to JSONL, and import the file. They also identified the necessary keys from the JSON.

8mo

Resolving JSONL File Import Issues in Python

Jon struggles importing a large JSONL file using Python, encountering decode errors and size restrictions. Kishore Nallan instructs to use curl for imports under 10GB, and references an update to the Python client which could more capably handle large imports.

11mo

Resolving Typesense Documents Import Error

Aljosa experienced an error while using typesense `documents().import()`, related to handling of large document arrays. Jason clarified that batch_size controls server-side batching, not client-side. He advised splitting arrays to address the issue and committed to elaborating its functionality in the docs. Aljosa proposed amending the TypeScript types to accommodate batch_size in the import options.

26mo