#community-help

Importing data to Typesense Cloud with JS client

TLDR Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.

23mo

Join the chat

Dec 06, 2021 (23 months ago)

Photo of md5-b12c0749a9dbaefc03cb5817250a595d

Zaiste

12:39 PM

I’m playing with the Typesense Cloud. I have about 0.5 GB of data to import. It goes well for a few seconds, but then I’m getting ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging. I’m using the JS client. The connectionTimeoutSeconds is set to 10 . I tried different values but it didn’t work. Is there something I could change to make the import work?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

12:41 PM

👋 Are you importing via batches?

12:42

Kishore Nallan

12:42 PM

So what's happening here is some backpressure. If you try to send batches of imports or writes too fast and Typesense is not able to keep up (especially on a small instance like this one), then it will return 503: you should handle this, add a sleep() and try again.

12:43

Kishore Nallan

12:43 PM

This is done so that the search performance is not affected by too much writes slamming the server at once.

12:44

Kishore Nallan

12:44 PM

You can just import all documents one go either via the JS client itself or via curl. We can handle several GBs of uploads via curl. The backpressure exists to protect searches from getting affected adversely.

Zaiste

01:07 PM

Thanks! I’m importing in batches of 100 elements (the batchSize option to import).

01:08

Zaiste

01:08 PM

I’ll play with it a bit more. Thanks for the suggestions

Kishore Nallan

01:08 PM

If you use curl, you can just send it one shot.

Zaiste

01:10 PM

OK! I’ll try with curl - if I understand correctly the JS library is just a wrapper around your HTTP API right?

Kishore Nallan

01:10 PM

Yes, correct. But the JS client has a problem with sending too many docs at once, because of: https://github.com/typesense/typesense-js/issues/81

Zaiste

01:11 PM

ahh! that makes sense

Kishore Nallan

01:11 PM

So curl is preferred, but this is unrelated to the other 503 error you saw, which is server side backpressure as I explained earlier.

Zaiste

01:12 PM

Thank you for your help!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2786 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Bulk Import 50MB JSON Files Error - Timeout and Solutions

madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.

24mo

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

30mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

8mo

Typesense Server Bulk Import/Upsert Issue Resolved

Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.

5mo

Troubleshooting Indexing Duration in Typesense Import

Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.

13mo