#community-help

Importing data to Typesense Cloud with JS client

TLDR Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.

Powered by Struct AI

3

1

13
23mo
Solved
Join the chat
Dec 06, 2021 (23 months ago)
Zaiste
Photo of md5-b12c0749a9dbaefc03cb5817250a595d
Zaiste
12:39 PM
Iโ€™m playing with the Typesense Cloud. I have about 0.5 GB of data to import. It goes well for a few seconds, but then Iโ€™m getting ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging. Iโ€™m using the JS client. The connectionTimeoutSeconds is set to 10 . I tried different values but it didnโ€™t work. Is there something I could change to make the import work?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:41 PM
๐Ÿ‘‹ Are you importing via batches?
12:42
Kishore Nallan
12:42 PM
So what's happening here is some backpressure. If you try to send batches of imports or writes too fast and Typesense is not able to keep up (especially on a small instance like this one), then it will return 503: you should handle this, add a sleep() and try again.

1

12:43
Kishore Nallan
12:43 PM
This is done so that the search performance is not affected by too much writes slamming the server at once.
12:44
Kishore Nallan
12:44 PM
You can just import all documents one go either via the JS client itself or via curl. We can handle several GBs of uploads via curl. The backpressure exists to protect searches from getting affected adversely.
Zaiste
Photo of md5-b12c0749a9dbaefc03cb5817250a595d
Zaiste
01:07 PM
Thanks! Iโ€™m importing in batches of 100 elements (the batchSize option to import).
01:08
Zaiste
01:08 PM
Iโ€™ll play with it a bit more. Thanks for the suggestions
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:08 PM
If you use curl, you can just send it one shot.

1

Zaiste
Photo of md5-b12c0749a9dbaefc03cb5817250a595d
Zaiste
01:10 PM
OK! Iโ€™ll try with curl - if I understand correctly the JS library is just a wrapper around your HTTP API right?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:10 PM
Yes, correct. But the JS client has a problem with sending too many docs at once, because of: https://github.com/typesense/typesense-js/issues/81
Zaiste
Photo of md5-b12c0749a9dbaefc03cb5817250a595d
Zaiste
01:11 PM
ahh! that makes sense
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:11 PM
So curl is preferred, but this is unrelated to the other 503 error you saw, which is server side backpressure as I explained earlier.

1

Zaiste
Photo of md5-b12c0749a9dbaefc03cb5817250a595d
Zaiste
01:12 PM
Thank you for your help!

1