TLDR Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.
I’m playing with the Typesense Cloud. I have about 0.5 GB of data to import. It goes well for a few seconds, but then I’m getting `ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging`. I’m using the JS client. The `connectionTimeoutSeconds` is set to `10` . I tried different values but it didn’t work. Is there something I could change to make the import work?
:wave: Are you importing via batches?
So what's happening here is some backpressure. If you try to send batches of imports or writes too fast and Typesense is not able to keep up (especially on a small instance like this one), then it will return 503: you should handle this, add a sleep() and try again.
This is done so that the search performance is not affected by too much writes slamming the server at once.
You can just import all documents one go either via the JS client itself or via `curl`. We can handle several GBs of uploads via curl. The backpressure exists to protect searches from getting affected adversely.
Thanks! I’m importing in batches of `100` elements (the `batchSize` option to `import`).
I’ll play with it a bit more. Thanks for the suggestions
If you use curl, you can just send it one shot.
OK! I’ll try with `curl` - if I understand correctly the JS library is just a wrapper around your HTTP API right?
Yes, correct. But the JS client has a problem with sending too many docs at once, because of:
ahh! that makes sense
So curl is preferred, but this is unrelated to the other 503 error you saw, which is server side backpressure as I explained earlier.
Thank you for your help!
Indexed 3051 threads
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI