TLDR Digamber is facing a cURL error 28 while bulk indexing 3 million posts. Kishore Nallan suggests checking post body size and possibly encountering a 503 status code.
I have 3 million posts, and i’m bulk indexing and during the indexing process i run tino cURL error 28,
The post body might be too large
The thing is it bails after different times
What's the size of the file being posted?
Additionally - the current code is doing a request to /health to check condition
that’s when it’s being bailed
Are you using the php client?
No we’re making cURL requests
So the thing is if you write too fast, Typesense can return 503 status code as a backpressure mechanism to let writes catch up
If you get a 503, backoff and retry after a short interval.
ok let me see if i can reduce the payload and test it out again.
Also - not sure if i’m getting a 503 error - will verify and check
Also can you let me know how to measure payload size here ?
The raw jsonl post you are passing. Max payload size is 10 G so I don't think you are running into that.
yeah - ok, that’s probably not it - no way i’m going over 10G
will do some more test
Indexed 3051 threads
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI