Bulk Indexing Issue with 3 Million Posts
TLDR Digamber is facing a cURL error 28 while bulk indexing 3 million posts. Kishore Nallan suggests checking post body size and possibly encountering a 503 status code.
Kishore Nallan
Thu, 09 Mar 2023 11:50:22 UTCThe post body might be too large
Digamber
Thu, 09 Mar 2023 11:51:48 UTCThe thing is it bails after different times
Kishore Nallan
Thu, 09 Mar 2023 11:52:24 UTCWhat's the size of the file being posted?
Digamber
Thu, 09 Mar 2023 11:52:28 UTCAdditionally - the current code is doing a request to /health to check condition
Digamber
Thu, 09 Mar 2023 11:52:51 UTCthat’s when it’s being bailed
Kishore Nallan
Thu, 09 Mar 2023 11:53:09 UTCAre you using the php client?
Digamber
Thu, 09 Mar 2023 11:53:29 UTCNo we’re making cURL requests
Kishore Nallan
Thu, 09 Mar 2023 11:53:56 UTCSo the thing is if you write too fast, Typesense can return 503 status code as a backpressure mechanism to let writes catch up
Kishore Nallan
Thu, 09 Mar 2023 11:54:21 UTCIf you get a 503, backoff and retry after a short interval.
Digamber
Thu, 09 Mar 2023 11:57:32 UTCok let me see if i can reduce the payload and test it out again. Also - not sure if i’m getting a 503 error - will verify and check
Digamber
Thu, 09 Mar 2023 12:06:33 UTCAlso can you let me know how to measure payload size here ?
Kishore Nallan
Thu, 09 Mar 2023 12:07:17 UTCThe raw jsonl post you are passing. Max payload size is 10 G so I don't think you are running into that.
Digamber
Thu, 09 Mar 2023 12:07:43 UTCyeah - ok, that’s probably not it - no way i’m going over 10G
Digamber
Thu, 09 Mar 2023 12:07:53 UTCwill do some more test
Typesense
Indexed 3051 threads
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
View in App
Digamber
Thu, 09 Mar 2023 11:49:10 UTCHi Guys, I have 3 million posts, and i’m bulk indexing and during the indexing process i run tino cURL error 28,