Errors in Batch Import with Typesense and OpenAI API
TLDR Gustavo encountered errors when importing documents into a collection. After discussion with Jason, it was concluded that the issue stemmed from OpenAI API's handling of batch requests with problematic documents, and improvements to Typesense's error messages and handling were suggested.
Powered by Struct AI
4
1
64
3mo
Solved
Jun 15, 2023 (3 months ago)
Gustavo
Gustavo
08:50 PMWhen importing documents into a collection, after about 2500 successful imports, I started to get an error saying
The server had an error while processing your request. Sorry about that!. I suspect the error has some relation with the fact the collection has a built-in embedding field. Cluster: v601y2x3upjea4tipJason
Jason
08:51 PMHmm, that doesn’t seem like an error from Typesense…
08:51
Jason
08:51 PMAre you using a remote embedding service?
Gustavo
Gustavo
08:51 PMI'm importing in batches of 100. Connection timeout is set to 3 minutes. Retry interval is set to 5 seconds.
08:51
Gustavo
08:51 PMYes, OpenAI.
Jason
Jason
08:52 PMCould you share the full JSON message that Typesense returns?
08:52
Jason
08:52 PMI suspect it’s from OpenAI’s API that we just proxy through
Gustavo
Gustavo
08:53 PM"error": {
"message": "The server had an error while processing your request. Sorry about that!",
"type": "server_error",
"param": null,
"code": null
}08:54
Gustavo
08:54 PMYeah, I think it's from OpenAI. Maybe exceeding the rate limit or something.
Jason
Jason
08:54 PMWe should probably indicate where the error is originating from, in cases like this
08:55
Jason
08:55 PMConfirming that it’s an OpenAI message: https://community.openai.com/t/openai-api-error-the-server-had-an-error-while-processing-your-request-sorry-about-that/53263
Gustavo
Gustavo
08:55 PMAnd maybe also retry the request to OpenAI's API when it makes sense if it's not already retrying.
Jason
Jason
08:56 PMWe didn’t add a retry built-in to Typesense for remote services, to prevent any potential (billing) surprises. So if you see an error message in the API response from Typesense, you want to retry the import on those docs
Gustavo
Gustavo
08:57 PMMakes sense
08:57
Gustavo
08:57 PMAlthough...
08:58
Gustavo
08:58 PMImporting with
action: upsert doesn't work because of that error where Typesense sends an empty string to OpenAI.08:58
Gustavo
08:58 PMSo I guess I'll have to retry with
action: create and just ignore errors saying the document already exists.Jason
Jason
08:59 PMYeah… For now 😞
09:00
Jason
09:00 PMWe’re just about to start addressing all the reported bugs in the last week. Should have something for you to test next week, if no other surprises come up
1
Gustavo
Gustavo
09:00 PMI was trying to delete and recreate instead of upserting, but it puts too much pressure into the server and starts to constantly give me errors when dealing with thousands of documents.
Jason
Jason
09:01 PMDeleting one by one is not as performant as deleting in a batch by query
Gustavo
Gustavo
09:02 PMCan I delete using the IDs in a query? Like `id in [a, b, c, d, ...]``
09:03
Gustavo
09:03 PMIn case it's not clear, I mean, I have the IDs of the documents I want to delete. So I'd need to make a query with those IDs.
Jason
Jason
09:03 PMYup,
id:=[a, b, c, d, ...]Gustavo
Gustavo
09:03 PMGonna try it
09:03
Gustavo
09:03 PMHow many items can I send in one query to be safe?
Jason
Jason
09:04 PMSince the parameter is sent as a query parameter, it takes a max of 2K characters
Gustavo
Gustavo
09:05 PMOk, I'll try here and let it know
1
09:17
Gustavo
09:17 PMWeirdly, sometimes I get the error
'$.input' is invalid. Please check the API reference: <https://platform.openai.com/docs/api-reference>. even doing the workaround of deleting the document and recreating.09:18
Gustavo
09:18 PMI was able to import 9900 documents (very fast using "delete by query") and the next batch gave me a lot of consecutive errors with that message.
09:19
Gustavo
09:19 PMIt will probably work if I just restart my script skipping the successful batches, but still I'm intrigued by the error. It should only happen when updating a document, not when recreating.
09:19
Gustavo
09:19 PMAnd the error happened in the whole batch:
0 documents imported successfully, 100 documents failed during import.09:20
Gustavo
09:20 PMThe code:
Jason
Jason
09:21 PMAre you seeing this error on delete on creation? https://typesense-community.slack.com/archives/C01P749MET0/p1686863856037699?thread_ts=1686862232.273089&cid=C01P749MET0
Gustavo
Gustavo
09:21 PMI assume it's on creation.
Jason
Jason
09:21 PMI wonder if there’s some API response from OpenAI API that we’re not handling properly
09:21
Jason
09:21 PMOr we’re may be passing blank strings somehow
09:22
Jason
09:22 PMCould you give me a script like this that replicates the $.input error message: https://gist.github.com/jasonbosco/7c3432713216c378472f13e72246f46b
Gustavo
Gustavo
09:24 PMI'm afraid I won't be able to reproduce with a Bash script because the error only happens when dealing with thousands of documents, and I'm not very good at Bash.
09:25
Gustavo
09:25 PMDoes a JS repro work for you?
Jason
Jason
09:25 PMYeah JS works too
Gustavo
Gustavo
09:25 PMI'll try to reproduce, let's see
Jason
Jason
09:35 PMCould you also give me a similar snippet to reproduce this error message: https://typesense-community.slack.com/archives/C01P749MET0/p1686862403175659?thread_ts=1686862232.273089&cid=C01P749MET0
Gustavo
Gustavo
09:52 PMWhat I know is: when I had the first error, I didn't have the
https://typesense-community.slack.com/archives/C01P749MET0/p1686864054410659?thread_ts=1686862232.273089&cid=C01P749MET0
delete and the retry in this code. Other than that, the same code.https://typesense-community.slack.com/archives/C01P749MET0/p1686864054410659?thread_ts=1686862232.273089&cid=C01P749MET0
09:52
Gustavo
09:52 PMI'm trying to reproduce here without success.
09:53
Gustavo
09:53 PMWill try one more thing.
09:57
Gustavo
09:57 PMBTW, it would help a lot to identify the issue if the error from Typesense included the request that was sent to OpenAI's API.
10:29
Gustavo
10:29 PMSo, here's what I found:
10:33
Gustavo
10:33 PM1. I couldn't reproduce the
2. I found that the
3. There's a single document like that, but the whole batch fails saying
server_error from my first message.2. I found that the
invalid_request_error error happening in the 100th batch is because Typesense is trying to generate the embedding from a field that's an array and is empty.3. There's a single document like that, but the whole batch fails saying
0 documents imported successfully, 100 documents failed during import.10:34
Gustavo
10:34 PMIndeed, if Typesense's error included the request being sent to OpenAI, I'd probably immediately identify the invalid input being sent in that specific document.
Jason
Jason
10:35 PMAh good idea
10:36
Jason
10:36 PMAdded this to our todo list
1
Gustavo
Gustavo
10:36 PMOr maybe not because the problematic document was the 75th in the batch, so I wouldn't necessarily read all the 100 requests/errors and notice that specific one. So one thing that could be improved would be to prevent failing the whole batch, make it give me an error only for the actually problematic document.
Jason
Jason
10:37 PMHmm, we shouldn’t be failing the whole batch on a single document failure already… Will look into this
1
Gustavo
Gustavo
10:37 PMI guess it could be an unhandled error crashing the whole thing in your code.
10:40
Gustavo
10:40 PMIt's probably sending an empty string or something like that to OpenAI's API, which is the cause of the error, so maybe just assign some sort of null embedding (
[0, 0, ...]?) in that case instead of crashing.10:41
Gustavo
10:41 PMI mean, assign a null embedding instead of making the request to OpenAI, so it doesn't crash.
10:42
Gustavo
10:42 PMBut I'm not sure if failing silently is ideal. Just writing some ideas here without too much thought.
Jason
Jason
11:38 PMSo it turns out that OpenAI’s API fails the whole API call, even if one of the strings in a batch embedding request has an issue
11:39
Jason
11:39 PMWe make one batch embedding call to OpenAI’s API for all the documents in a Typesense import API call
Jun 16, 2023 (3 months ago)
Gustavo
Gustavo
12:00 AMOh, got it. So there doesn't seem to be a lot you can do. Maybe simply document that behavior.
Jason
Jason
12:01 AMIn the upcoming build, we’re going to filter out all blank strings before we send it to openai, so at least that error is avoided. But if there are any other errors, yeah we have to fail the full batch on our side
Gustavo
Gustavo
12:04 AMHow will the embedding field look like in that case?
Jason
Jason
12:06 AMWe’ll set it to
null1
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2779 threads (79% resolved)
Similar Threads
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
46
7mo
Solved
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
22
5mo
Solved
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
43
12mo
Solved
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
91
22mo
Investigating Unhealthy Cluster and Typesense Issues
Gustavo reported an unhealthy cluster, which Jason identified was due to stalled writes from OpenAI's API. The cluster was temporarily stabilized by downgrading to RC35, but a full resolution is still pending.
44
3mo