#community-help

Troubleshooting 409 Errors with Firestore to Typesense Cloud Function

TLDR Orion encounters 409 errors with ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite cloud function. Jason suggests possible solutions like querying Firestore on each change or tracking sync state in a collection. Both agreed on adding a config option. Orion proposed contributing a PR for the change.

Powered by Struct AI

1

1

1

47
7mo
Solved
Join the chat
Oct 21, 2022 (12 months ago)
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:54 PM
I’m getting occasional 409 errors with the ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite cloud function. I don’t understand why this would throw an error as it shouldn’t matter if a document exists to update it… any idea what I’m doing wrong here?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:37 PM
Could you post the full log message with a few lines before and after?
Feb 25, 2023 (7 months ago)
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
10:15 PM
Sorry this is a few months behind... Here's a recent instance of the error.

Log says "creating document" and not the usual "upserting" which I'm sure is part of this problem but its unclear how. This is the log from document creation:
{
  "textPayload": "Creating document {\"measure_text_length\":343,\"rank\":0.5,\"text\":\"Indexers certainly have a very strong want for data determinism, they are constantly monitoring and worried about their data veracity. it's very time consuming\\nConsumers are also feeling this too since we've seen cases with \\\"wrong\\\" data returned by queries\\nBut no we have not had an attack like that yet that we know of, but what's your point?\",\"title\":\"slack\",\"type\":\"message\",\"url\":\"\",\"id\":\"aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDM2TkFXUDBDVC9wMTY3NzI2MTc1MjM5NTI0OT9jaWQ9QzAzNk5BV1AwQ1QmdGhyZWFkX3RzPTE2NzYwNTk2NjEuMjQzMDg5\"}",
  "insertId": "63f8fbbb0005cfd5c189c11f",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "knowledge-management-333914",
      "region": "us-central1",
      "function_name": "ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite"
    }
  },
  "timestamp": "2023-02-24T18:02:35.380885Z",
  "severity": "DEBUG",
  "labels": {
    "instance_id": "00c61b117c5a189878a95fae39681518ed338d4fa826a35986e68469d2ec5b3070fede155da6333715bb7dfc26d0c4c0f650d7a4a775945d9115",
    "execution_id": "qu4q7bkf0lq7"
  },
  "logName": "projects/knowledge-management-333914/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/knowledge-management-333914/traces/a9c2419e85dd694ae3fe11d14d31bebf",
  "receiveTimestamp": "2023-02-24T18:02:35.518729400Z"
}

And this is the error:
{
  "textPayload": "Error: Request failed with HTTP code 409 | Server said: A document with id aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDM2TkFXUDBDVC9wMTY3NzI2MTc1MjM5NTI0OT9jaWQ9QzAzNk5BV1AwQ1QmdGhyZWFkX3RzPTE2NzYwNTk2NjEuMjQzMDg5 already exists.\n    at ApiCall.customErrorForResponse (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:229:21)\n    at ApiCall.performRequest (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:118:48)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
  "insertId": "63f8fbbb000bdb5feae60206",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "knowledge-management-333914",
      "region": "us-central1",
      "function_name": "ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite"
    }
  },
  "timestamp": "2023-02-24T18:02:35.777055Z",
  "severity": "ERROR",
  "labels": {
    "instance_id": "00c61b117c5a189878a95fae39681518ed338d4fa826a35986e68469d2ec5b3070fede155da6333715bb7dfc26d0c4c0f650d7a4a775945d9115",
    "execution_id": "qu4q7bkf0lq7"
  },
  "logName": "projects/knowledge-management-333914/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/knowledge-management-333914/traces/a9c2419e85dd694ae3fe11d14d31bebf",
  "receiveTimestamp": "2023-02-24T18:02:35.852702927Z"
}
Feb 26, 2023 (7 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:43 AM
Hmm, so it looks like this document already exists in Typesense, but Firestore is calling the extension as a new document…
12:43
Jason
12:43 AM
Does this document get created and then updated shortly after in Firestore?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
12:48 AM
In the above case I see a log for upserting and then creating the doc, in other cases where creation precedes updating it seems to run fine
12:49
Orion
12:49 AM
{
  "textPayload": "Upserting document {\"measure_text_length\":343,\"platform\":\"\",\"rank\":0.5,\"text\":\"Indexers certainly have a very strong want for data determinism, they are constantly monitoring and worried about their data veracity. it's very time consuming\\nConsumers are also feeling this too since we've seen cases with \\\"wrong\\\" data returned by queries\\nBut no we have not had an attack like that yet that we know of, but what's your point?\",\"title\":\"slack\",\"type\":\"message\",\"url\":\"\",\"id\":\"aHR0cHM6Ly9ibG9ja3NjaWVuY2V0ZWFtLnNsYWNrLmNvbS9hcmNoaXZlcy9DMDM2TkFXUDBDVC9wMTY3NzI2MTc1MjM5NTI0OT9jaWQ9QzAzNk5BV1AwQ1QmdGhyZWFkX3RzPTE2NzYwNTk2NjEuMjQzMDg5\"}",
  "insertId": "63f8fbba00080662c369637e",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite",
      "project_id": "knowledge-management-333914"
    }
  },
  "timestamp": "2023-02-24T18:02:34.525922Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "qu4q00yzm4k2",
    "instance_id": "00c61b117c5a189878a95fae39681518ed338d4fa826a35986e68469d2ec5b3070fede155da6333715bb7dfc26d0c4c0f650d7a4a775945d9115"
  },
  "logName": "projects/knowledge-management-333914/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/knowledge-management-333914/traces/88dc0c0598a37d19e7d0901e25dfeb9c",
  "receiveTimestamp": "2023-02-24T18:02:34.855731622Z"
}
12:49
Orion
12:49 AM
^ that's about 300ms before the crash
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:50 AM
I see… This is most probably a result of Firestore not triggering change events in order unfortunately: https://github.com/typesense/firestore-typesense-search/issues/32
12:51
Jason
12:51 AM
So in your case the update event shows up before the create event
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
12:51 AM
Yeah I wondered about that from the seemingly random probability of this error
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:53 AM
Question for you: I was thinking of this as a solution:

> One potential solution to this could be to query Firestore on each change trigger and push the latest version of the Firestore document to Typesense, instead of using the snapshot document from the event
But from a cost perspective, do you think doing these reads for each Firestore write is reasonable? I guess it will essentially increase write costs by 33%?
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
12:56 AM
Hmmmm, hard to say, it feels very much like a workaround. It would probably work for my case but only because it's not already a high-cost function.
12:57
Orion
12:57 AM
Is there a need to separate upserting and creation on the firestore side? Would it be reasonable to have a single call for both? An upsert-or-create-if-it-doesnt-exist kinda thing...
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:58 AM
Yeah that would solve the 409 error from showing up… But then if the create shows up after the update, the document will now be stale in Typesense, since the create event will have a stale version of the doc (when it was created) in its payload
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:03 AM
Wouldn't it just mean you'd have two update_or_create calls? So the order of events in my case would be a bit like:
1. document added to firestore
2. TS update_or_create triggered -> TS document doesn't exist, creates it
3. document updated with "platform" field
4. TS update_or_create triggered -> TS document exists, updates it
Ahhh... I see what you mean now...
01:03
Orion
01:03 AM
Because you're using the snapshot from the event...
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:03 AM
Right
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:08 AM
The other solution mentioned on the GH issue, does that sync state need to be stored in firestore? if write triggers have an accurate timestamp can that be used to order TS updates?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:10 AM
We still need to store the state of what the last synced timestamp is for each document, to be able to know whether to discard an event
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:11 AM
To avoid redundant calls to typesense?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:11 AM
To prevent processing stale events
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:12 AM
Right
01:13
Orion
01:13 AM
Hard to see a perfect solution here... Just different cost tradeoffs
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:14 AM
Unfortunately yeah… That’s why I didn’t make this change. Figured if this is an issue for a particular use case, the 409 error will at least be a hint that this is happening
01:15
Jason
01:15 AM
And then a good (may be $$ solution) could be to do something like this using a set of custom functions: https://typesense.org/docs/guide/syncing-data-into-typesense.html#polling-your-primary-database
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:22 AM
So the options are:
• pull from firestore directly instead of using the snapshot -> increase write costs
• track sync state in a collection -> increased storage usage (and still need more than just the snapshot anyway? I guess this requires the first option too)
• periodically sync -> increased cost from queries and storage and unneeded periodic checks when no activity is happening?
01:23
Orion
01:23 AM
For me personally pulling from firestore directly would be fine
01:23
Orion
01:23 AM
I don't know that any solution is perfect but I wonder if there's a good option to expose in the extension config
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:27 AM
I think #1 is the simplest to expose as a config option…

#2 also increases storage costs, so #1 might be better

#3 requires a completely different extension, since it doesn’t rely on triggers
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:28 AM
Yeah and I suppose #3 cant be automatically configured as an extension afaik and would require extra action to setup periodic events/triggers in GCP?
01:29
Orion
01:29 AM
#1 would be a nice and simple option for sure
01:31
Orion
01:31 AM
So with these out-of-order issues does that mean that some of the documents in TS will be out-of-date? Perhaps not in my case as quick subsequent writes only happen after initial creation. But if you had an existing document with two quick writes you can end up with stale data on TS?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:31 AM
Correct
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:33 AM
Gotcha, I would definitely turn on a config option for #1 if it was there. Not because it's vitally important to be in perfect sync for users but because it makes debugging so much easier when the Firestore-TS link is 100% predictable. Then I know a bug has to be my fault :rolling_on_the_floor_laughing:
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:34 AM
Haha!
01:35
Jason
01:35 AM
Could you summarize this config flag in that Github issue? I can take a stab at it in the coming weeks
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:35 AM
Yeah sure, if I get a chance I can try and create a PR, seems like it should be a simple change looking at the src now
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:36 AM
A PR would be awesome!
Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:37 AM
Got any links to help test firestore extensions? Or I can just PR and cross my fingers (or you could test it on your end)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
01:38
Jason
01:38 AM
There’s also a set of commands to enable extensions in the emulator. The CI config shows these: https://github.com/typesense/firestore-typesense-search/blob/4e46be1b35494d11dda7c5c1a799e9be227d35e6/.circleci/config.yml

1

Orion
Photo of md5-8e802b48c0369226a7b50a22ab6e9e0c
Orion
01:45 AM
Do you think the flag should be "read from firestore" (off by default) or the opposite, "read from event" (on by default)?
01:45
Orion
01:45 AM
^ flag names tbd of course
03:14
Orion
03:14 AM
Jason I created a very quick PR that I haven't had a chance to test (also never touched firestore extensions, etc, etc) but figured it worth getting the ball rolling.
03:15
Orion
03:15 AM
If no one picks it up I will get to testing but i don't know when I'll have a sec to do that.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:08 PM
Thank you! I’ll take a look this coming week or early week after.

1