#community-help

Memory Limit Issues in Firestore Backfill with Typesense

TLDR Ne experienced memory limit issues when performing backfills with the Firestore Typesense extension. Kishore Nallan pointed them to a similar issue on a forum for potential solutions.

Powered by Struct AI

1

Jan 31, 2023 (10 months ago)
Ne
Photo of md5-90b523329c7c04dac1151f862a3be7ff
Ne
07:07 AM
Anybody getting Memory limit exceeded when backfilling data through Firestore with Typesense extension? Trying to backfill 150K documents, but it just stops; the issue is with the cloud function (i bet), not the Typesense cluster.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:08 AM
Can you please post the exact error message?
Ne
Photo of md5-90b523329c7c04dac1151f862a3be7ff
Ne
07:17 AM
Errors I see on cloud logs for the extension.:

Warning: "The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability"

Error: "Function invocation was interrupted. Error: memory limit exceeded."


Backfilling works but fails after a few records ("few" can range from 10 to 500 docs). If I trigger false, then true the "backfill" document it starts then fail again.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:25 AM
Looks like we've had others run into this limitation, probably because of the size of documents hitting the memory limits for the function: https://github.com/typesense/firestore-typesense-search/issues/10
07:26
Kishore Nallan
07:26 AM
The last comment on that thread talks about a work around.
Ne
Photo of md5-90b523329c7c04dac1151f862a3be7ff
Ne
07:26 AM
Thanks! Looking at this now. Will update.

1