#community-help

Memory Limit Issues in Firestore Backfill with Typesense

TLDR Ne experienced memory limit issues when performing backfills with the Firestore Typesense extension. Kishore Nallan pointed them to a similar issue on a forum for potential solutions.

10mo

Join the chat

Jan 31, 2023 (10 months ago)

Photo of md5-90b523329c7c04dac1151f862a3be7ff

Ne

07:07 AM

Anybody getting Memory limit exceeded when backfilling data through Firestore with Typesense extension? Trying to backfill 150K documents, but it just stops; the issue is with the cloud function (i bet), not the Typesense cluster.

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

07:08 AM

Can you please post the exact error message?

Ne

07:17 AM

Errors I see on cloud logs for the extension.:

Warning: "The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability"

Error: "Function invocation was interrupted. Error: memory limit exceeded."

Backfilling works but fails after a few records ("few" can range from 10 to 500 docs). If I trigger false, then true the "backfill" document it starts then fail again.

Kishore Nallan

07:25 AM

Looks like we've had others run into this limitation, probably because of the size of documents hitting the memory limits for the function: https://github.com/typesense/firestore-typesense-search/issues/10

07:26

Kishore Nallan

07:26 AM

The last comment on that thread talks about a work around.

Ne

07:26 AM

Thanks! Looking at this now. Will update.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Typesense and Firebase Integration Troubleshooting

Bobby struggled with getting data to Typesense from Firebase. After sharing his logs and settings screenshots, Jason helped him fix the path configuration and suggested ways to successful re-trigger a backfill. This resolved the issue.

21mo

Issues with typesense-firestore extension in Typesense Cloud

J is experiencing issues with typesense-firestore extension in Typesense Cloud. Jason suggested checking the extension logs for potential data validation errors.

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

14mo