#community-help

Issues in Transferring Data from Firestore to Typesense

TLDR Ramees encountered issues in Google Cloud Function Log for data transfer from firestore to typesense. Jason helped troubleshoot, updated the cluster, increased the connection timeout, and resolved the issue.

Powered by Struct AI

1

Jul 23, 2021 (30 months ago)
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:09 AM
Hi, I am getting an error at Google Cloud Function Log while transferring data from firestore to typesense.
Request #1627012857493: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:12 AM
Could you share a screenshot of your extension configuration, excluding the api key?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:17 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:20 AM
Could you DM me the Typesense Host field?
04:27
Jason
04:27 AM
May I know the number of records you have in your Firestore collection?
04:27
Jason
04:27 AM
And the rough size of each record?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:29 AM
I have only 7 documents in the collection
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:29 AM
The issue is that the cluster has run out of memory, for the amount of data indexed so far. And once RAM runs out things get unpredictable.
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:30 AM
I am just starting so. for the mean time i kept the ram to .5gb
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:32 AM
Oh hmmm, 7 documents shouldn't fill up memory so much...
04:33
Jason
04:33 AM
Looking deeper
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:34 AM
by the way this is my cluster config
04:34
Ramees
04:34 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:37 AM
I've restarted the node which has addressed memory consumption and stabilized the node. But I'd like to see if we can replicate this issue.
04:37
Jason
04:37 AM
Could you try inserting another doc into Firestore?
04:40
Jason
04:40 AM
Also, was the last document you added to Firestore particularly large (like say 500KB for eg)?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:43 AM
still having the same error.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:44 AM
Can I upgrade your cluster to v0.21.0? We addressed one related issue in there
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:45 AM
ok
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:46 AM
I think I'm going to have to also increase the write timeout in the extension, that might be part of the issue
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:46 AM
regarding the size. I have around 20 fileds with each lessthan 500 character length. should be around 1 - 2 kb

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:46 AM
It's basically retrying multiple times, before the previous request is done and that's clogging things up
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:47 AM
ah ok.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:54 AM
Ok, I've updated the cluster to v0.21.0 and also published a new version of the extension that increases the connection timeout.

Could you visit the "Extensions" section of your Firebase project, and click on Upgrade next to the firestore-typesense-search extension? v0.2.4 is the version with the increased timeout
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:54 AM
Ok, I will do
05:19
Ramees
05:19 AM
It is working now. but got new problem.
05:20
Ramees
05:20 AM
05:24
Ramees
05:24 AM
If I change the field to geopoint, I got another error
05:24
Ramees
05:24 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:32 AM
Could you make the elements of the location array integers?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
05:32 AM
Ok I will
05:34
Ramees
05:34 AM
thank you. it is working
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:41 AM
Awesome!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

5

96
14mo

Troubleshooting Indexing Duration in Typesense Import

Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.

5

43
15mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

3

30
10mo

Bulk Import 50MB JSON Files Error - Timeout and Solutions

madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.

4

21
26mo

Revisiting Typesense for Efficient DB Indexing and Querying

kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.

1

46
9mo